The library is available via Nuget with the package name 'Spidey'. If you, instead, new up a Crawler object then you will need to compose the Pipeline object. After that the system will automatically pick it up if resolved from the service provider. In order to replace the default in any of these systems all you need to do is create a class that implements the interface that you want to replace. These subsystems all implement interfaces found in the namespace. Pipeline (IPipeline) - Manages the various parts of the process by feeding the content to the next bit of the process.Scheduler (IScheduler) - Handles handing out work to the various workers.The default one simply calls the method provided in the options. Processor (IProcessor) - Takes the parsed content and hands it off to your code.

Link Discoverer (ILinkDiscoverer) - Takes the content from the engine and looks for links to other resources.Engine (IEngine) - This downloads the content from the server.

Content Parser (IContentParser) - This parses the resulting data and converts it to the ResultFile object.

The system is divided into the following sections: Note that it's possible to customize the crawler's various parts. The callback method is what will be called by the system once a link's info has been received and looks like this: Anyway, the Options class has a number of properties, some of which are not displayed above such as NetworkCredentials, UseDefaultCredentials, and Proxy. You can simply new up an instance of Crawler if you want. Note that it is recommended that you actually register the Options object in your ServiceCollection and resolve the Crawler object from the service provider but it is not required.

ItemFound = FoundFile=> //When the system hits one of the keys in the dictionary, it will replace it with the value. ServiceCollection.AddSingleton(new Options

Download spidey crawler