Best practices for multi-domain crawling and per-host configuration

[loic-bellinger]

Hi Crawlee Team,

First off, thank you for this fantastic library.

As I've been integrating Crawlee into a project that requires crawling many different hosts simultaneously, I've noticed that the current documentation and examples primarily focus on single-domain scenarios. While these are great for getting started, they don't address some of the common challenges that arise in large-scale, multi-domain crawling.
I would be incredibly valuable to have a guide or an example that details the recommended strategies for this kind of setup. Specifically, I'm trying to figure out the best approach for the following:

• Overall Architecture: Is it better to spawn a separate crawler instance for each host, or to use a single, powerful crawler instance to handle all hosts? My intuition leans towards a single crawler to efficiently manage resources (like a browser pool), but I'm unsure how to manage per-host configurations in that context.
• Per-Host Concurrency and Rate Limiting: How can we effectively manage concurrency when different hosts have vastly different rate limits? For example, Host A might handle 1,000 requests per minute, while Host B only allows 30. What is the recommended way to enforce maxConcurrency or maxRequestsPerMinute on a per-hostname basis within a single crawler?
• Session Management: To manage things like cookies, headers, and proxies for different sites, the most logical approach seems to be using the SessionPool with a session ID tied to the hostname (e.g., session_id=host). Is this the recommended best practice?
  - Resource Allocation: In a browser-based crawling scenario, how do we ensure that a slow, heavily-loaded website doesn't monopolize the entire browser pool, starving out other, faster websites?

Having a "Best Practices" guide for multi-domain crawling would be a fantastic addition to the documentation for anyone moving beyond single-site scraping. In the meantime, any guidance you could provide on the questions above would be greatly appreciated.

[Mantisus]

Thank you for using Crawlee and for your kind words about the library!

Optimal settings are quite specific to each individual project, as they depend on target websites, crawling volumes, infrastructure used, and proxy quality. Therefore, it's most optimal to configure the crawler for each specific domain.

However, to address your questions in more detail:

Is it better to spawn a separate crawler instance for each host, or to use a single, powerful crawler instance to handle all hosts?

I would divide target sites into groups. Simple HTML sites can be handled by a single crawler through different handlers. Heavy sites with extensive JS and slow performance should use one crawler per domain, especially if this requires something like Camoufox.

How can we effectively manage concurrency when different hosts have vastly different rate limits?

This topic has been coming up more frequently lately. I'd say that #1437 and #1396 are related to per-host concurrency capabilities. However, we don't have native support for this yet. Our current recommendation is to use different crawlers for domains if they require different rate limiting.

To manage things like cookies, headers, and proxies for different sites, the most logical approach seems to be using the SessionPool with a session ID tied to the hostname

The current cookie implementation in Session supports multi-domain usage, so in multi-domain scraping the main challenge for the pool is when one site works well with current proxies while another blocks them. If a site requires fine-grained control of cookie initialization and usage, I would recommend using a separate crawler for that domain, as it will be easier to maintain and debug.

In a browser-based crawling scenario, how do we ensure that a slow, heavily-loaded website doesn't monopolize the entire browser pool?

You can't. Isolate the heavy website into a separate crawler.

Thus, my main recommendation is to divide target sites into groups based on configuration requirements:

• Simple HTML sites with high availability can be handled by a single crawler
• Sites with low request-per-minute thresholds by multiple crawlers running in the same process
• Heavy JS sites with one crawler per site

This approach provides the best balance between resource efficiency and maintainability for multi-domain crawling scenarios.

[Pijukatel]

Are there any potential pitfalls, race conditions, or subtle issues with manipulating the global configuration via service_locator in a forked process like this?

Hello, there is one potential pitfall to be aware of when using ProcessPoolExecutor. The pool will reuse workers, and those workers will not be isolated from whatever happened to them previously. So, for example, if on one specific worker you already modified a class attribute, the next run on this same worker will also start with the modified class attribute, but that can be different from the value of this class attribute on another worker from the pool...

Unfortunately, we sometimes use some class attributes as global variables, which could cause you some trouble. Or it might not; it depends on your use case.

So if you have 4 sites and 4 workers, then no problem. But imagine 8 sites and 4 workers. It means that some sites will use a Crawler that will run on a previously used worker, and so the class attributes could be in a non-default state.

[Mantisus]

Is this a robust way to manage multiple, isolated crawler runs with Crawlee ?

I don't see any problems with this approach.

Although I myself would most likely use celery for ease of management.

But I admit that I may have missed something.

Are there any potential pitfalls, race conditions, or subtle issues with manipulating the global configuration via service_locator in a forked process like this?

This will work by isolating memory. But in general, I want to point out that now each crawler can be configured independently.
https://crawlee.dev/python/docs/upgrading/upgrading-to-v1#:~:text=This%20allows%20two%20crawlers%20with%20different%20services%20at%20the%20same%20time.

Thank you, @Pijukatel, for clarifying some details.

[loic-bellinger]

The pool will reuse workers, and those workers will not be isolated from whatever happened to them previously

To fix this, I think I can pass max_tasks_per_child=1 to ProcessPoolExecutor (but I'm not familiar with that setting). From the official doc:

max_tasks_per_child is an optional argument that specifies the maximum number of tasks a single process can execute before it will exit and be replaced with a fresh worker process. By default max_tasks_per_child is None which means worker processes will live as long as the pool. When a max is specified, the “spawn” multiprocessing start method will be used by default in absence of a mp_context parameter. This feature is incompatible with the “fork” start method.

I want to point out that now each crawler can be configured independently.

I'm not sure I understood what you meant. I can instantiate a new storage service per crawler easily, but setting a new storage_dir per crawler is more cumbersome. I guess it should instantiate a Configuration(storage_dir="path") and pass that configuration during my crawler instantiation ? That's more code, but if it improves isolation it's worth it.

[loic-bellinger]

Also, I wonder if the autoscaling /snapshotter logic would work in a multiprocessing pool context (1 crawler per process). have not time to dig into crawly internals to check that. I guess that two possible solutions would be to run a single, centralized Snapshotter in the parent process (system-wide metrics) or to give each worker its own per-worker resource budget (divide total memory/CPU among workers).

[Pijukatel]

I will try to answer to the best of my knowledge, but Crawlee was not designed with multiprocessing scenarios in mind. It can work, but there might be other hidden pitfalls that I am not aware of.

1. FileSystemStorageClient
FileSystemStorageClient based storages do not fully support multiple consumers working with them. It might lead to unexpected results as there is no locking mechanism implemented. So if you have two crawlers running at the same time, it is better to let each one of them use its own storage dir and isolate their storages. This you can do as you suggested, by using custom storage_dir per each new crawler and setting it in the Configuration used by that crawler.
   (Or you can try to use a different storage client, that supports multiple consumers)

2. autoscaling /snapshotter
   We have never tried using it like this in a multiprocessing context. Splitting your resources by number of crawlers and letting each one of them run with this reduced resource limit should work without an issue, I hope, but it will underutilize your system in many situations. (As one crawler that is running at full capacity can not take the unused resources of another crawler.)

Trying to use a single centralized autoscaling could be challenging. There are issues I can foresee, and I am not sure how to solve them out of the box:

• You have to ensure that all the resource measurements (memory, CPU, event loop delay) are correct. This might be very challenging.
• You should probably have some lock in place, that will allow to upscale/downscale only one crawler at the time - otherwise you might run out of resources, when all crawlers try to upscale at the same time.