Check out the blog post.

1. Fork/Clone

2. Create and activate a virtualenv

3. Install the requirements

4. Sign up for Digital Ocean and generate an access token

5. Add the token to your environment:

   $ export DIGITAL_OCEAN_ACCESS_TOKEN=[your_token]

6. Spin up four droplets and deploy Docker Swarm:

   $ sh project/create.sh

7. Run the scraper:

   $ NODE=$(docker service ps --format "{{.Node}}" selenium_hub) $ for i in {1..8}; do { python project/script.py ${i} $(docker-machine ip $NODE) & }; done

8. Bring down the resources:

   $ sh project/destroy.sh