A simple web crawler, mainly targets for link validation test.
- running in BFS or DFS mode
- specify concurrent running workers in BFS mode
- crawl seeds can be set to more than one urls
- support crawl with cookies
- configure hyper links regex, including match type and ignore type
- group visited urls by HTTP status code
- flexible configuration in YAML
- send test result by mail, through SMTP protocol or mailgun service
- cancel jobs
$ pip install -U git+https://github.com/debugtalk/WebCrawler.git#egg=requests-crawler --process-dependency-linksTo ensure the installation or upgrade is successful, you can execute command webcrawler -V to see if you can get the correct version number.
$ webcrawler -V jenkins-mail-py version: 0.2.4 WebCrawler version: 0.3.0$ webcrawler -h usage: webcrawler [-h] [-V] [--log-level LOG_LEVEL] [--config-file CONFIG_FILE] [--seeds SEEDS] [--include-hosts INCLUDE_HOSTS] [--cookies COOKIES] [--crawl-mode CRAWL_MODE] [--max-depth MAX_DEPTH] [--concurrency CONCURRENCY] [--save-results SAVE_RESULTS] [--grey-user-agent GREY_USER_AGENT] [--grey-traceid GREY_TRACEID] [--grey-view-grey GREY_VIEW_GREY] [--mailgun-api-id MAILGUN_API_ID] [--mailgun-api-key MAILGUN_API_KEY] [--mail-sender MAIL_SENDER] [--mail-recepients [MAIL_RECEPIENTS [MAIL_RECEPIENTS ...]]] [--mail-subject MAIL_SUBJECT] [--mail-content MAIL_CONTENT] [--jenkins-job-name JENKINS_JOB_NAME] [--jenkins-job-url JENKINS_JOB_URL] [--jenkins-build-number JENKINS_BUILD_NUMBER] A web crawler for testing website links validation. optional arguments: -h, --help show this help message and exit -V, --version show version --log-level LOG_LEVEL Specify logging level, default is INFO. --config-file CONFIG_FILE Specify config file path. --seeds SEEDS Specify crawl seed url(s), several urls can be specified with pipe; if auth needed, seeds can be specified like user1:pwd1@url1|user2:pwd2@url2 --include-hosts INCLUDE_HOSTS Specify extra hosts to be crawled. --cookies COOKIES Specify cookies, several cookies can be joined by '|'. e.g. 'lang:en,country:us|lang:zh,country:cn' --crawl-mode CRAWL_MODE Specify crawl mode, BFS or DFS. --max-depth MAX_DEPTH Specify max crawl depth. --concurrency CONCURRENCY Specify concurrent workers number. --save-results SAVE_RESULTS Specify if save results, default is NO. --grey-user-agent GREY_USER_AGENT Specify grey environment header User-Agent. --grey-traceid GREY_TRACEID Specify grey environment cookie traceid. --grey-view-grey GREY_VIEW_GREY Specify grey environment cookie view_gray. --mailgun-api-id MAILGUN_API_ID Specify mailgun api id. --mailgun-api-key MAILGUN_API_KEY Specify mailgun api key. --mail-sender MAIL_SENDER Specify email sender. --mail-recepients [MAIL_RECEPIENTS [MAIL_RECEPIENTS ...]] Specify email recepients. --mail-subject MAIL_SUBJECT Specify email subject. --mail-content MAIL_CONTENT Specify email content. --jenkins-job-name JENKINS_JOB_NAME Specify jenkins job name. --jenkins-job-url JENKINS_JOB_URL Specify jenkins job url. --jenkins-build-number JENKINS_BUILD_NUMBER Specify jenkins build number. Specify config file.
$ webcrawler --seeds http://debugtalk.com --crawl-mode bfs --max-depth 5 --config-file path/to/config.ymlCrawl in BFS mode with 20 concurrent workers, and set maximum depth to 5.
$ webcrawler --seeds http://debugtalk.com --crawl-mode bfs --max-depth 5 --concurrency 20Crawl in DFS mode, and set maximum depth to 10.
$ webcrawler --seeds http://debugtalk.com --crawl-mode dfs --max-depth 10Crawl several websites in BFS mode with 20 concurrent workers, and set maximum depth to 10.
$ webcrawler --seeds http://debugtalk.com,http://blog.debugtalk.com --crawl-mode bfs --max-depth 10 --concurrency 20Crawl with different cookies.
$ webcrawler --seeds http://debugtalk.com --crawl-mode BFS --max-depth 10 --concurrency 50 --cookies 'lang:en,country:us|lang:zh,country:cn' WebCrawler supports Python 2.7, 3.3, 3.4, 3.5, and 3.6.
Open source licensed under the MIT license (see LICENSE file for details).