Here's an example that I've used to get all the pages from Paul Graham's website:
$ wget --recursive --level=inf --no-remove-listing --wait=6 --random-wait --adjust-extension --no-clobber --continue -e robots=off --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" --domains=paulgraham.com https://paulgraham.com | Parameter | Description |
|---|---|
--recursive | Enables recursive downloading (following links) |
--level=inf | Sets the recursion level to infinite |
--no-remove-listing | Keep ".listing" files that are created to keep track of directory listings |
--wait=6 | Wait the given number of seconds between requests |
--random-wait | Multiplies --wait randomly between 0.5 and 1.5 for each request |
--adjust-extension | Make sure that ".html" is added to the files |
--no-clobber | Do not redownload a file if exists locally |
--continue | Allows resuming downloading a partially downloaded file |
-e robots=off | Ignores robots.txt instructions. |
--user-agent | Sends the given "User-Agent" header to the server |
--domains | Comma-separated list of domains to be followed |
--span-hosts | Allows navigating to subdomains |
Other useful parameters:
| Parameter | Description |
|---|---|
--page-requisites | Downloads things as inlined images, sounds, and referenced stylesheets |
--span-hosts | Allows downloading files from links that point to different hosts |
--convert-links | Converts links to local links (allowing local viewing) |
--no-check-certificate | Bypasses SSL certificate verification. |
--directory-prefix=/my/directory | Sets up the destination directory. |
--include-directories=posts | Comma-separated list of allowed directories to be followed when crawling |
--reject "*?*" | Rejects URLs that contain query strings |
Top comments (0)