Suck the InTernet

SuckIT allows you to recursively visit and download a website's content to your disk.

 suckit 0.2.0
 CLI arguments

 USAGE:
     suckit [FLAGS] [OPTIONS] <url>

 FLAGS:
     -c, --continue-on-error                  Flag to enable or disable exit on error
         --disable-certs-checks               Dissable SSL certificates verification
         --dry-run                            Do everything without saving the files to the disk
     -h, --help                               Prints help information
     -V, --version                            Prints version information
     -v, --verbose                            Enable more information regarding the scraping process
         --visit-filter-is-download-filter    Use the dowload filter in/exclude regexes for visiting as well

 OPTIONS:
     -a, --auth <auth>...
             HTTP basic authentication credentials space-separated as "username password host". Can be repeated for
             multiple credentials as "u1 p1 h1 u2 p2 h2"
         --delay <delay>
             Add a delay in seconds between downloads to reduce the likelihood of getting banned [default: 0]

     -d, --depth <depth>
             Maximum recursion depth to reach when visiting. Default is -1 (infinity) [default: -1]

     -e, --exclude-download <exclude-download>
             Regex filter to exclude saving pages that match this expression [default: $^]

         --exclude-visit <exclude-visit>
             Regex filter to exclude visiting pages that match this expression [default: $^]

         --ext-depth <ext-depth>
             Maximum recursion depth to reach when visiting external domains. Default is 0. -1 means infinity [default:
             0]
     -i, --include-download <include-download>
             Regex filter to limit to only saving pages that match this expression [default: .*]

         --include-visit <include-visit>
             Regex filter to limit to only visiting pages that match this expression [default: .*]

     -j, --jobs <jobs>                            Maximum number of threads to use concurrently [default: 1]
     -o, --output <output>                        Output directory
         --random-range <random-range>
             Generate an extra random delay between downloads, from 0 to this number. This is added to the base delay
             seconds [default: 0]
     -t, --tries <tries>                          Maximum amount of retries on download failure [default: 20]
     -u, --user-agent <user-agent>                User agent to be used for sending requests [default: suckit]

 ARGS:
     <url>    Entry point of the scraping

This is an unofficial package of the SuckIT source which can be found at https://github.com/Skallwar/suckit. Made using the snap configuration found at https://github.com/popey/suckit-snap