Skip to content

Wget

"GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc." - https://www.gnu.org/software/wget

Examples

Mirror site for local viewing

From the man page: to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to -p:

wget -E -H -k -K -p "https://$site/$dir"

Download all images from a site

To politely download all images from within a current remote directory:

wget \
  --accept ".jpg" \
  --adjust-extension \
  --continue \
  --no-parent \
  --random-wait \
  --recursive \
  --timestamping \
  --tries=0 \
  --wait=2 \
  --waitretry=30 \
  "https://$site/$dir/"

Simple use of cookies

Some servers that need referrers and cookies can be accessed by doing:

wget --save-cookies="cookies.txt" "foo.html"
wget --load-cookies="cookies.txt" --referer="foo.html" "foo.mp3"

Set default behavior

~/.wgetrc sets default parameter values

tries=0
continue=1

See Also