Wget
"GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc." - https://www.gnu.org/software/wget
Examples
Mirror site for local viewing
From the man page: to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to -p:
wget -E -H -k -K -p "https://$site/$dir"
Download all images from a site
To politely download all images from within a current remote directory:
wget \
--accept ".jpg" \
--adjust-extension \
--continue \
--no-parent \
--random-wait \
--recursive \
--timestamping \
--tries=0 \
--wait=2 \
--waitretry=30 \
"https://$site/$dir/"
Simple use of cookies
Some servers that need referrers and cookies can be accessed by doing:
wget --save-cookies="cookies.txt" "foo.html"
wget --load-cookies="cookies.txt" --referer="foo.html" "foo.mp3"
Set default behavior
~/.wgetrc sets default parameter values
tries=0
continue=1