cURL
"command line tool and library for transferring data with URLs" - https://curl.haxx.se
curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
Examples
Follow location redirects
curl -L http://whatever
Show the HTTP code for a given request
Use the http method HEAD to only the headers of the remote URI and shows the HTTP code. This is useful for efficiently checking the existence of a URI that would be a large download.
curl --head -s -w "%{http_code}\n" -o /dev/null http://www.example.com/
Not all HTTP servers support the HEAD method though, so a slightly worse alternative is:
curl --max-filesize 1 -s -w "%{http_code}\n" -o /dev/null http://www.example.com/
--max-filesize 1 will download something more than 0 bytes, though usually not the whole file. Notable is that curl does not seem to tell the truth about how many bytes were actually downloaded when using --max-filesize 1 -w '%{size_download}'. This can be verified by opening a simple http server (eg: python3 -m http.server), then running tcpflow -c, and then requesting the root dir of your simple http server. You will see the directory listing being sent across the wire, but curl will report 0 bytes being downloaded. The true download size depends on what file is being downloaded and how big it is. For large binary files, I have observed this as being 0 bytes.
Request a specific vhost from a server
This is useful for testing production code on non-production multi-tennant name based virtual hosts.
curl -H 'Host: www.domain.com' http://example.com
Get the length of the file to be downloaded
curl -qI https://www.google.com/index.php 2>/dev/null | awk '/Length/ {print $2}'
Fetch only HTTP headers, not content
curl -I http://www.example.com/some_huge_file.iso
Send POST variables
curl --data "user=foo&pass=bar" http://example.com/login.php
Scrape URLs from a page
This appears to have problems with some strings. For instance, this doesn't catch the full https://accounts.google.com string. The regex is correct according to http://regexpal.com, but egrep is apparently not handling it correctly.
curl -s http://www.google.com | egrep -o '(((https?|ftp|gopher)://|(mailto|file|news):)[^’ <>\n"]+|(www|web|w3)\.[-a-z0-9.]+)[^’ .,;<>":]'
Use curl to fetch the current RFC 2822 time
If you don't have NTP you can use this to manually feed the current time into date -s to set your system clock to within a few seconds of accuracy.
curl -sIH 'Cache-Control: no-cache' example.org | grep '^Date'
Using GNU cut and date (IE: on linux, like on a Raspberry Pi image that does not have NTP properly set up) you can set your time using this command:
sudo date -s "$(curl -sIH 'Cache-Control: no-cache' example.org | grep '^Date:' | cut -f 1 -d ' ' --complement)"