Downloading with wget

wget speaks HTTP: give it a URL and it downloads what the server sends back. It needs no browser and no mouse, so it is how servers fetch software, scripts grab data and admins test that a web service answers. All the live examples in this lesson need the machine's network switched on (Built-in mode) and an address (udhcpc -i eth0).

~% cd /tmp
~% wget http://httpbin.org/robots.txt
Connecting to httpbin.org (192.168.87.1:80)
robots.txt           100% |*******************************|    30   0:00:00 ETA
~% cat robots.txt
User-agent: *
Disallow: /deny

By default the file lands in the current directory, named after the last part of the URL. The progress bar and the *Connecting to* line go to stderr, so they do not pollute redirected output. Choose where the data goes with these options:

~% wget -q -O - http://httpbin.org/user-agent
{
  "user-agent": "Wget"
}
~% wget -S -O /dev/null http://httpbin.org/get
Connecting to httpbin.org (192.168.87.1:80)
  HTTP/1.1 200 OK
  access-control-allow-origin: *
  content-type: application/json
~% wget -q --spider http://httpbin.org/status/404; echo $?
1

The first line of a response is the status code: 200 OK, 301/302 redirect, 404 not found, 500 server error. wget exits with 0 only on success, which makes it a good health check in scripts: wget -q --spider URL && echo alive.

What the sandbox lets through

Your VM runs inside a browser tab, and a web page cannot open raw network connections. TempMV's virtual network accepts the guest's TCP connection to port 80, reads the HTTP request, and replays it as an ordinary browser fetch(). That has honest consequences, and learning them teaches you how the web works:

• Write http://. This BusyBox wget has no TLS (not an http or ftp url: https://…); the browser upgrades the request to HTTPS for you when needed.
• Only port 80: http://host:8080/ gets Connection refused. SSH, mail or database ports cannot work.
• Only sites that allow cross-origin requests (CORS) answer. Others produce server returned error: HTTP/1.1 502 Fetch Error, written by the emulator when the browser blocked the response. Sites that work: http://httpbin.org/get, http://api.github.com/zen, files on http://cdn.jsdelivr.net/.