wget

Download files and mirror sites from the command line

Updated 2026-08-16

wget retrieves things over HTTP, HTTPS and FTP and writes them to disk. Use it when you want a file on the filesystem rather than a response on your terminal, and when you want something that will walk an entire site and bring back every page it links to.

wget or curl?

They overlap heavily, and on Debian both are a package away. Day to day, the difference is what each does by default:

wget https://example.com     # saves the body to ./index.html
curl https://example.com     # prints the body to stdout

Everything follows from that. wget has a progress bar, resumes interrupted transfers with -c, and can recurse through links with -r; curl has finer control over the request itself and pipes naturally into other commands. Use wget to fetch files and mirror sites, and curl to talk to an API.

Output goes to stderr

wget writes its progress report to stderr, not stdout, so wget -O - URL | tar -xz works without the report corrupting the pipe. That is also why wget URL > file.log captures nothing useful: use -o file.log for the report, -O file for the payload.

Recursion needs limits

wget -r will follow links until it runs out, which on a real site means far more than you intended. Two flags do most of the work: -l caps the depth, and -np stops it climbing above the directory you started in. Add -A/-R to filter by extension.

Sample files used on this page

Every example below was run against these files. Recreate them to follow along.

the local test server start it with python3 http-mock.py before running the examples - the script is in the site repo at scripts/fixtures/http-mock.py, and the curl page uses the same one

Local test server for the curl and wget pages.

GET  /get                     echo the query string and request headers
POST /post                    echo the body, as form fields or JSON
PUT  /put                     echo the body
DELETE /delete                echo the request
GET  /headers                 echo the request headers only
GET  /user-agent              echo the User-Agent header only
GET  /basic-auth/user/pass    401 until the right credentials arrive
GET  /cookies/set/NAME/VALUE  set a cookie, then redirect to /cookies
GET  /cookies                 echo the cookies the request carried
GET  /redirect/N              redirect N times, then land on /get
GET  /status/CODE             answer with that status code
GET  /delay/SECONDS           wait, then answer
GET  /robots.txt              a small static file
GET  /page.html               a small static HTML page
GET  /site/index.html         a small linked site, for recursive download
GET  /top.html                one level above /site/, for --no-parent
GET  /large.bin               64KB, supports Range so a download can be resumed

Run a second copy with --tls to serve all of the above over HTTPS, with a self-signed
certificate that nothing trusts.

urls.txt the URL list the batch-download examples read

http://127.0.0.1:8080/site/a.html
http://127.0.0.1:8080/site/b.html

body.json the request body the --post-file example sends

{"name": "deb1", "role": "backup"}
50 outputs, collapsed by default

Downloading a single file

wget saves to a file named after the URL unless you tell it otherwise.

Download a file

wget http://127.0.0.1:8080/page.html

With no flags at all, wget works out the filename from the URL and writes it into the current directory, reporting progress on stderr as it goes.

Show output

Your output will differ: the timestamps and transfer rate are from the run that produced this

--2026-08-16 12:13:58--  http://127.0.0.1:8080/page.html
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 120 [text/html]
Saving to: 'page.html'

2026-08-16 12:13:58 (49.1 MB/s) - 'page.html' saved [120/120]

Save under a different name

wget -O homepage.html http://127.0.0.1:8080/page.html && ls homepage.html

-O (--output-document) names the output file explicitly, which also stops wget inventing a name from a URL that ends in a slash or a query string.

Show output

Your output will differ: the timestamps and transfer rate are from the run that produced this

--2026-08-16 12:13:58--  http://127.0.0.1:8080/page.html
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 120 [text/html]
Saving to: 'homepage.html'

2026-08-16 12:13:58 (61.0 MB/s) - 'homepage.html' saved [120/120]

homepage.html

Save into a different directory

wget -q -P downloads http://127.0.0.1:8080/page.html && ls downloads

-P (--directory-prefix) sets where files land without changing their names, and creates the directory if it isn't there.

Show output 
page.html

Download without any progress output

wget -q http://127.0.0.1:8080/page.html && echo "saved, silently"

-q (--quiet) silences everything, including errors, which is what you want inside a script that checks the exit code itself.

Show output 
saved, silently

One summary line instead of a progress bar

wget -nv http://127.0.0.1:8080/page.html

-nv (--no-verbose) is the middle ground between the full progress bar and -q: one line per file, still reporting errors.

Show output

Your output will differ: the timestamp and transfer rate are from the run that produced this

2026-08-16 12:13:58 URL:http://127.0.0.1:8080/page.html [120/120] -> "page.html" [1]

Print the file to stdout instead of saving it

wget -q -O - http://127.0.0.1:8080/page.html

-O - sends the payload to stdout, so it can be piped. The progress report goes to stderr, so it never mixes into the pipe.

Show output 
<!doctype html>
<title>Test page</title>
<h1>Test page</h1>
<p>A small static page served by the local test server.</p>

A second download does not overwrite the first

wget -q http://127.0.0.1:8080/page.html
wget -q http://127.0.0.1:8080/page.html
ls page.html*

By default wget keeps the existing file and writes the new copy as page.html.1, which surprises people who expect an overwrite.

Show output 
page.html
page.html.1

Skip the download if the file already exists

wget -q http://127.0.0.1:8080/page.html
wget -nc http://127.0.0.1:8080/page.html
ls page.html*

-nc (--no-clobber) leaves the existing file alone and doesn't fetch a numbered copy either, which makes a re-run of a download script cheap.

Show output 
File 'page.html' already there; not retrieving.

page.html

Resuming, retrying, and rate limits

The flags that make a download survive a bad connection.

Resume a partial download

head -c 20000 /dev/zero > large.bin
wget -c http://127.0.0.1:8080/large.bin
ls -l large.bin

-c (--continue) asks the server for the rest of the file with a Range request rather than starting again. The server has to support ranges; this one does.

Show output

Your output will differ: the timestamps and transfer rate are from the run that produced this

--2026-08-16 12:13:58--  http://127.0.0.1:8080/large.bin
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 65000 (63K), 45000 (44K) remaining [application/octet-stream]
Saving to: 'large.bin'

2026-08-16 12:13:58 (3.47 GB/s) - 'large.bin' saved [65000/65000]

-rw-r--r-- 1 user user 65000 Jul  5 15:35 large.bin

Retry a transient failure a fixed number of times

wget --tries=2 --timeout=1 http://127.0.0.1:8080/delay/5 -O /dev/null

--tries caps the attempts, and the report labels each one after the first with (try: N). The default is 20, which is a long time to wait on a server that has stopped answering.

Show output

Your output will differ: the timestamps are from the run that produced this

--2026-08-16 12:13:58--  http://127.0.0.1:8080/delay/5
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--2026-08-16 12:14:00--  (try: 2)  http://127.0.0.1:8080/delay/5
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.

What counts as worth retrying

wget --tries=3 -nv http://127.0.0.1:8080/status/500 2>&1 | tail -1

Only a transient failure is retried: a timeout, a dropped connection, a partial transfer. An HTTP error status is a definite answer, so --tries=3 still reports a 500 exactly once.

Show output

Your output will differ: the timestamps are from the run that produced this

2026-08-16 12:14:01 ERROR 500: Internal Server Error.

Time out instead of hanging

wget --timeout=2 --tries=1 -nv http://127.0.0.1:8080/delay/5 -O /dev/null 2>&1 | tail -1

--timeout bounds every stage of the transfer: name resolution, connecting, and reading. Without it a stalled server holds the script open indefinitely.

Show output 
Read error (Connection timed out) in headers.

Cap the download rate

wget -q --limit-rate=200k http://127.0.0.1:8080/large.bin && ls -l large.bin

--limit-rate leaves bandwidth for everything else on the connection. Accepts k and m suffixes.

Show output 
-rw-r--r-- 1 user user 65000 Jul  5 15:35 large.bin

Wait between retries

wget --tries=2 --waitretry=1 --timeout=1 http://127.0.0.1:8080/delay/5 -O /dev/null 2>&1 | grep -E "Retrying|Giving up"

--waitretry backs off between attempts instead of hammering a server that is already struggling, waiting one second longer each time up to the value given.

Show output 
Retrying.
Giving up.

Checking a URL without downloading it

Check that a URL exists

wget --spider -nv http://127.0.0.1:8080/page.html 2>&1 | tail -2

--spider makes the request but writes no file, the closest thing wget has to curl -I.

Show output

Your output will differ: the timestamp is from the run that produced this

2026-08-16 12:14:07 URL: http://127.0.0.1:8080/page.html 200 OK

Check a URL that isn't there

wget --spider -nv http://127.0.0.1:8080/missing.html 2>&1 | tail -2

A missing page reports the status and exits non-zero, which is what makes --spider usable as a link check.

Show output 
http://127.0.0.1:8080/missing.html:
Remote file does not exist -- broken link!!!

Show the server's response headers

wget -S --spider http://127.0.0.1:8080/page.html 2>&1 | grep -E "HTTP/|Content-Type|Content-Length"

-S (--server-response) prints the response headers. Combined with --spider it is a request that costs nothing but the round trip.

Show output 
  HTTP/1.0 200 OK
  Content-Type: text/html
  Content-Length: 120

Read the exit code after a failure

wget -q --spider http://127.0.0.1:8080/missing.html
echo "exit=$?"

wget exits 8 when the server answered with an error status, as opposed to 4 for a network problem and 1 for a generic failure.

Show output 
exit=8

Headers, authentication, and sending data

Send a custom header

wget -q -O - --header="X-Test: yes" http://127.0.0.1:8080/headers

--header adds one header per use, repeat it for several. The server here echoes back what it received.

Show output 
{
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "identity",
    "Host": "127.0.0.1:8080",
    "User-Agent": "Wget/1.25.0",
    "X-Test": "yes"
  }
}

Set the User-Agent

wget -q -O - --user-agent="MyScript/1.0" http://127.0.0.1:8080/user-agent

--user-agent replaces the default, which names wget and the version installed, and which some servers refuse outright.

Show output 
{
  "user-agent": "MyScript/1.0"
}

Authenticate with HTTP basic auth

wget -q -O - --user=user --password=pass http://127.0.0.1:8080/basic-auth/user/pass

--user and --password cover HTTP basic auth. Both end up in the process list, so prefer --ask-password or a .netrc file for anything real.

Show output 
{
  "authenticated": true,
  "user": "user"
}

See what happens without credentials

wget -nv --tries=1 http://127.0.0.1:8080/basic-auth/user/pass 2>&1 | tail -2

A protected URL answers 401 and wget stops rather than prompting, which is why an unattended script fails here rather than hanging.

Show output 
Username/Password Authentication Failed.

POST form data

wget -q -O - --post-data="name=deb1" http://127.0.0.1:8080/post

--post-data switches the method to POST and sends the string as application/x-www-form-urlencoded.

Show output 
{
  "args": {},
  "data": "",
  "form": {
    "name": "deb1"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "identity",
    "Content-Length": "9",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "127.0.0.1:8080",
    "User-Agent": "Wget/1.25.0"
  },
  "json": null,
  "url": "http://127.0.0.1:8080/post"
}

POST the contents of a file

wget -q -O - --post-file=body.json --header="Content-Type: application/json" http://127.0.0.1:8080/post

--post-file reads the body from disk. wget does not set a content type for you, so pair it with --header when the server cares.

Show output 
{
  "args": {},
  "data": "{\"name\": \"deb1\", \"role\": \"backup\"}\n",
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "identity",
    "Content-Length": "35",
    "Content-Type": "application/json",
    "Host": "127.0.0.1:8080",
    "User-Agent": "Wget/1.25.0"
  },
  "json": {
    "name": "deb1",
    "role": "backup"
  },
  "url": "http://127.0.0.1:8080/post"
}

Use a method other than GET or POST

wget -q -O - --method=PUT --body-data="status=updated" http://127.0.0.1:8080/put

--method sets the verb and --body-data supplies the body, the pair that covers PUT, DELETE and PATCH.

Show output 
{
  "args": {},
  "data": "",
  "form": {
    "status": "updated"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "identity",
    "Content-Length": "14",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "127.0.0.1:8080",
    "User-Agent": "Wget/1.25.0"
  },
  "json": null,
  "url": "http://127.0.0.1:8080/put"
}

Save cookies a server sets

wget -q --save-cookies cookies.txt --keep-session-cookies -O /dev/null http://127.0.0.1:8080/cookies/set/session/abc123
cat cookies.txt

--save-cookies writes a Netscape-format jar. Without --keep-session-cookies a session cookie is dropped on exit and the file comes back empty.

Show output

Your output will differ: the timestamp in the jar's header comment is when wget wrote it

# HTTP Cookie File
# Generated by Wget on 2026-08-16 12:14:07.
# Edit at your own risk.

127.0.0.1:8080	FALSE	/	FALSE	0	session	abc123

Send saved cookies with a later request

wget -q --save-cookies cookies.txt --keep-session-cookies -O /dev/null http://127.0.0.1:8080/cookies/set/session/abc123
wget -q -O - --load-cookies cookies.txt http://127.0.0.1:8080/cookies

--load-cookies replays the jar, which is how a scripted download gets past a login that sets a session cookie.

Show output 
{
  "cookies": {
    "session": "abc123"
  }
}

Redirects

Redirects are followed automatically

wget -nv http://127.0.0.1:8080/redirect/1 -O /dev/null 2>&1 | tail -2

Unlike curl, wget follows redirects without being asked, and reports each hop as it goes.

Show output

Your output will differ: the timestamps and transfer rate are from the run that produced this

2026-08-16 12:14:07 URL:http://127.0.0.1:8080/get [193/193] -> "/dev/null" [1]

Refuse to follow redirects

wget --max-redirect=0 -nv http://127.0.0.1:8080/redirect/1 -O /dev/null 2>&1 | tail -3

--max-redirect=0 turns a redirect into an error, useful when you want to know that a URL moved rather than silently following it.

Show output 
0 redirections exceeded.

Cap how many hops to follow

wget --max-redirect=1 -nv http://127.0.0.1:8080/redirect/3 -O /dev/null 2>&1 | tail -3

A redirect chain longer than the cap fails instead of looping, the protection against a server that redirects to itself.

Show output 
1 redirections exceeded.

Recursive download and mirroring

wget's distinctive feature, and the one most worth putting limits on.

Download a page and everything it links to

wget -q -r http://127.0.0.1:8080/site/ && find 127.0.0.1:8080 -type f | sort

-r (--recursive) follows links from the starting page. Files land under a directory named after the host, which is rarely what you want on its own.

Show output 
127.0.0.1:8080/robots.txt
127.0.0.1:8080/site/a.html
127.0.0.1:8080/site/img/logo.png
127.0.0.1:8080/site/index.html
127.0.0.1:8080/site/sub/deep.html
127.0.0.1:8080/site/sub/deeper.html
127.0.0.1:8080/top.html

Drop the hostname directory

wget -q -r -nH http://127.0.0.1:8080/site/ && find . -name "*.html" | sort

-nH (--no-host-directories) writes straight into the current directory instead of nesting everything under the hostname.

Show output 
./site/a.html
./site/index.html
./site/sub/deep.html
./site/sub/deeper.html
./top.html

Limit how deep the recursion goes

wget -q -r -nH -l 1 http://127.0.0.1:8080/site/ && find site -type f | sort

-l (--level) caps the depth. -l 1 fetches the starting page and what it links to directly, stopping before sub/deeper.html, which is one link further on.

Show output 
site/a.html
site/img/logo.png
site/index.html
site/sub/deep.html

Don't climb above the starting directory

wget -q -r -nH -np http://127.0.0.1:8080/site/ && find . -name "*.html" | sort

-np (--no-parent) refuses to follow a link that points at a parent directory, which is what stops a download of one section pulling in the whole site.

Show output 
./site/a.html
./site/index.html
./site/sub/deep.html
./site/sub/deeper.html

Download only certain file types

wget -q -r -nH -A html http://127.0.0.1:8080/site/ && find site -type f | sort

-A (--accept) keeps only files matching the given suffixes or patterns. wget still fetches pages to find links, then deletes the ones that don't match.

Show output 
site/a.html
site/index.html
site/sub/deep.html
site/sub/deeper.html

Skip certain file types

wget -q -r -nH -R png http://127.0.0.1:8080/site/ && find site -type f | sort

-R (--reject) is the inverse of -A, for skipping images or archives when you only want the text.

Show output 
site/a.html
site/index.html
site/sub/deep.html
site/sub/deeper.html

Rewrite links so the copy works offline

wget -q -r -nH -k http://127.0.0.1:8080/site/
grep -o 'href="[^"]*"' site/index.html

-k (--convert-links) rewrites each link to point at the downloaded copy where one exists, and to the absolute original where it doesn't.

Show output 
href="a.html"
href="http://127.0.0.1:8080/site/b.html"
href="img/logo.png"
href="sub/deep.html"
href="../top.html"
href="https://example.com/"

Mirror a site

wget -q -m -nH http://127.0.0.1:8080/site/ && find site -type f | sort

-m (--mirror) is shorthand for -r -N -l inf --no-remove-listing: recurse without a depth limit and only re-fetch what changed.

Show output 
site/a.html
site/img/logo.png
site/index.html
site/sub/deep.html
site/sub/deeper.html

Ignore robots.txt

wget -q -r -nH -e robots=off http://127.0.0.1:8080/site/ && find site -type f | sort

wget obeys robots.txt by default. -e robots=off overrides that, which is reasonable on a server you own and rude on one you don't.

Show output 
site/a.html
site/b.html
site/img/logo.png
site/index.html
site/sub/deep.html
site/sub/deeper.html

Timestamps and repeat downloads

Only download if the remote copy is newer

wget -q http://127.0.0.1:8080/site/a.html
wget -N http://127.0.0.1:8080/site/a.html 2>&1 | tail -2

-N (--timestamping) compares the local file's date against the server's Last-Modified and skips the transfer when there is nothing new.

Show output 
File 'a.html' not modified on server. Omitting download.

See the timestamp the server reports

wget -q http://127.0.0.1:8080/site/a.html
ls -l --time-style=long-iso a.html | awk '{print $6, $7, $8}'

With -N or a plain download, wget sets the local file's modification time from the server's Last-Modified header rather than the time of the download.

Show output 
2026-07-05 15:35 a.html

Batch downloads and logging

Download every URL listed in a file

wget -q -i urls.txt && ls *.html

-i (--input-file) reads one URL per line, the simplest way to hand wget a queue.

Show output 
a.html
b.html

Read the URL list from a pipe

printf 'http://127.0.0.1:8080/site/a.html\n' | wget -q -i - && ls a.html

-i - reads the list from stdin, so the URLs can be generated by whatever produced them rather than written to a file first.

Show output 
a.html

Write the report to a log file

wget -o wget.log http://127.0.0.1:8080/page.html
grep -c . wget.log

-o (lowercase) redirects the progress report to a file. Note the case: -O names the downloaded file, -o names the log.

Show output 
7

Append to an existing log instead of replacing it

wget -nv -a wget.log http://127.0.0.1:8080/page.html
wget -nv -a wget.log http://127.0.0.1:8080/site/a.html
grep -c "URL:" wget.log

-a (--append-output) keeps the previous contents where -o would truncate them, which is what a nightly download script wants. Pair it with -nv, not -q: -q silences the report the log is there to collect, so the file stays empty.

Show output 
2

Practical patterns

Unpack an archive without saving it first

wget -q -O - http://127.0.0.1:8080/large.bin | wc -c

Piping -O - into another command avoids a temporary file entirely. The same shape works with tar: wget -O - URL | tar -xz.

Show output 
65000

Fail a script when the download fails

set -e
wget -q http://127.0.0.1:8080/page.html
echo "download succeeded"

wget exits non-zero on failure, so set -e stops the script rather than carrying on with a missing or half-written file.

Show output 
download succeeded

Verify a download against a checksum

wget -q http://127.0.0.1:8080/large.bin
sha256sum large.bin

Downloading a project's published SHA256SUMS alongside the file and running sha256sum -c is the standard way to confirm you got what the publisher shipped.

Show output 
5a83a98fc4047337cacd8fc2bb1f0905aabaa181942d96e03661f3ac75819e00  large.bin

Download several files at once

wget -q http://127.0.0.1:8080/site/a.html &
wget -q http://127.0.0.1:8080/site/b.html &
wait
ls *.html

wget has no parallel mode of its own. Backgrounding each call and waiting is enough for a handful of files; use xargs -P for a long list.

Show output 
a.html
b.html