How to Download Files With curl And wget

Both [.inline-code] curl[.inline-code]  and [.inline-code] wget[.inline-code]  are command-line tools used for downloading files from the internet. More specifically, [.inline-code] curl[.inline-code]  is a flexible, feature-rich, and multi-purpose tool designed for user authentication, proxy support, cookie handling, file uploading, and more. [.inline-code] wget[.inline-code] , on the other hand, is a simpler, more straightforward tool primarily designed for downloading files and mirroring websites efficiently.

[#similarities-and-differences] Similarities and Differences [#similarities-and-differences]

• Both tools support the HTTP and FTP protocols, enabling users to download files, web pages, and other resources from web servers.
• Both tools support basic authentication, allowing users to provide usernames and passwords for protected resources.
• Both can be configured to use proxy servers for making requests, useful for scenarios where network traffic needs to be routed through a proxy.
• Both offer mechanisms to resume interrupted downloads.
• Both are available on various operating systems, making them versatile tools that work across different platforms.

Differences

• [.inline-code] curl[.inline-code]  has a more extensive and flexible set of options, making it highly customizable for a variety of tasks, whereas [.inline-code] wget[.inline-code]  is much simpler and more straightforward to use.
• [.inline-code] wget[.inline-code]  saves downloaded contents to local files, whereas [.inline-code] curl[.inline-code]  outputs the content to the terminal by default.
• [.inline-code] wget[.inline-code]  is particularly well-suited for mirroring websites and recursively downloading content, whereas [.inline-code] curl[.inline-code]  doesn't have built-in support for this kind of task and requires more manual configuration.
• [.inline-code] curl[.inline-code]  supports a wide range of protocols, such as HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, etc, whereas [.inline-code] wget[.inline-code]  only supports HTTP, HTTPS, and FTP.
• [.inline-code] wget[.inline-code]  automatically follows redirects, making it convenient for tasks that involve multiple URLs, whereas [.inline-code] curl[.inline-code]  doesn't and requires explicit use of the [.inline-code] -L[.inline-code]  option to do so.
• [.inline-code] curl[.inline-code]  is often preferred for scripting and automation due to its versatility, whereas [.inline-code] wget[.inline-code]  is more user-friendly for basic tasks.

[#download-a-file-with-curl] Downloading a single file with [.inline-code]curl[.inline-code] [#download-a-file-with-curl]

By default, the [.inline-code] curl[.inline-code]  command writes the content of the requested file to the standard output of the terminal.

To physically download the file on your machine in the current directory, you can use the [.inline-code] -O[.inline-code]  flag (short for [.inline-code] --remote-name[.inline-code] ) as follows:

 $ curl -O <url>

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the current directory, under its original name:

 $ curl -O http://example.com/logs/20231003.txt

[#download-and-rename-with-curl] Downloading and renaming files [#download-and-rename-with-curl]

To download a file in the current directory under a different name, you can use the [.inline-code] -o[.inline-code]  flag (short for [.inline-code] --output[.inline-code] ) as follows:

 $ curl -o <filename> <url>

Where:

• [.inline-code] filename[.inline-code]  is the name of the downloaded file on your local machine.

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the current directory, under the [.inline-code] log.txt[.inline-code]  name:

 $ curl -o log.txt http://example.com/logs/20231003.txt

[#download-in-a-directory-with-curl] Downloading in a specific directory [#download-in-a-directory-with-curl]

To download a file in a specific directory instead of the current working directory, you can use the [.inline-code] --output-dir[.inline-code] flag as follows:

 $ curl --output-dir <path> -O <url>

Where:

• [.inline-code] path[.inline-code]  is the relative or absolute path of the directory you want to download the file into.

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the [.inline-code] /tmp/logs[.inline-code]  directory:

 $ curl --output-dir /tmp/logs -O http://example.com/logs/20231003.txt

Note that the [.inline-code] --output-dir[.inline-code]  flag can also be combined with the [.inline-code] -o[.inline-code]  flag.

[#download-a-file-with-wget] Downloading a single file with with [.inline-code] wget[.inline-code] [#download-a-file-with-wget]

By default, the [.inline-code] wget[.inline-code]  command automatically downloads the requested resource into a file on your local machine.

To download a file into the current directory, you can use the [.inline-code] wget[.inline-code] command as follows:

 $ wget <url>

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the current directory, under its original name:

 $ wget http://example.com/logs/20231003.txt

[#download-and-rename-with-wget] Downloading and renaming files [#download-and-rename-with-wget]

To download a file in the current directory under a different name, you can use the [.inline-code] wget[.inline-code]  command with the [.inline-code] -O[.inline-code]  flag (short for [.inline-code] --output-document[.inline-code] ) as follows:

 $ wget -O <filename> <url>

Where:

• [.inline-code] filename[.inline-code]  is the name of the downloaded file on your local machine.

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the current directory, under the [.inline-code] log.txt[.inline-code]  name:

 $ wget -O log.txt http://example.com/logs/20231003.txt

[#download-in-a-directory-with-wget] Downloading in a specific directory [#download-in-a-directory-with-wget]

To download a file in a specific directory instead of the current directory, you can use the [.inline-code] -P[.inline-code]  flag (short for [.inline-code] --directory-prefix[.inline-code] ) as follows:

 $ wget -P <path> <url>

Where:

• [.inline-code] path[.inline-code]  is the relative or absolute path of the directory you want to download the file into.

For example, this command will download and save the [.inline-code] 20231003.txt[.inline-code]  file in the [.inline-code] /tmp/logs[.inline-code]  directory:

 $ wget -P /tmp/logs http://example.com/logs/20231003.txt

[#download-multiple-files-with-curl] Downloading multiple files with [.inline-code] curl[.inline-code] [#download-multiple-files-with-curl]

To download multiple files at once with [.inline-code] curl[.inline-code] , you can repeat the [.inline-code] -o[.inline-code] , [.inline-code] -O[.inline-code] , and [.inline-code] --output-dir[.inline-code]  flags several times, for each URL:

For example, this command will download both of the [.inline-code] 20231003.txt[.inline-code]  and [.inline-code] 20231004.txt[.inline-code]  files in the current directory:

 $ curl \
-O http://example.com/logs/20231003.txt \
-O http://example.com/logs/20231004.txt

And this command will download the same files in the current directory, while respectively renaming them to [.inline-code] logs_03.txt[.inline-code]  and [.inline-code] logs_04.txt[.inline-code] :

 $ curl \
-o logs_03.txt http://example.com/logs/20231003.txt \
-o logs_04.txt http://example.com/logs/20231004.txt

[#download-with-a-config-file-with-curl] Downloading files using a configuration file [#download-with-a-config-file-with-curl]

Alternatively, you can create a configuration file containing the command flags and URLs of the files to download:

 -O
url = "http://example.com/logs/20231003.txt"
-O
url = "http://example.com/logs/20231004.txt"

And use the [.inline-code] curl[.inline-code]  command with the [.inline-code] -K[.inline-code]  flag (short for [.inline-code] --config[.inline-code] ) as follows:

 $ curl -K <file>

Where:

• [.inline-code] file[.inline-code]  is the relative or absolute path to the configuration file.

[#download-multiple-files-with-wget] Downloading multiple files with [.inline-code] wget[.inline-code] [#download-multiple-files-with-wget]

To download multiple files at once with [.inline-code] wget[.inline-code] , you can simply list the URLs as arguments of the command as follows:

 $ wget <url> …

For example, this command will download both of the [.inline-code] 20231003.txt[.inline-code]  and [.inline-code] 20231004.txt[.inline-code]  files at once in the current directory:

 $ wget \
http://example.com/logs/20231003.txt \
http://example.com/logs/20231004.txt

[#download-files-recursively] Downloading files recursively [#download-files-recursively]

The [.inline-code] wget[.inline-code]  command is often used to download an entire web page or website for mirroring, offline viewing, or archiving, including the additional pages, images, styles, scripts and other associated files. It follows the links present in the HTML, XHTML, and CSS files and automatically recreates the directory structure of the target URL.

To download files recursively, you can use the the [.inline-code] wget[.inline-code]  command with the [.inline-code] -r[.inline-code]  flag (short for [.inline-code] --recursive[.inline-code] ) as follows:

 $ wget -r <url>

Note that, by default, the [.inline-code] wget[.inline-code]  command has a maximum recursion depth of 5 levels, which can be changed using the [.inline-code] -l[.inline-code]  (short for [.inline-code] --level[.inline-code] ) flag as follows:

 $ wget -r -l <depth> <url>

Also note that it respects the Robot Exclusion Standard (i.e. [.inline-code] robots.txt[.inline-code] ) and ignores the entries listed in that file.

[#download-files-via-ftp] Downloading files via FTP [#download-files-via-ftp]

To download files via the FTP protocol, you will need to provide a username and password in order to authenticate to the remote server. It ensures that only authorized individuals or systems can access and transfer files, thus protecting data and server resources.

[#download-files-via-ftp-with-curl] Downloading files via FTP with [.inline-code]curl[.inline-code] [#download-files-via-ftp-with-curl]

To download files with [.inline-code] curl[.inline-code]  via the FTP protocol, you can use the [.inline-code] -u[.inline-code]  flag (short for [.inline-code] --user[.inline-code] ) as follows:

 $ curl -u <username>:<password> -O <ul>

Where:

• [.inline-code] username[.inline-code]  is the FTP username.
• [.inline-code] username[.inline-code]  is the FTP password.

For example, this command will download the [.inline-code] logs.txt[.inline-code]  file using the username [.inline-code] admin[.inline-code]  and the password [.inline-code] admin[.inline-code].

 $ curl -u admin:admin -O ftp://ftp.example.com/resources/logs.txt

[#download-files-via-ftp-with-wget] Downloading files via FTP with [.inline-code]wget[.inline-code] [#download-files-via-ftp-with-wget]

To download files with [.inline-code] wget[.inline-code]  via the FTP protocol, you can use the [.inline-code] --ftp-user[.inline-code]  and [.inline-code] --ftp-password[.inline-code]  flags as follows:

 $ wget --ftp-user=<username> --ftp-password=<password> <url>

Where:

• [.inline-code] username[.inline-code]  is the FTP username.
• [.inline-code] password[.inline-code]  is the FTP password.

For example, this command will download the [.inline-code] logs.txt[.inline-code]  file using the username [.inline-code] admin[.inline-code]  and the password [.inline-code] admin[.inline-code].

 $ wget --ftp-user=admin --ftp-password=admin ftp://ftp.example.com/

[#resume-the-download-of-incomplete-files] Resuming the download of incomplete files [#resume-the-download-of-incomplete-files]

Both [.inline-code] curl[.inline-code]  and [.inline-code] wget[.inline-code]  offer the possibility to resume the download of incomplete files whose downloads have been interrupted.

[#resume-the-download-with-curl] Resuming the download with [.inline-code]curl[.inline-code] [#resume-the-download-with-curl]

To resume the download of a file with [.inline-code] wget[.inline-code] , you can use the [.inline-code] -C[.inline-code]  flag (short for [.inline-code] --continue-at[.inline-code] ) as follows:

 $ curl -C <offset> -O <url>

Where:

• [.inline-code] offset[.inline-code]  is either the exact number of bytes to skip, counting from the beginning of the source file, or the [.inline-code] -[.inline-code]  value to let [.inline-code] curl[.inline-code]  automatically determine the offset.

For example, this command will resume the download of the [.inline-code] video.mp4[.inline-code] file automatically:

 $ curl -C - -O http://example.com/video.mp4

And this command will resume the download starting from the file’s byte offset 400:

 $ curl -C 400 -O http://example.com/video.mp4

Note that if the file size is smaller than the [.inline-code] offset[.inline-code] , [.inline-code] curl[.inline-code]  will automatically start the download from the beginning.

[#resume-the-download-with-wget] Resuming the download with [.inline-code]wget[.inline-code] [#resume-the-download-with-wget]

To resume the download of a file with [.inline-code] wget[.inline-code] , you can use the [.inline-code] -c[.inline-code]  flag (short for [.inline-code] --continue[.inline-code] ) as follows:

 $ wget -c <url>

If present in the current directory of the local machine, [.inline-code] wget[.inline-code]  will ask the server to continue the retrieval from an offset equal to the length of the local file.

For example, this command will resume the download of the [.inline-code] video.mp4[.inline-code]  file automatically:

 $ wget -c http://example.com/video.mp4

Note that as of version 1.7, if the remote server does not support continued downloading, [.inline-code] wget[.inline-code]  will refuse to start the download from scratch. You should therefore remove the local file to restart the download from scratch.