cURL is awesome tool and it comes by default installed most of the distributions. If you don’t have then you can simply installed it using yum command or apt-get command.
cURL supports various protocols such as, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP.
In this article I will provide you 10 practical cURL usage examples:
1. Download a Single File
Use the below command to get the content of the URL and display it in the STDOUT on your terminal.
# curl http://www.looklinux.com
If you want to store the output in a file you can redirect it as shown below. It will also display some additional information regarding download statistics.
# curl http://www.looklinux.com > looklinux.html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 39514 0 39514 0 0 60002 0 --:--:-- --:--:-- --:--:-- 72636
2. Save the cURL Output In a File
You can also save the curl output to a file using -o/-O options.
–o (Lowercase) the output will be saved in the filename provided in the command line.
–O (Uppercase) the filename in the URL will be taken and it will be used as the filename to store the output.
# curl -o looklinux.html http://www.looklinux.com/robots.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 102 3689 0 3689 0 0 443k 0 --:--:-- --:--:-- --:--:-- 514k
Now the page robots.txt saved in the file named ” looklinux.html “. You can also see the progress meter for the download.
If you use curl -O (Uppercase) it will save the content in the file named ‘gettext.html’ itself in the local machine.
# curl -O http://www.looklinux.com/robots.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 102 3689 0 3689 0 0 594k 0 --:--:-- --:--:-- --:--:-- 720k
You can also use wget to download files. Refer to wget examples.
3. Download Multiple Files at a Time
You can download multiple files in a single shot by specifying the URLs on the command line.
# curl -O http://www.gnu.org/software/gettext/manual/html_node/index.html -O http://www.gnu.org/software/gettext/manual/gettext.html
The above command will download both index.html and gettext.html and save it in the same name under the current directory.
Note: When you download multiple files from a same server, curl will try to re-use the connection.
4. Follow HTTP Location Headers
Curl does’t not follow the HTTP location headers by default. It is also termed as redirects. When a requested web page is moved to another place, then an HTTP Location header will be sent as a Response and it will have where the actual web page is located.
For example, when someone types google.com in the browser from India, it will be automatically redirected ‘google.co.in’. This is done based on the HTTP Location header.
# curl http://www.google.com <TITLE>302 Moved</TITLE> <H1>302 Moved</H1> The document has moved <A HREF="http://www.google.co.in/">here</A>
Above you can see that the requested document is moved to “http://www.google.co.in/”.
5. Resume Previous Download
You can continue or resume download using curl -C option which was stopped already for some reason. This will be helpful if you download large files and the download got interrupted.
For example, now start downloading big file using curl and press Ctrl+C to stop it in between the download.
# curl -O http://www.gnu.org/data/download/manual/myfile.html ############## 20.1%
# displayed a progress bar instead of a progress meter.
Above download was stopped at 20.1%. We can continue the download the using curl -C. Download continues from 20.1%.
# curl -C - -O http://www.gnu.org/data/download/manual/myfile.html ############### 21.1%
6. Set Limit The Data Transfer Rate
You can set the limit the amount at which the data gets transferred using –limit-rate option.
# curl --limit-rate 1000B -O http://www.gnu.org/data/download/manual/myfile.html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 1 1215k 1 13601 0 0 957 0 0:21:40 0:00:14 0:21:26 999 1 1215k 1 14601 0 0 960 0 0:21:36 0:00:15 0:21:21 999 1 1215k 1 15601 0 0 962 0 0:21:34 0:00:16 0:21:18 999
Above is the progress meter. You can see that the current speed is near to the 1000 bytes.
7. Download File If Modified Before/After The Given Time
Using -z option you can download the file that are modified after a particular time in curl. It works for both HTTP and FTP.
# curl -z 20-Jul-17 http://looklinux.com/xy.html
The above command will download the yy.html only if it is modified later than the given date and time
# curl -z -20-Jul-17 http://looklinux.com/xy.html
The above command will download the yy.html, if it is modified before than the given date and time.
8. Pass HTTP Authentication in Curl
If any websites require a username and password to view the content you can use curl -u option to get its content. You can pass credentials from CURL to the web server like below:
# curl -u username:password URL
9. Download File From FTP Server
You can also download file from ftp servers. If FTP path is a directory, by default it will list the files under the specific directory.
# curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xyz.html
Above command will download the xyz.html file from the ftp server and save it in the locat directory.
# curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/
Above url will list all the files and directories under the given URL
10. Use Proxy To Download A File
You can specify CURL to use proxy to do the specific operation using -x option. You will need hostname and port of the proxy.
# curl -x proxysever.test.com:3128 http://google.co.in