1 Online: http://curl.haxx.se/docs/httpscripting.shtml
2 Author: Daniel Stenberg <daniel@haxx.se>
6 The Art Of Scripting HTTP Requests Using Curl
7 =============================================
9 This document will assume that you're familiar with HTML and general
12 The possibility to write scripts is essential to make a good computer
13 system. Unix' capability to be extended by shell scripts and various tools to
14 run various automated commands and scripts is one reason why it has succeeded
17 The increasing amount of applications moving to the web has made "HTTP
18 Scripting" more frequently requested and wanted. To be able to automatically
19 extract information from the web, to fake users, to post or upload data to
20 web servers are all important tasks today.
22 Curl is a command line tool for doing all sorts of URL manipulations and
23 transfers, but this particular document will focus on how to use it when
24 doing HTTP requests for fun and profit. I'll assume that you know how to
25 invoke 'curl --help' or 'curl --manual' to get basic information about it.
27 Curl is not written to do everything for you. It makes the requests, it gets
28 the data, it sends data and it retrieves the information. You probably need
29 to glue everything together using some kind of script language or repeated
34 HTTP is the protocol used to fetch data from web servers. It is a very simple
35 protocol that is built upon TCP/IP. The protocol also allows information to
36 get sent to the server from the client using a few different methods, as will
39 HTTP is plain ASCII text lines being sent by the client to a server to
40 request a particular action, and then the server replies a few text lines
41 before the actual requested content is sent to the client.
43 Using curl's option -v will display what kind of commands curl sends to the
44 server, as well as a few other informational texts. -v is the single most
45 useful option when it comes to debug or even understand the curl<->server
50 The Uniform Resource Locator format is how you specify the address of a
51 particular resource on the Internet. You know these, you've seen URLs like
52 http://curl.haxx.se or https://yourbank.com a million times.
56 The simplest and most common request/operation made using HTTP is to get a
57 URL. The URL could itself refer to a web page, an image or a file. The client
58 issues a GET request to the server and receives the document it asked for.
59 If you issue the command line
61 curl http://curl.haxx.se
63 you get a web page returned in your terminal window. The entire HTML document
66 All HTTP replies contain a set of headers that are normally hidden, use
67 curl's -i option to display them as well as the rest of the document. You can
68 also ask the remote server for ONLY the headers by using the -I option (which
69 will make curl issue a HEAD request).
73 Forms are the general way a web site can present a HTML page with fields for
74 the user to enter data in, and then press some kind of 'OK' or 'submit'
75 button to get that data sent to the server. The server then typically uses
76 the posted data to decide how to act. Like using the entered words to search
77 in a database, or to add the info in a bug track system, display the entered
78 address on a map or using the info as a login-prompt verifying that the user
79 is allowed to see what it is about to see.
81 Of course there has to be some kind of program in the server end to receive
82 the data you send. You cannot just invent something out of the air.
86 A GET-form uses the method GET, as specified in HTML like:
88 <form method="GET" action="junk.cgi">
89 <input type=text name="birthyear">
90 <input type=submit name=press value="OK">
93 In your favorite browser, this form will appear with a text box to fill in
94 and a press-button labeled "OK". If you fill in '1905' and press the OK
95 button, your browser will then create a new URL to get for you. The URL will
96 get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
99 If the original form was seen on the page "www.hotmail.com/when/birth.html",
100 the second page you'll get will become
101 "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
103 Most search engines work this way.
105 To make curl do the GET form post for you, just enter the expected created
108 curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
112 The GET method makes all input field names get displayed in the URL field of
113 your browser. That's generally a good thing when you want to be able to
114 bookmark that page with your given data, but it is an obvious disadvantage
115 if you entered secret information in one of the fields or if there are a
116 large amount of fields creating a very long and unreadable URL.
118 The HTTP protocol then offers the POST method. This way the client sends the
119 data separated from the URL and thus you won't see any of it in the URL
122 The form would look very similar to the previous one:
124 <form method="POST" action="junk.cgi">
125 <input type=text name="birthyear">
126 <input type=submit name=press value=" OK ">
129 And to use curl to post this form with the same data filled in as before, we
132 curl -d "birthyear=1905&press=%20OK%20" www.hotmail.com/when/junk.cgi
134 This kind of POST will use the Content-Type
135 application/x-www-form-urlencoded and is the most widely used POST kind.
137 The data you send to the server MUST already be properly encoded, curl will
138 not do that for you. For example, if you want the data to contain a space,
139 you need to replace that space with %20 etc. Failing to comply with this
140 will most likely cause your data to be received wrongly and messed up.
144 Back in late 1995 they defined a new way to post data over HTTP. It was
145 documented in the RFC 1867, why this method sometimes is referred to as
148 This method is mainly designed to better support file uploads. A form that
149 allows a user to upload a file could be written like this in HTML:
151 <form method="POST" enctype='multipart/form-data' action="upload.cgi">
152 <input type=file name=upload>
153 <input type=submit name=press value="OK">
156 This clearly shows that the Content-Type about to be sent is
159 To post to a form like this with curl, you enter a command line like:
161 curl -F upload=@localfilename -F press=OK [URL]
165 A very common way for HTML based application to pass state information
166 between pages is to add hidden fields to the forms. Hidden fields are
167 already filled in, they aren't displayed to the user and they get passed
168 along just as all the other fields.
170 A similar example form with one visible field, one hidden field and one
171 submit button could look like:
173 <form method="POST" action="foobar.cgi">
174 <input type=text name="birthyear">
175 <input type=hidden name="person" value="daniel">
176 <input type=submit name="press" value="OK">
179 To post this with curl, you won't have to think about if the fields are
180 hidden or not. To curl they're all the same:
182 curl -d "birthyear=1905&press=OK&person=daniel" [URL]
184 4.5 FIGURE OUT WHAT A POST LOOKS LIKE
186 When you're about fill in a form and send to a server by using curl instead
187 of a browser, you're of course very interested in sending a POST exactly the
188 way your browser does.
190 An easy way to get to see this, is to save the HTML page with the form on
191 your local disk, modify the 'method' to a GET, and press the submit button
192 (you could also change the action URL if you want to).
194 You will then clearly see the data get appended to the URL, separated with a
195 '?'-letter as GET forms are supposed to.
199 The perhaps best way to upload data to a HTTP server is to use PUT. Then
200 again, this of course requires that someone put a program or script on the
201 server end that knows how to receive a HTTP PUT stream.
203 Put a file to a HTTP server with curl:
205 curl -T uploadfile www.uploadhttp.com/receive.cgi
209 Authentication is the ability to tell the server your username and password
210 so that it can verify that you're allowed to do the request you're doing. The
211 Basic authentication used in HTTP (which is the type curl uses by default) is
212 *plain* *text* based, which means it sends username and password only
213 slightly obfuscated, but still fully readable by anyone that sniffs on the
214 network between you and the remote server.
216 To tell curl to use a user and password for authentication:
218 curl -u name:password www.secrets.com
220 The site might require a different authentication method (check the headers
221 returned by the server), and then --ntlm, --digest, --negotiate or even
222 --anyauth might be options that suit you.
224 Sometimes your HTTP access is only available through the use of a HTTP
225 proxy. This seems to be especially common at various companies. A HTTP proxy
226 may require its own user and password to allow the client to get through to
227 the Internet. To specify those with curl, run something like:
229 curl -U proxyuser:proxypassword curl.haxx.se
231 If your proxy requires the authentication to be done using the NTLM method,
234 If you use any one these user+password options but leave out the password
235 part, curl will prompt for the password interactively.
237 Do note that when a program is run, its parameters are possible to see when
238 listing the running processes of the system. Thus, other users may be able to
239 watch your passwords if you pass them as plain command line options. There
240 are ways to circumvent this.
244 A HTTP request may include a 'referer' field, which can be used to tell from
245 which URL the client got to this particular resource. Some programs/scripts
246 check the referer field of requests to verify that this wasn't arriving from
247 an external site or an unknown page. While this is a stupid way to check
248 something so easily forged, many scripts still do it. Using curl, you can put
249 anything you want in the referer-field and thus more easily be able to fool
250 the server into serving your request.
252 Use curl to set the referer field with:
254 curl -e http://curl.haxx.se daniel.haxx.se
258 Very similar to the referer field, all HTTP requests may set the User-Agent
259 field. It names what user agent (client) that is being used. Many
260 applications use this information to decide how to display pages. Silly web
261 programmers try to make different pages for users of different browsers to
262 make them look the best possible for their particular browsers. They usually
263 also do different kinds of javascript, vbscript etc.
265 At times, you will see that getting a page with curl will not return the same
266 page that you see when getting the page with your browser. Then you know it
267 is time to set the User Agent field to fool the server into thinking you're
268 one of those browsers.
270 To make curl look like Internet Explorer on a Windows 2000 box:
272 curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
274 Or why not look like you're using Netscape 4.73 on a Linux (PIII) box:
276 curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
280 When a resource is requested from a server, the reply from the server may
281 include a hint about where the browser should go next to find this page, or a
282 new page keeping newly generated output. The header that tells the browser
283 to redirect is Location:.
285 Curl does not follow Location: headers by default, but will simply display
286 such pages in the same manner it display all HTTP replies. It does however
287 feature an option that will make it attempt to follow the Location: pointers.
289 To tell curl to follow a Location:
291 curl -L www.sitethatredirects.com
293 If you use curl to POST to a site that immediately redirects you to another
294 page, you can safely use -L and -d/-F together. Curl will only use POST in
295 the first request, and then revert to GET in the following operations.
299 The way the web browsers do "client side state control" is by using
300 cookies. Cookies are just names with associated contents. The cookies are
301 sent to the client by the server. The server tells the client for what path
302 and host name it wants the cookie sent back, and it also sends an expiration
303 date and a few more properties.
305 When a client communicates with a server with a name and path as previously
306 specified in a received cookie, the client sends back the cookies and their
307 contents to the server, unless of course they are expired.
309 Many applications and servers use this method to connect a series of requests
310 into a single logical session. To be able to use curl in such occasions, we
311 must be able to record and send back cookies the way the web application
312 expects them. The same way browsers deal with them.
314 The simplest way to send a few cookies to the server when getting a page with
315 curl is to add them on the command line like:
317 curl -b "name=Daniel" www.cookiesite.com
319 Cookies are sent as common HTTP headers. This is practical as it allows curl
320 to record cookies simply by recording headers. Record cookies with curl by
321 using the -D option like:
323 curl -D headers_and_cookies www.cookiesite.com
325 (Take note that the -c option described below is a better way to store
328 Curl has a full blown cookie parsing engine built-in that comes to use if you
329 want to reconnect to a server and use cookies that were stored from a
330 previous connection (or handicrafted manually to fool the server into
331 believing you had a previous connection). To use previously stored cookies,
334 curl -b stored_cookies_in_file www.cookiesite.com
336 Curl's "cookie engine" gets enabled when you use the -b option. If you only
337 want curl to understand received cookies, use -b with a file that doesn't
338 exist. Example, if you want to let curl understand cookies from a page and
339 follow a location (and thus possibly send back cookies it received), you can
342 curl -b nada -L www.cookiesite.com
344 Curl has the ability to read and write cookie files that use the same file
345 format that Netscape and Mozilla do. It is a convenient way to share cookies
346 between browsers and automatic scripts. The -b switch automatically detects
347 if a given file is such a cookie file and parses it, and by using the
348 -c/--cookie-jar option you'll make curl write a new cookie file at the end of
351 curl -b cookies.txt -c newcookies.txt www.cookiesite.com
355 There are a few ways to do secure HTTP transfers. The by far most common
356 protocol for doing this is what is generally known as HTTPS, HTTP over
357 SSL. SSL encrypts all the data that is sent and received over the network and
358 thus makes it harder for attackers to spy on sensitive information.
360 SSL (or TLS as the latest version of the standard is called) offers a
361 truckload of advanced features to allow all those encryptions and key
362 infrastructure mechanisms encrypted HTTP requires.
364 Curl supports encrypted fetches thanks to the freely available OpenSSL
365 libraries. To get a page from a HTTPS server, simply run curl like:
367 curl https://that.secure.server.com
371 In the HTTPS world, you use certificates to validate that you are the one
372 you you claim to be, as an addition to normal passwords. Curl supports
373 client-side certificates. All certificates are locked with a PIN-code, why
374 you need to enter the unlock-code before the certificate can be used by
375 curl. The PIN-code can be specified on the command line or if not, entered
376 interactively when curl queries for it. Use a certificate with curl on a
379 curl -E mycert.pem https://that.secure.server.com
381 curl also tries to verify that the server is who it claims to be, by
382 verifying the server's certificate against a CA cert bundle. Failing the
383 verification will cause curl to deny the connection. You must then use -k in
384 case you want to tell curl to ignore that the server can't be verified.
388 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
391 RFC 2396 explains the URL syntax.
393 RFC 2109 defines how cookies are supposed to work.
395 RFC 1867 defines the HTTP post upload format.
397 http://www.openssl.org is the home of the OpenSSL project
399 http://curl.haxx.se is the home of the cURL project