Hypertext Transfer Protocol (HTTP) is an application-layer protocol for transmitting hypermedia documents, such as HTML. It was designed for communication between web browsers and web servers, but it can also be used for other purposes. HTTP follows a classical client-server model, with a client opening a connection to make a request, then waiting until it receives a response.
Clients and servers communicate by exchanging individual messages (as opposed to a stream of data). The messages sent by the client, usually a Web browser, are called requests and the messages sent by the server as an answer are called responses. Requests and responses are also called HTTP messages. HTTP makes use of the Uniform Resource Identifier (URI) to identify a given resource and to establish a connection. Once the connection is established, HTTP messages are passed in a format/structure for transferring the required data.
An HTTP client sends an HTTP request to a server in the form of a request message which includes following format:
Zero or more header (General|Request|Entity) fields
An empty line indicating the end of the header fields
Optionally a message-body
Now let’s make a few requests using
curl in terminal as examples.
Send requests with
Curl is a linux utility that is used to make HTTP requests to a given url. It outputs HTTP response to standard output and is actually very easy to use.
GET request example:
This line below sends an HTTP GET request to the url “https://www.google.com".
-Hallows adding extra headers to use when getting a web page. For detailed explanations of other parameters in this command please check out explainshell.
$ ~ curl -s -v -H "Jade: halo world" -- "https://www.google.com"
From all the things we received, these lines are the actual request content we made from curl (note that all the ‘>’ symbols are only for readability use):
> GET / HTTP/2 > Host: www.google.com > User-Agent: curl/7.54.0 > Accept: */* > Jade: halo world >
POST request example:
Here we are making a POST request and the
-dparameter specifies the data we want to post to the server. In this case, we are sending a string ‘meow’ with length of 4.
$ curl -X POST -d "meow" -s -v -H "Jade: halo world" -- "https://www.google.com"
> POST / HTTP/2 > Host: www.google.com > User-Agent: curl/7.54.0 > Accept: */* > Jade: halo world > Content-Length: 4 > Content-Type: application/x-www-form-urlencoded >
HTTP request structure
There are maximum four parts and minimum three parts in an HTTP request message, each of the entities can be used in this structure:
1 Request-Method SPACE Request-URI SPACE HTTP-Version CRLF 2 Key1: value1 2 Key2: value2 2 Key3: value3 2 ... 3 4 Optional data to send
- The 1st part (Request-Line) begins with a method token, followed by the Request-URI and the protocol version, and ending with CRLF. The elements are separated by space SP characters. Methods could be GET, POST, PUT, DELETE, etc. according to the HTTP version.
- The Request-URI can include queries but not #fragment.
- The absolute path cannot be empty; if none is present in the original URI, it will be given as “/” (the server root).
- The request-header fields allow the client to pass additional information about the request, and about the client itself, to the server. Each of the field is written in a key-value pair format.
- Content-Type tells the server that the passed data is a simple web form data and Content-Length will be the actual length of the data put in the optional message body (the 4th part).
After receiving and interpreting a request message, a server responds with an HTTP response message:
Zero or more header (General|Response|Entity)
An empty line indicating the end of the header fields
Optionally a message-body
Receive responses with
Response for the GET request example:
The ‘<’ symbols are also for readability uses.
< HTTP/2 200 < date: Wed, 23 May 2018 16:02:40 GMT < expires: -1 < cache-control: private, max-age=0 < content-type: text/html; charset=ISO-8859-1 < p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info." < server: gws < x-xss-protection: 1; mode=block < x-frame-options: SAMEORIGIN < set-cookie: 1P_JAR=2018-05-23-16; expires=Fri, 22-Jun-2018 16:02:40 GMT; path=/; domain=.google.com < set-cookie: NID=130=WiAnlBdoQAOkGLE5cKKLLmb3bqK3y9mcKglhtQX8B6wPIiK9bIqGu2tWsyBvJySvqhHB40dRJqthccXAo6NUlGZ2FB3vXQF9iyz0gvhKA9tt8Yg-VN5GZ7eZR6JtB07i; expires=Thu, 22-Nov-2018 16:02:40 GMT; path=/; domain=.google.com; HttpOnly < alt-svc: hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35" < accept-ranges: none < vary: Accept-Encoding < <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." ...
Response for the POST request example:
< HTTP/2 405 < allow: GET, HEAD < date: Wed, 23 May 2018 22:50:05 GMT < content-type: text/html; charset=UTF-8 < server: gws < content-length: 1589 < x-xss-protection: 1; mode=block < x-frame-options: SAMEORIGIN < alt-svc: hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35" < <!DOCTYPE html> <html lang=en> <meta charset=utf-8> <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"> <title>Error 405 (Method Not Allowed)!!1</title> ...
HTTP response structure
The response structure is similar to the request one, it also consists of maximum four parts:
1 HTTP-Version SPACE Status-Code SPACE Reason-Phrase CRLF 2 Key1: value1 2 Key2: value2 2 Key3: value3 2 ... 3 4 Optional data to fetch
A Status-Line consists of the protocol version followed by a numeric status code and its associated textual phrase.
The Status-Code element is a 3-digit integer where first digit of the Status-Code defines the class of response and the last two digits do not have any categorization role. There are 5 values for the first digit:
It means the request was received and the process is continuing.
It means the action was successfully received, understood, and accepted.
It means further action must be taken in order to complete the request.
4xx: Client Error
It means the request contains incorrect syntax or cannot be fulfilled. In the second example we get a 405 error since the google server does not allow POST request method.
5xx: Server Error
It means the server failed to fulfill an apparently valid request.
In responses, a Content-Type header tells the client what type of the returned content (the 4th part) actually is. Content-Type follows MIME standards.
Inspect HTTP requests and responses with Chrome DevTools
We can also send an HTTP request to a web server somewhere using Chrome Developer Tools.
- Type in a URL in Chrome’s address bar (I’ll use https://github.com as an example).
- Press Command+Option+I (Mac) or Control+Shift+I (Windows, Linux) to open DevTools on the page.
- In DevTools, click the Network tab.
- Press Command+R (Mac) or Control+R (Windows, Linux) to reload the page. Note that the new stuff happened on this panel is a set of requests that were generated in order to render what you’ve seen on the screen. As your application runs and additional requests are sent to the server, those requests are going to show up here as well.
- Click on the first document listed on the panel (github.com in this case), then select the Headers tab.
Expand the Request Headers list, and click view source (it will turn into ‘view parsed’ after clicking). The first three parts of the request message can be viewed in the Request Headers list. The fourth part (that is the optional message body after the
CRLF) will be shown in FormData or Payload when sending POST or PUT request, the detailed difference can be found here.
Expand the Response Headers list, and click view source. In the list, we have the 1st and 2nd part of the response message.
The 4th part of the response message can be viewed in Preview or Response tab.