Exploring HTTP

I've been taking a closer look at HTTP recently. To that end I've been using a tool I'm familiar with, Curl, and also a new tool that I'd not used before, HTTPie.

The basics

So, the basics of HTTP, you make a request for a resource using a URL, and the server responds with the resource or an error code. Typically you would make the request with some kind of client, such as a web browser, or for a closer look you can do this with Curl's verbose option:

curl -v https://tonys-notebook.com

This shows you the details of the request which looks like:

> GET / HTTP/2
> Host: tonys-notebook.com
> User-Agent: curl/7.54.0
> Accept: */*
>

You'll notice you are using a basic HTTP verb, GET to get the resource indicated by the URL. You also get back a response from the server:

< HTTP/2 200
< date: Thu, 07 Nov 2019 07:54:46 GMT
< content-type: text/html
< content-length: 18331
< vary: Accept-Encoding
< last-modified: Sun, 03 Nov 2019 12:58:57 GMT
< etag: "5dbecf11-479b"
< server: neocities
< strict-transport-security: max-age=16416000; includeSubDomains
< access-control-allow-methods: GET, HEAD, OPTIONS
< access-control-allow-origin: *
< content-security-policy: upgrade-insecure-requests; default-src 'unsafe-inline' 'unsafe-eval' 'self' data: blob: *
< x-neocities-cdn: cdn-lhr
< upgrade-insecure-requests: 1
< x-cached: HIT
< accept-ranges: bytes
<
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />
    ...

There are a couple of interesting things in here, but the most important for now is the HTTP response code, 200 in this case, which means "all good".

The general structure of the request and response is similar, they both have:

Header
Body

Posting data

As well as getting a resource you might want to create a new resource, such as a new user account, a new blog post and so on. To do this the HTTP verb you use is POST. Here's an example again using Curl:

curl -X POST http://localhost:9000/test -d name="Tony" -d age=60

You'll see the data is specified here as key-value pairs. It is effectively JSON data. If you are new to JSON see my JSON article for an introduction.

You can also put the JSON data in a file and POST that. This is extremely when you have complex JSON as escaping JSON on the command line is Not a Fun Thing. Once you have your JSON data in a file you can POST it using:

curl -X POST http://localhost:9000/test -d @data.json

Where data.json contains:

{
  "name": "Tony",
  "age": 60
}

Headers

Curl also lets you specify headers. I will go into headers in more detail in future articles, but for now, an important one for the client to set is one for the type of content it wants to accept. This is done using the Accept header. You also specify the MIME type. This is important because there's no point sending JSON data to a client that expects HTML. That would not be good. To set the header using Curl:

curl -H "Accept: text/html" http://localhost:9000/test

Cookies

You can also use Curl to play with cookies. Cookies are key-value pairs sent from the web server to the client, and stored on the client. They can be longer term cookies stored by the browser (or client), or they can be "session cookies" which expire once the browser is closed.

To receive and store cookies in a cookie jar using Curl:

curl -c cookiejar.txt http://localhost:9000/test

Hello HTTPie

I recently discovered HTTPie and am impressed by it. It has a nice simple syntax, and will return pretty-printed and syntax-coloured responses, which makes exploring responses much more user friendly. To install HTTPie on Mac you can do (I'm assuming you have Homebrew installed):

brew install httpie

To do the basic GET:

http https://tonys-notebook.com

Notice the syntax colouring of the response - very nice!

To get more information use the verbose option:

http -v https://tonys-notebook.com

This shows request and response objects:

GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: tonys-notebook.com
User-Agent: HTTPie/1.0.3



HTTP/1.1 200 OK
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: upgrade-insecure-requests; default-src 'unsafe-inline' 'unsafe-eval' 'self' data: blob: *
Content-Type: text/html
Date: Thu, 07 Nov 2019 09:18:47 GMT
ETag: W/"5dc3d662-48c1"
Last-Modified: Thu, 07 Nov 2019 08:31:30 GMT
Server: neocities
Strict-Transport-Security: max-age=16416000; includeSubDomains
Transfer-Encoding: chunked
Upgrade-Insecure-Requests: 1
Vary: Accept-Encoding
X-Cached: HIT
X-Neocities-CDN: cdn-lhr

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />
    ...

HTTPie lets you control which headers your print. For example:

http -p Hh https://tonys-notebook.com

Would print(p) request header (H) and response header (h).

GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: tonys-notebook.com
User-Agent: HTTPie/1.0.3

HTTP/1.1 200 OK
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: upgrade-insecure-requests; default-src 'unsafe-inline' 'unsafe-eval' 'self' data: blob: *
Content-Type: text/html
Date: Thu, 07 Nov 2019 09:25:24 GMT
ETag: W/"5dc3d662-48c1"
Last-Modified: Thu, 07 Nov 2019 08:31:30 GMT
Server: neocities
Strict-Transport-Security: max-age=16416000; includeSubDomains
Transfer-Encoding: chunked
Upgrade-Insecure-Requests: 1
Vary: Accept-Encoding
X-Cached: HIT
X-Neocities-CDN: cdn-lhr

If you just wanted to see the response header you could do this:

http -p h https://tonys-notebook.com

This allows you to easily control the amount of info on the screen.

Using Curl you can set the content type header and post JSON via a data file like this:

curl -X POST -H "Content-Type: application/json" -d @data.json http://localhost:9000/webhooks/event

With HTTPie you can do this:

http POST http://localhost:9000/webhooks/event < data.json

HTTPie and JWTs

HTTPie supports JWT authentication, but you need to install the plugin:

pip3 install -U httpie-jwt-auth

You can then specify the JWT this way:

http --auth-type=jwt --auth="<token>" example.org -h

It's probably more useful to set the JWT via an env variable:

export JWT_1=secret

So let's say we are using the Nexmo Messages API, and my data.json file contains:

{
  "from": { "type": "sms", "number": "ACME" },
  "to": { "type": "sms", "number": "447724000000" },
  "message": {
    "content": {
      "type": "text",
      "text": "This is an SMS sent from the Messages API via HTTPie! 💩"
    }
  }
}

The HTTPie command to send the message is:

http --auth-type=jwt --auth=$JWT_1 POST https://api.nexmo.com/v0.1/messages < data.json

So, I've just scratched the surface here but I hope you found this quick intro to HTTP and HTTPie useful. Stay tuned for more on this in future articles.

More information

HTTPie

Welcome to Tony's Notebook