Projectscurl
A command-line HTTP client like curl, written directly on a TCP connection: it takes a URL, writes an HTTP/1.1 request by hand, reads the response off the connection and supports GET, DELETE, POST and PUT. Building it shows what every HTTP library hides: how a URL becomes an address and a request line, how a client knows where a response ends, what each method promises, and why a tool keeps its data and its diagnostics apart.
From Coding Challenges #40 ↗ by John Crickett
No sign-up. You tell the mentor your language: Java, Python or Node.js.
What you'll learn · 7 concepts
In the order the steps use them. Each one links to the best places to learn it, and you explain it in your own words once its steps are built.
Step 1 turns the URL into the address to connect to and the request line to send, and a port written in the URL must win over the default.
Covers: the scheme and the protocol it names · the host, a name or an address, with IPv6 in brackets · the port and each scheme's default · the path and query as the request target · the fragment stays with the client · the host and port in the Host header
What is a URL? (MDN) ↗ · RFC 3986: Uniform Resource Identifier (URI): Generic Syntax ↗ · RFC 9112, section 3.2: Request Target ↗
Step 1 builds the request line and headers, and step 5 adds user headers and a body that the server can only read correctly when Content-Length is right.
Covers: the request line · the Host header · default headers and headers the user replaces · CRLF line endings · the empty line that ends the headers · a body and its Content-Length in bytes · Connection close
RFC 9112: HTTP/1.1 ↗ · HTTP messages (MDN) ↗ · RFC 9110, section 8.6: Content-Length ↗
Step 2 reads until the server closes the connection, and step 3 must split the head from the body to show only the body by default. Keeping the connection open, in the going further step, means finding the end from the headers alone.
Covers: reading in a loop into a buffer · the status line and its code · headers up to the first empty line · responses that never have a body · chunked transfer coding · Content-Length · reading until the server closes · the body as bytes, not text
RFC 9112, section 6.3: Message Body Length ↗ · Transfer-Encoding (MDN) ↗ · Connection management in HTTP/1.x (MDN) ↗
Step 2 resolves the host from the URL and connects to its port before sending anything, and a local test server is often reachable on only one of the addresses its name gives.
Covers: name resolution and the hosts file · several addresses, IPv4 and IPv6 · trying each address in turn · connection refused versus a timeout · the client's temporary port · writing until every byte is sent
Beej's Guide to Network Programming ↗ · RFC 8305: Happy Eyeballs Version 2 ↗ · getaddrinfo(3), Linux manual page ↗
Step 4 sends a DELETE, and step 6 sends a PUT, which carries a body just like POST but promises something different. The going further step adds HEAD and PATCH.
Covers: GET reads · DELETE removes · POST hands the body to the server to process · PUT replaces the target with the body · safe and idempotent methods · which requests carry a body · what a client may retry · HEAD and PATCH
RFC 9110, section 9: Methods ↗ · HTTP request methods (MDN) ↗ · Idempotent (MDN Glossary) ↗
Step 3 shows only the body by default and adds the request and response lines with -v, which forces a decision about where each kind of output goes.
Covers: the body alone on standard output · verbose and error messages on standard error · marking sent and received lines · exit codes for network failures · an HTTP error status is still a completed transfer
curl man page ↗ · stdin(3), Linux manual page ↗ · Bash manual: Exit Status ↗
The going further step suggests HTTPS, which wraps everything the earlier steps write and read in TLS and changes the default port.
Covers: the https scheme and port 443 · the handshake after the TCP connection · certificate chains and trusted authorities · checking the name on the certificate · sending the host name in the handshake
RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3 ↗ · The Illustrated TLS 1.3 Connection ↗ · Transport Layer Security (MDN) ↗
What you'll build · 7 steps
Steps of 30 to 60 minutes, in your own editor and language. Each one has a check, so you know when it works.
Create the project for a command-line program that takes a URL. Choose an HTTP server to test against: the page uses a public service that echoes each request back as JSON, and any server you can reach by a URL works too, including one on your own machine on a port of your choice.
Works whenThe real curl gets an answer from the server you'll test against, so you have its output to compare with yours.
Take a URL from the command line, check that its scheme is http, and pull out the host, the port, which is 80 when the URL gives none, and the path. Print the host you would connect to and the GET request you would send: the request line, a Host header and an Accept header of */*. Don't send anything yet.
Works whenThe echo service's /get URL prints a request line for /get with the right Host header, and the same URL with :80 written in it prints exactly the same request. A URL with another port names that port as the one to connect to.
Concepts: URLs · writing an HTTP/1.1 request
Connect to the host and port from the URL and send the request from step 1, adding a Connection: close header and ending the headers with an empty line. Read until the server closes the connection, and print everything it sent: the status line, the headers and the body.
Works whenThe echo service's /get URL gives a 200 status line, then the headers, then a JSON body listing the URL and the headers you sent. A server on your own machine, on a port other than 80, works the same way.
Concepts: from host name to connection · reading a response from a stream
By default print only the body. With -v, also show each line of the request marked with > and the status line and headers received marked with <, kept on a separate stream from the body, as the real curl does.
Works whenWithout -v the output is the body alone. With -v the request and response headers appear with their markers, and saving or piping the program's standard output still gives only the body.
Concepts: data and diagnostics · reading a response from a stream
Add -X to choose the method, and use it to send a DELETE.
Works whenA DELETE to the echo service's /delete URL gets a 200 and a body whose URL ends in /delete, and with -v the request line starts with DELETE.
Concepts: what a method promises
Add -d to send a string as the request body and -H to add a header such as Content-Type. Send the body after the empty line that ends the headers, with a Content-Length header giving its size in bytes.
Works whenPOSTing a small JSON object with a JSON Content-Type to the echo service's /post URL comes back with your body both as raw data and as parsed JSON, your Content-Type header, and a Content-Length equal to the body's size in bytes.
Concepts: writing an HTTP/1.1 request
Support PUT, which sends a body in exactly the same way as POST.
Works whenPUTting a JSON body to the echo service's /put URL echoes it back with the right Content-Length, just as POST did.
Concepts: what a method promises
Add HEAD and PATCH. Send several requests over one connection by leaving out Connection: close, which means finding the end of each response from its headers instead of waiting for the server to close. Support https URLs by running the same exchange inside TLS.
Works whenA HEAD request prints the status and headers and returns at once without waiting for a body. Two requests sent over one connection both get complete responses. An https URL gives the same answer as its http form, and a site with an invalid certificate is refused.
Concepts: reading a response from a stream · what a method promises · HTTPS