ProjectsWeb Server
A small HTTP/1.1 server written directly on sockets that serves files from a folder, answers many clients at once and refuses to hand out anything outside that folder. Building it shows what every web framework hides: how a request arrives as bytes on a connection, how the server knows where it ends, how a URL safely becomes a file, and how one program keeps many slow clients moving.
From Coding Challenges #11 ↗ by John Crickett
No sign-up. You tell the mentor your language: Java, Python or Node.js.
What you'll learn · 8 concepts
In the order the steps use them. Each one links to the best places to learn it, and you explain it in your own words once its steps are built.
Step 1 builds the accept, read, respond and close cycle by hand, with nothing between your code and the operating system's sockets.
Covers: bind and listen · the listening socket versus connected sockets · the queue of waiting connections · blocking reads and the end of the stream · closing cleanly versus a reset · addresses, ports and privileged ports
Beej's Guide to Network Programming ↗ · Berkeley sockets (Wikipedia) ↗ · accept(2), Linux manual page ↗
Step 1 has to find the request line in raw bytes, know when the request has fully arrived, and send back a response that a client such as curl can parse.
Covers: the request line · header fields and the Host header · CRLF line endings · the empty line that ends the headers · reading until the headers end · Content-Length and the body · persistent connections and Connection close
RFC 9112: HTTP/1.1 ↗ · HTTP messages (MDN) ↗ · Connection management in HTTP/1.x (MDN) ↗
Step 2 answers with either a file or a 404, and a browser only shows the page properly when the response says what it contains. Step 4 adds the choice of how to refuse a request.
Covers: the status line · status code classes · choosing between 400, 403, 404 and 500 · Content-Type and media types · Content-Length in bytes · content sniffing
HTTP response status codes (MDN) ↗ · Media types (MIME types) (MDN) ↗ · RFC 9110, section 15: Status Codes ↗
Step 2 turns request paths into files under www, with / meaning index.html, and step 4 lets the location of that folder be chosen when the server starts.
Covers: request target versus file path · the document root · query strings · percent-decoding · default index files · folders and trailing slashes
Mapping URLs to Filesystem Locations (Apache HTTP Server) ↗ · Percent-encoding (MDN Glossary) ↗ · Serve Static Content (NGINX documentation) ↗
The page asks you to consider the security risk that step 2 creates, and step 4 requires that no request, however it is written, reaches a file outside www.
Covers: dot-dot segments · encoded and double-encoded separators · absolute paths · symbolic links · resolve first, then check · refusing without giving clues
Path traversal (PortSwigger Web Security Academy) ↗ · CWE-22: Improper Limitation of a Pathname to a Restricted Directory (MITRE) ↗ · Path traversal attack (Wikipedia) ↗
Step 3 asks for many clients at once and names threads and asynchronous code as options, each with its own costs.
Covers: blocking calls · a thread per connection · a thread pool and its queue · an event loop and readiness · the cost of each model · shared state and races
The C10K problem (Dan Kegel) ↗ · An overview of parallelism, concurrency, and asynchrony (Coding Challenges) ↗ · Thread pool (Wikipedia) ↗
Once step 3 serves many clients at once, a few slow or hostile ones can still use up every thread or connection unless the server sets its own limits.
Covers: a cap on the size of the headers · a deadline for the whole request · idle connections · a bound on connections or threads · 408 and 431
Slowloris (Wikipedia) ↗ · OWASP Denial of Service Cheat Sheet ↗ · mod_reqtimeout (Apache HTTP Server) ↗
The going further section suggests CGI for dynamic content, which turns a server that hands out files into one that runs programs on behalf of whoever sends a request.
Covers: a process per request · request details in environment variables · the body on standard input · the script's headers and body on standard output · which files may run · the cost of starting a process
RFC 3875: The Common Gateway Interface (CGI) Version 1.1 ↗ · Common Gateway Interface (Wikipedia) ↗ · Shellshock (Wikipedia) ↗
What you'll build · 5 steps
Steps of 30 to 60 minutes, in your own editor and language. Each one has a check, so you know when it works.
Create the project for a long-running server that listens on the port given in the PORT environment variable, instead of the page's port 80, which needs special rights on most systems. Keep a www folder next to it with a small index.html like the page's sample.
Works whenStarted with PORT set to a free port, the program keeps running and accepts a connection on that port.
Handle one connection at a time. Accept a connection, read the request until its headers end, take the method, the path and the version from the first line, and answer with a 200 status line, an empty line and a body saying which path was requested. Then close the connection and wait for the next one.
Works whenA GET for / answers 200 with a body saying the requested path is /, and another path is named back the same way. The server keeps accepting new connections after each one closes.
Concepts: sockets and the life of a connection · the HTTP/1.1 message
Serve files from the www folder. Both / and /index.html return index.html with a 200 status, and a path with no matching file returns 404 with a short not-found message. Think about what else on the disk a request could reach.
Works when/ and /index.html return the same HTML with 200 and an HTML content type, and a request for a file that doesn't exist returns 404. The page's sample output shows 400 here by mistake, while its text asks for 404.
Concepts: mapping a URL path to a file · status codes and the headers a response needs
Let several clients be served at the same time, so that one slow request doesn't hold up the others. Use a thread per connection, a pool of threads or an event loop, and make sure a client that connects and then goes quiet can't hold the server up for ever.
Works whenWith a deliberate delay of a few seconds added before each response, several clients started together all get their answers after about one delay, not one after another, and the server still answers new clients afterwards.
Concepts: serving many clients · limits and timeouts for untrusted clients
Make sure every request resolves to a file inside the www folder or one of its subfolders and nowhere else, however the path is written. Let the location of the folder be chosen when the server starts, and design your own tests for the protection.
Works whenRequests that try to climb out of the folder, with dot-dot segments, encoded dots or slashes, an absolute path or a symbolic link pointing outside, get a 4xx answer and never the contents of a file planted outside the folder, while files in subfolders are still served.
Concepts: path traversal and staying inside the document root · mapping a URL path to a file · status codes and the headers a response needs
Run programs on the server to build pages on demand, following the Common Gateway Interface. For a request that maps to a script, start it with the request's details in its environment and the body on its standard input, and turn what it prints into the response.
Works whenA request for a script in the folder set aside for scripts returns the page that script prints, including the headers it sets, and the query string reaches the script. Files outside that folder are served as plain files and never run.
Concepts: CGI · path traversal and staying inside the document root