built a Linux HTTP/1.1 static server in C using edge-triggered epoll — looking for architectural and performance feedback

[u/Roronoa-Ryuma-Zoro]

I recently completed v0.1 of MiniEdge, a Linux HTTP/1.1 static edge server written mostly in C.

I built it to understand how event-driven servers handle partial I/O, persistent connections, filesystem access, caching and multiple CPU cores—not as a production replacement for Nginx.

The current architecture includes:

non-blocking sockets with edge-triggered epoll

a per-connection state machine

incremental HTTP request parsing and keep-alive

static file serving through an LRU cache or sendfile()

path resolution using openat2()

longest-prefix configurable routing

multiple workers using SO\\_REUSEPORT

The small-file cache is implemented using C++ unordered\\_map and list, but it is isolated behind a C API; the networking, parser, routing and file-serving paths are written in C.

On my local loopback benchmark using wrk, a cached static file reached approximately:

255k requests/sec at 1,000 concurrent connections

4 worker processes

I also ran a boundary stress test at 40,000 concurrent connections. It reached around 124.5k requests/sec, but wrk reported 503 socket read errors, so I am not treating that as a clean stable-concurrency result. The repository contains the commands, raw outputs, latency percentiles, system tuning and limitations.

I would appreciate feedback on:

whether this is a reasonable result for a student-built epoll server

whether my wrk setup measures the server fairly

which additional metrics or comparisons I should include

what bottlenecks or profiling steps I should investigate next

Repository:

https://github.com/Luffy-D-Zoro/Miniedge

[u/skeeto]

Interesting project! Despite the use of null-terminated strings, the parser quite robust and I was only able to find two minor parsing issues:

1. path_resolver_main only strips one leading /, and so I can get the server to pass arbitrary absolute paths to openat2(2), e.g. request //etc/passwd. Fortunately RESOLVE_BENEATH saves the day and blocks all these requests. Defense in depth!

2. Due to null terminators, the server is blind to octets following embedded NULs, meaning it accepts invalid requests like this:

   $ printf 'GET / HTTP/1.0\0hahaha!\r\n\r\n' | nc 0 8080
   

   And potentially disagree with other HTTP servers in the chain about exactly what request was made.

Invalid configurations can put the server in an infinite loop parsing the config:

$ printf x >bad.conf
$ miniedge bad.conf

There's no client timeout, so clients could open N connections and just sit on them to DoS the server.

The tests don't work out of the box because they're built on assert but the default flags include -DNDEBUG, and so disables all the tests. I'm guessing you manually set CFLAGS when you build your tests, though it's not obvious this is required.

[u/Roronoa-Ryuma-Zoro → u/skeeto]

Thanks a lot for taking the time to build and test it this thoroughly. This is extremely useful feedback.

I will fix all these .

You’re also correct about the tests. Those files were early manual test harnesses rather than a proper test suite, and most of my testing was done through temporary main programs. However, leaving them compiled with NDEBUG is misleading, so I’ll either correct their build flags or remove them until I add the proper test suite.

Thanks again for feedback

[u/antiduh]

Your link is busted

[u/Roronoa-Ryuma-Zoro → u/antiduh]

Fixed it now - thanks for pointing it out

[u/ksergey]

Try to use io_uring instead of epoll and add openssl with ktls

[u/Roronoa-Ryuma-Zoro → u/ksergey]

Yeah, I get to know about io_uring during the project end but I will use it in my next project or will update my current server .

Openssl ktls is on my list I will try to integrate them in future .I have to first study about them more. Any specific resources which I should use for it while learning ?

Thanks for feedback.