building a web server in aarch64 assembly to give my life (a lack of) meaning
ymawky is a small, static http web server written entirely in aarch64 assembly for macos. it uses raw darwin syscalls with no libc wrappers, serves static files, supports GET, HEAD, PUT, OPTIONS, DELETE, byte ranges, directory listing, custom error pages, and tries to be as hardened as possible.
why? why not? the dream of the 80s is alive in ymawky. everybody has nginx. having apache makes you a square. so why not strip every single convenience layer that computer science has given us since 1957? i wanted to understand how a web server actually works, something i know little about coming from a low-level/systems background. the risks that come up, the problems that need to be solved, the things you don’t think about when you’re writing python or c.
this (probably) won’t replace nginx, but it is doing something in the most difficult way possible.
constraints
i gave myself some constraints for this project:
aarch64 assembly only
macos/darwin, not linux. only because that’s the system i have right now. sorry linuxheads :(
raw syscalls only: no libc wrappers
static files only
no preexisting parsers
absolutely no external libraries
assembly, my beloved
assembly language is the layer between machine code and other languages. c gets compiled into assembly, which then gets assembled into an executable binary. assembly is essentially human-readable mnemonics that directly correspond to raw executable bytes: mov, add, ldr, str, cmp, among others. svc #0x80 is the human-readable equivalent to the bytes D4 00 10 01 you’ll find in the executable binary.
you get almost no abstractions. you move values around between cpu registers and memory, compare them, jump to different portions of your code, and call the kernel for syscalls. it makes simple things look complicated, but it also makes almost every step the cpu takes visible and under your control. it does exactly what you tell it to, without warnings, and without any help. if it’s behaving incorrectly, it’s because you wrote it incorrectly.
writing a web server in assembly means there are no http libraries. no automatic cleanup. no string types: strings are just regions of memory that hold individual bytes sequentially. a struct as it exists in c doesn’t really exist as a language feature. you have to know the exact offset in bytes between each field, and the total size of the struct, or the cpu will happily read the wrong memory.
raw syscalls
ymawky doesn’t use any libc wrappers, it just uses raw calls to the kernel. take, for example, this snippet of code that opens a file:
mov x16, #5 ; SYS_open syscall number adrp x0, filename@PAGE add x0, x0, filename@PAGEOFF mov x1, #0x0 ; O_RDONLY is just 0x0000 svc #0x80 b.cs open_failed
in darwin, the syscall number goes in the x16 register (in aarch64 linux, it goes in x8). syscall number 5 is open(), which takes a couple arguments: filename and mode. you put each argument in the registers by hand, then call the kernel with svc #0x80.
if open() fails, the carry flag is set. we check that with b.cs open_failed, which means “if the carry flag is set, branch to open_failed”. then we have to write open_failed to do whatever cleanup and response handling is needed.
this happens a lot. assembly doesn’t have “exceptions” or “objects”, it just sets a cpu flag that you have to check and deal with.
general overview
at its most basic, a web server receives a request, processes it, returns a status code, and maybe a file. a lot goes into that “receives a request” bit:
set up sockets with socket(AF_INET, SOCK_STREAM, 0)
configure the socket with setsockopt(serverfd, SOL_SOCKET, SO_REUSEADDR, &buf, sizeof(int))
bind a file descriptor to an address with bind(sockfd, &addr, 16)
listen to the socket for new connections with listen(sockfd, 5)
accept a connection with accept(sockfd, NULL, NULL)
ymawky is a fork-on-request server. that means for each new inbound connection, it calls the fork() syscall. this has some advantages:
memory is not shared between request handlers
it’s easier to understand
it’s easier to write
but it also has some pretty significant disadvantages:
bloat
each process has its own memory space
it fundamentally handles fewer concurrent connections than models like nginx’s event-driven async non-blocking model
with more concurrent connections, the kernel spends more time switching between processes than actually being in the process
did i mention the bloat? and memory consumption?
binding to sockets and listening is the easy part. the real soul-crushing task is processing requests. a lot goes into this:
determining request type: GET, HEAD, OPTIONS, PUT, or DELETE
extracting the requested path
normalizing the path, like decoding %20 into a space
performing safety checks on the path
parsing header fields the client sent over
getting information about the requested file
figuring out whether it is a directory or a regular file
writing upload bodies to temporary files for PUT
building response headers
writing the response, which is somehow not straightforward
closing any open files
handling errors without crashing the server
parsing http by hand
i hate string parsing. especially in assembly. unfortunately, an http request is just a string asking a server to do something, and the server has to understand it.
let’s walk through an example http request:
GET /index.html HTTP/1.0\r\n Range: bytes=1 – 5\r\n\r\n
that first line tells us a lot. it’s a GET request, which means the client would like us to send over index.html. HTTP/1.0 tells the server what version of http the client is using. the \r\n sequence, carriage return plus linefeed, tells the server “that’s the end of this line, please process the next one”. the \r\n\r\n at the end tells the server that’s the end of the header. if we never receive \r\n\r\n, we have to bail with 400 Bad Request.
then there is Range: bytes=3 – 5, which means “from this file, only give me bytes 3 through 5, ignore the rest.” if a file is 500gb large, but you only request bytes 3 through 5, you only receive 3 bytes back. yay! unfortunately for me, i have to process that header. boo!
first, ymawky determines the request type by comparing the first few bytes against every method it supports, then it extracts the path. we scan along the header one byte at a time until we find a / or *. but we can’t assume every / is the requested path. if somebody sends:
GET HTTP/1.0\r\n \r\n
there is a / in HTTP/1.0. once we hit a /, we check that the previous byte was a space. if it wasn’t, we reply with 400 Bad Request.
once we find the path, we need somewhere to store it. on most systems, PATH_MAX is 4096 bytes, so ymawky has a 4096 byte filename buffer plus one byte for the null terminator:
.bss filename_buffer: .skip 4097 .align 3
copying the filename is just a loop, but the loop has to constantly check both sides: don’t read past the header, and don’t write past the filename buffer. if the client requests GET /aa…[5000 A]…a HTTP/1.0, they should get 414 URI Too Long rather than overwriting 5KB of arbitrary memory.
in python, this is something like:
text.split(“GET /“)[1].split(” “)[0]
in assembly, it’s ~200 lines long, including ensuring HTTP legality. isn’t assembly the best?
then the path has to be percent-decoded. if the parser sees %, it has to read the next two bytes, verify that they are valid hex characters (0 – 9, a-f, A-F), convert them into the byte they represent, and continue.
GET requests can have a Range: header, and PUT requests require Content-Length:. unlike the requested URL, these can appear at any line in the header. we have to iterate through the header character by character. if we find a \r, we need to check if the next character is \n. if it’s not, it’s a malformed header, and we have to send a 400 Bad Request. likewise, if we find a \n without a preceding \r, it’s also malformed. once we find \r\n, that marks the end of the current line, and the beginning of the next. we check if this new line starts with a space, and send a 400 Bad Request if it does (header fields cannot start with whitespace). then we check for Range: (or Content-Length:, depending on the method), using a little string comparison function:
streqn: ldrb w3, [x0] ldrb w4, [x1] cmp w3, w4 b.ne Lstreqn_no_match
cbz w3, Lstreqn_match ;; both equal and both NULL = end of string = match
;; if we’ve reached the end, it’s a match yeah? subs x2, x2, #1 b.eq Lstreqn_match
add x0, x0, #1 add x1, x1, #1 b streqn
Lstreqn_match: mov x0, #1 ret
Lstreqn_no_match: mov x0, #0 ret
this takes two string pointers, x0 and x1, a max length in x2, and checks if each character is the same.
let’s see what a Range: header can look like:
Range: bytes=10- Range: bytes=-10 Range: bytes=5 – 10
both sides of the range are optional, but at least one of them is required. since “10” is a string and not a literal 10, each side has to be converted from ascii digits into an integer. we have to write an atoi-style function, being careful to check for an integer overflow:
;; x0 -> pointer to string atoi: mov x1, #0 mov x3, #10 mov x4, #0 1: ; if the number is >=19 digits long, it could overflow the 64-bit registers cmp x4, #19 b.hs Latoi_error
ldrb w2, [x0] cbz w2, 2f
cmp w2, #‘0’ b.lo Latoi_error cmp w2, #‘9’ b.hi Latoi_error
; result = (result * 10) + current digit mul x1, x1, x3 sub w2, w2, #‘0’ add x1, x1, x2 add x0, x0, #1 add x4, x4, #1
b 1b 2: cmn xzr, xzr ; clear carry to signal success mov x0, x1 ret
Latoi_error: cmp xzr, xzr ; set carry to signal failure mov x0, #0 ret
in python, that would be int(string). isn’t assembly magical?
put
PUT is interesting. it’s idempotent, meaning the end result on the server is the same regardless of how many times you send the same request. PUT /file.txt will create file.txt, or completely overwrite it if it already exists. putting 1234 to file.txt twice in a row results in one file that contains 1234, not 12341234.
this makes PUT honestly pretty dangerous to have open globally, but hey, who cares?
there are a few things to consider when handling PUT:
what if the process crashes in the middle of handling the request?
what if the client says the Content-Length is 2kb, but only sends 100 bytes?
what if the client says the Content-Length is huge, like 50gb?
that last one is easy to fix. configure a maximum file size. in config.S, MAX_BODY_SIZE is 1gb by default. if Content-Length is larger than that, ymawky refuses the request with 413 Content Too Large. easy peasy.
the first two have the same basic fix. if we blindly opened file.txt and started writing into it, the file could be left half-written if something goes wrong. so instead, ymawky writes to a temporary file:
.ymawky_tmp_<pid>
to get the pid, we use getpid() (syscall #20), then a custom itoa() to convert the number to a string (while checking for buffer overflow, of course). then, the requested content from the client gets written to the temp file. if everything goes smoothly, the temp file is renamed in place, and file.txt now exists on the server. if the client disconnects unexpectedly, times out, or sends a malformed body, the temp file is unlink()’d (syscall #10/syscall #472 for unlinkat()). existing files are only overwritten after a complete request was sent over successfully.
directory listing and more string parsing yay
have you ever noticed sometimes you visit a directory on a website, and it lists all the files with links you can click on? it seems like pretty basic functionality, and it’s not too complicated. but like everything in assembly, you have to do everything by hand.
if you GET /somedir/, we check if directory listing is enabled (ALLOW_DIR_LISTING in config.S). if it’s not, we send a 403 Forbidden and call it a day.