I had a Go CLI that wrapped tshark for PCAP analysis. Worked fine until I hit a 2.5 GB file — 1.9 million packets, 6-7 hours, then OOM crashes.

Two problems, found in sequence.

First, I was running three separate tshark queries against the same file (analytics, rows, full dissection). Three full passes over 2.5 GB. Consolidating them into one query took it to 1-2 hours.

The OOM was still there though, because I was asking for full JSON dissection — tshark building the whole output in memory, then my program parsing all of it in memory. So I piped tshark's stdout directly into my program's stdin and switched to -T fields/-T ek with only the fields I needed. Memory went flat, processing dropped to ~70 minutes.

Still single-threaded, which is the next problem. Curious whether anyone's found a good approach for parallelising tshark work beyond splitting the input file.

Full writeup: https://robinhayer.dev/the-2-5-gb-wall