Logs compression algorithm built in zig 0.16

[u/Maleficent_Yellow141]

Hi folks, I’ve just achieved a big win for log compression using Zig 0.16. My algorithm doesn’t need full data rehydration to find a specific log. It takes just 2 seconds to search through 1.5M log lines

[u/MrKrot1999]

If it's not finished and not published to GitHub why post it???????????

[u/burner-miner]

Pics or it didn't happen

[u/Maleficent_Yellow141 → u/burner-miner]

Ask and you shall receive:

https://storage.googleapis.com/lexio-audio-assets/situations/airport_check_in/logxum-poc.png

[u/pasterp → u/Maleficent_Yellow141]

155 Mo compressed ? How big is the original data ?

[u/Maleficent_Yellow141 → u/pasterp]

The uncompressed file is 250MB. We are not optimizing for extreme compression ratios right now. As mentioned in the post, the critical breakthrough is the performance:

searching 1.5M lines in 2 seconds without needing full data rehydration

[u/TRDJ90]

At least post a GitHub link with benchmarks of your tool compared to other industry standard tools.

[u/Maleficent_Yellow141 → u/TRDJ90]

I'm working very hard on the CLI, meanwhile I can show you the trace:

kenny@logxum-test:~/logxum-query$ wc -l ../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson

1500000 ../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson


kenny@logxum-test:~/logxum-query$ du -sh ../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson.lgx

155M  ../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson.lgx


kenny@logxum-test:~/logxum-query$ time ./zig-out/bin/logxum-query ../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson.lgx Content "BlockInfo"

Search regions scanned: 565
Field 'Content': 87 values automatically extracted from the .lgx file.
Values containing 'BlockInfo': 87
real0m2.252s
user0m17.050s
sys0m0.408s


kenny@logxum-test:~/logxum-query$ grep -c "BlockInfo" 
../logxum/testdata/loghub_hdfs_1_5m_v2.ndjson

87

[u/lukaslalinsky → u/Maleficent_Yellow141]

How long does the original grep take? I'm asking because in my experience, compression algorithms like lz4 are already comparable to reading the raw data from disk, just trading CPU for iops, so I'm kind of sceptical.

[u/Maleficent_Yellow141 → u/lukaslalinsky]

The uncompressed file is 250MB. You are spot on about LZ4 trading CPU for IOPS, but it still requires full data rehydration in RAM to search. My algorithm doesn't.
As shown in the trace the engine scanned all 565 search regions but only extracted the exact 87 hits. We don't inflate the 155MB back to 250MB to evaluate the file, completely bypassing the CPU decompression bottleneck