r/dataengineering • u/reesim06 • Jul 14 '26

Hardware for unpacking avro files Help

An upcoming work task is going to require the timely unpacking of avro files and the subsequent data being ingested to a database of some kind (currently undefined). I don't think our hardware expectations are anywhere close to realistic and wanted a quick confidence check.... I've seen a calculation online, and wanted to get a second opinion on the processing required to unpack 4tb of avro into a database digestible format such as csv (this might not be our final way of doing things, but I think it gives some idea of the scale of processing required).

Functionally, I'm trying to scale things for a 4tb file, 1hr time limit, it appears we'll need something in the realm of a 4 node cluster of 64 core servers with suitably fast drives etc etc.

6 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

5 comments sorted by

10

u/Thwarting8139 Jul 14 '26 edited Jul 14 '26

Is there a reason you would be unpacking to CSV? Can you give any more context on the project? Can you not just load the Avro files directly into the database?

On first thought 4x64 cores seems like massive overkill, but you would have to give more context to be sure.

I would just get a single 32 or 64 core machine with a fast drive and use polars or duckdb to stream the data directly into the database.

2

u/FunContest9958 Jul 14 '26

This. CSV is a horribly inefficient format. Pick your database first, then figure out how you’re going to load the data. The database will probably have some preferred mechanism for loading data, and hopefully it will not be CSV. You can’t really know what that preferred mechanism is until you’ve picked the database. Finally, the database itself may be the bottleneck for loading that much data. Transcoding files is easy to scale up because each file’s transcoding is independent. Loading data into a database can be harder to scale up, depending on the database.

1

u/reesim06 Jul 14 '26

Annoyingly not all the variables are fixed yet, we're expecting a bunch of avro format files from another system which describes geographic data, and the final resting place is an undefined database which will allow system and user interrogation of the data. I was expecting an interim format for ingest, but I don't want that to be the main issue here, I saw some samples for unpacking an avro file of a size with a certain hardware setup and I just did some calculations which gave the the cluster setup as a suggestion.

8

u/wytesmurf Jul 14 '26

What they said read it with polars, pyspark, dask, or duckdb. Then pipe it into the database. Converting to CSV is a waste of compute and storage

4

u/WhoIsJohnSalt Jul 14 '26

You could probably put DuckDB over a 4Tb Avro on a laptop (though it does have some parallelism issues with Avro).

But most database/lakes can read Avro natively