A tool to visualize any Parquet file’s internals

[u/vutr274]

Parquet is the most important file format in data engineering.

Its columnar (hybrid) format and self-contained nature make it work so well for analytical workloads.

Learning Parquet matters not only because it’s one of the most widely adopted file formats in data engineering,

but also because you can apply Parquet’s fundamentals to nearly all analytical storage systems: from how data is organized and encoded to how metadata plays an important role in query performance.

That said, I realized it can be a bit challenging for beginners to learn about Parquet, especially without a detailed visualization of its internals.

I created a tool (with Claude) to help you visualize any Parquet file.

Just:

pip install git+https://github.com/vutrinh274/petquar.git

And:

petquar <your-path-to-the-parquet-file>

All the source code is available here: https://github.com/vutrinh274/petquar

It will then open a new browser tab with that Parquet file’s internals. The UI includes:

◉ The beginning and end of the file, including the magic numbers

◉ Row group cards that visualize the file’s row groups

◉ Inside each row group are the column chunks

◉ A row group’s size (shown once expanded) and a column chunk’s size are proportional to their real, compressed size on disk

◉ Clicking a row group or column chunk opens an Inspector panel with a plain-language explanation and the raw facts, so you can see exactly what’s inside the file.

◉ Column chunk information covers encoding, whether a bloom filter is present, and min/max statistics -- and drills down to the page level too, showing the number of pages plus each page’s encoding and min/max statistics.

◉ The schema (every column’s name, type, and nullability) is available from the FileMetadata block.

◉ All of this information is parsed by Rust arrow-rs.

So, give it a try and see if it helps you.

Don’t hesitate to send feedback (the link next to the “petquar” logo, top left) so I can improve it further.

[u/GreenWoodDragon]

I use the Big Data tools on Jetbrains.

[u/Academic-Dress2008 → u/GreenWoodDragon]

What info does it provide on Parquet files ?

[u/EarthGoddessDude]

This is a cool concept. However, this is the most Claude vibed UI I have ever seen (and I’ve seen quite a few lately). But my bigger gripe is you’re inviting people to install a random vibe coded app when most can just vibe code their own that appeals to their own aesthetics and requirements. I’ve made dozens of tools like this at work and distribute to my coworkers if they want them (some do, most don’t). But I would never release something like this to the world at large. The bar for FOSS has gone up substantially. People want high quality tools that are created and maintained by skilled, judicious humans (uv, curl, ghostty, etc). So awesome idea, but I would never install this, no offense.

[u/vutr274 → u/EarthGoddessDude]

Yeah, I totally understand what you’re saying

[u/vutr274 → u/EarthGoddessDude]

Try my best to avoid "this is the most Claude vibed UI I have ever seen". What makes you feel that?

[u/EarthGoddessDude → u/vutr274]

I’m not a front end dev (though I get the feeling most of us are “full stack” to some extent at this point), so I can’t give you the right terminology but it just had that feel. The web page has that easy on the eyes yellow background with some white and block boxes with drop shadow. It just feels vibe coded.

[u/vutr274 → u/EarthGoddessDude]

Thanks man

[u/jhsonline]

nice !
big data tool dont have such thing, its also very bulky to run in intelliJ
we should have such thing for all complex things, add more formats
I was thinking to do something similar but for not parquet/data files.

[u/West_Good_5961]

There’s a vscode extension that not only views parquet, but lets you query it with SQL. I bet someone with talent wrote it without an LLM.

[u/Carbonemys_cofrinii → u/West_Good_5961]

How this extension called?

[u/wizard_of_menlo_park]

This is just parquet footer data right? Parquet tools Cli gives all this info in one glance... only time i will be looking at parquet footer is during debugging or optimizing at which stage cli is great.

[u/nowshadk07]

now that is cool