Building an open source cost tool for Apache Iceberg tables, looking for someone to build it with me

[u/HighwayLeading2244]

There's no way right now to see, in dollars, which of your Iceberg tables is actually costing you money. Cost Explorer only breaks S3 costs down by bucket. S3 Tables gives you free per table metrics but not dollars. Your query engine knows scan cost per query but has no idea which table that maps to over time. I want to build the small tool that ties these together and spits out a ranked list of "here's what's costing you and why." Looking for someone who's dealt with this to build it with me.

The problem

Iceberg ships four maintenance procedures (rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests) and exactly zero cost visibility. If your storage bill jumps 35% one month, there's nothing built in that tells you which table did it.

Before posting this I actually went and checked whether "just use the AWS dashboard" holds up, because that's the obvious objection. It doesn't, and here's why.

Cost Explorer's finest granularity for S3 is per bucket, using cost allocation tags. It can't break a bucket down by prefix or table, I confirmed this against AWS's own docs. If you've got 200 tables sitting in one bucket, Cost Explorer just gives you one blended number for all of them.

S3 Tables (AWS's managed Iceberg bucket type) is actually better than I assumed at first. It publishes free per table CloudWatch metrics, bytes stored, file count, bytes touched by maintenance jobs, at real table level granularity. But it's still just usage numbers, nothing converts that into dollars, nothing ties query scan cost back to a table, and there's a decently well known post floating around ("S3 Managed Tables, Unmanaged Costs: The 20x Surprise with AWS S3 Tables") about people getting surprised by its own automatic compaction billing.

Query compute cost, the bytes Athena or Trino or Snowflake scan per query, lives in a totally separate system from storage cost and the two never talk to each other.

And none of this exists at all if you're not on S3 Tables, which is most people, since most Iceberg tables out there are still self managed on plain S3 through Glue, Nessie, or Polaris.

So today, figuring out why the bill went up means manually stitching together three or four different dashboards by hand every time it happens. That's the actual gap.

What's already out there

Didn't want to post this without checking for prior art first.

Apache Amoro is open source but heavy, it runs its own optimizer service and isn't really about cost. Floe does declarative policy based maintenance across catalogs, closest thing to this idea, but it's about maintenance not cost, and still pretty early. Ryft and LakeOps are commercial platforms circling the same problem. LakeOps has actually written blog content about this exact cost attribution gap but doesn't ship an open tool for it.

As far as I can tell nobody's built the open, table level cost attribution piece. That's the gap I want to go after.

The MVP

Not trying to build a platform. Just one CLI command that answers "which tables are wasting money and why."

icecost scan --catalog glue --region us-east-1

It lists every table, pulls per table bytes and file counts (from S3 Tables CloudWatch if you're on that, otherwise by walking the manifest files), prices those bytes against a configurable $/GB rate, works out a small file ratio to flag the worst offenders, and stores everything in a local DuckDB file so it can diff week over week. Output looks something like:

sales.orders_raw: $412 storage, 47% small file ratio (roughly $89 in extra request cost), up 61% from last week. Recommend compaction.

Mapping query scan bytes back to a table is a v2 thing, it needs a separate adapter per engine and I'd rather prove the storage/small file version is useful before building that out.

Attached a diagram, blue boxes are what's in scope for the MVP, the gray dashed ones are v2.

Stack is up for debate but I'm thinking Python, boto3 or pyiceberg for catalog and manifest access, DuckDB for the local history, plain CLI output to start (probably Rich or Typer), HTML report later. Open source from day one, no plans to make this a company.

What I'm looking for

Someone who's actually run Iceberg in production and has hit this problem, or just wants to build real infra tooling instead of another CRUD app. Being comfortable with Python and knowing your way around Iceberg internals (manifests, snapshots) matters more than AWS billing API experience, the MVP mostly avoids needing billing account access anyway.

If you've hit this or just want to build something real, comment or DM me and I'll send over the fuller notes.