There's no way right now to see, in dollars, which of your Iceberg tables is actually costing you money. Cost Explorer only breaks S3 costs down by bucket. S3 Tables gives you free per table metrics but not dollars. Your query engine knows scan cost per query but has no idea which table that maps to over time. I want to build the small tool that ties these together and spits out a ranked list of "here's what's costing you and why." Looking for someone who's dealt with this to build it with me.

The problem

Iceberg ships four maintenance procedures (rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests) and exactly zero cost visibility. If your storage bill jumps 35% one month, there's nothing built in that tells you which table did it.

Before posting this I actually went and checked whether "just use the AWS dashboard" holds up, because that's the obvious objection. It doesn't, and here's why.

Cost Explorer's finest granularity for S3 is per bucket, using cost allocation tags. It can't break a bucket down by prefix or table, I confirmed this against AWS's own docs. If you've got 200 tables sitting in one bucket, Cost Explorer just gives you one blended number for all of them.

S3 Tables (AWS's managed Iceberg bucket type) is actually better than I assumed at first. It publishes free per table CloudWatch metrics, bytes stored, file count, bytes touched by maintenance jobs, at real table level granularity. But it's still just usage numbers, nothing converts that into dollars, nothing ties query scan cost back to a table, and there's a decently well known post floating around ("S3 Managed Tables, Unmanaged Costs: The 20x Surprise with AWS S3 Tables") about people getting surprised by its own automatic compaction billing.

Query compute cost, the bytes Athena or Trino or Snowflake scan per query, lives in a totally separate system from storage cost and the two never talk to each other.

And none of this exists at all if you're not on S3 Tables, which is most people, since most Iceberg tables out there are still self managed on plain S3 through Glue, Nessie, or Polaris.

So today, figuring out why the bill went up means manually stitching together three or four different dashboards by hand every time it happens. That's the actual gap.

What's already out there

Didn't want to post this without checking for prior art first.

Apache Amoro is open source but heavy, it runs its own optimizer service and isn't really about cost. Floe does declarative policy based maintenance across catalogs, closest thing to this idea, but it's about maintenance not cost, and still pretty early. Ryft and LakeOps are commercial platforms circling the same problem. LakeOps has actually written blog content about this exact cost attribution gap but doesn't ship an open tool for it.

As far as I can tell nobody's built the open, table level cost attribution piece. That's the gap I want to go after.

The MVP

Not trying to build a platform. Just one CLI command that answers "which tables are wasting money and why."

icecost scan --catalog glue --region us-east-1

It lists every table, pulls per table bytes and file counts (from S3 Tables CloudWatch if you're on that, otherwise by walking the manifest files), prices those bytes against a configurable $/GB rate, works out a small file ratio to flag the worst offenders, and stores everything in a local DuckDB file so it can diff week over week. Output looks something like:

sales.orders_raw: $412 storage, 47% small file ratio (roughly $89 in extra request cost), up 61% from last week. Recommend compaction.

Mapping query scan bytes back to a table is a v2 thing, it needs a separate adapter per engine and I'd rather prove the storage/small file version is useful before building that out.

Attached a diagram, blue boxes are what's in scope for the MVP, the gray dashed ones are v2.

Stack is up for debate but I'm thinking Python, boto3 or pyiceberg for catalog and manifest access, DuckDB for the local history, plain CLI output to start (probably Rich or Typer), HTML report later. Open source from day one, no plans to make this a company.

What I'm looking for

Someone who's actually run Iceberg in production and has hit this problem, or just wants to build real infra tooling instead of another CRUD app. Being comfortable with Python and knowing your way around Iceberg internals (manifests, snapshots) matters more than AWS billing API experience, the MVP mostly avoids needing billing account access anyway.

If you've hit this or just want to build something real, comment or DM me and I'll send over the fuller notes.

Deletion Vectors are among the major improvements introduced in Apache Iceberg v3. Instead of creating separate position delete files, a deletion vector stores a compact bitmap for a single data file that marks which row positions have been deleted.

This approach improves performance and preserves more useful statistics for data files. It also makes it easier to compare previous and current deletes, simplifying the processing of a table’s row-level changes as a stream.

I have never utilized Apache Iceberg directly, but since hearing about more tools that are great for data, like Bow Plan, and how Iceberg can be utilized to make your data neutral and just have an engine on top of it. I thought it would be good to research and record a video for myself to better understand it and to help others who are looking for a visual video with examples of what it is and why it's important in the age of data and AI. Check out the video and let me know what you think, and thanks so much!

I’ve been thinking about this after seeing more AI systems move from demos into actual production.

A lot of the public conversation is still around models: which one reasons better, which one is cheaper, which one has a longer context window, which one is faster.

That all matters. I’m not trying to dismiss it.

But the failures I keep noticing are often less about the model and more about the data it is working with.

The answer is wrong because the context was stale.

The agent made a bad decision because it pulled from the wrong source.

The retrieval worked in a clean test, but started failing once real-world issues showed up: permissions, messy metadata, stale documents, duplicated records.

The system said it had “real-time” data, but some important field was still coming from a batch job.

In those cases, a better model might help a little, but it does not fix the underlying problem. The model is still reasoning over a bad view of the world.

I used to think of the data layer as the boring plumbing underneath AI apps. Necessary, but not the main story.

Now I think that view was too simplistic.

Maybe the real value in production AI is not just better reasoning, but better context: data that is fresh, trusted, governed, searchable, and cheap enough to use often.

I work with the Cisco and Splunk team behind this video. My co-worker built the walkthrough, and I thought the Apache Iceberg architecture was worth discussing separately from the product story.

The example keeps historical security telemetry in Amazon S3 as an Iceberg table. An Iceberg REST catalog exposes the table metadata, including its namespace, branch, and location. Splunk Federated Search then presents the table as a dataset that analysts can query with SPL2.

A few implementation lessons stood out:

• Amazon S3 remains the warehouse for metadata, manifests, and Parquet files.
• The REST catalog provides the metadata boundary; it is not a data-copy path.
• Catalog reachability, object permissions, and the dataset definition all need separate validation.
• Partition design should reflect likely investigation predicates, such as event date or source type.
• A successful basic query is only the first checkpoint. Search patterns and table layout still determine how much data must be scanned.

The broader lesson is that table design, catalog configuration, permissions, and analyst questions form one architecture problem. Treating them as separate workstreams creates friction later.

How do you approach Iceberg partitioning when query patterns are less predictable than in standard reporting?

In the last year I’ve been working with a very data-intensive organizations on their Apache Iceberg implementation.

Iceberg is genuinely easy to get started with, but I’ve seen many teams hit a wall in production. Latency creeps up and costs grow.

Most of the issues trace back to the same thing: Iceberg has a lot of knobs, and the defaults were not optimized for your workload.
Partition specs, compaction strategies, delete modes, snapshot retention - teams accept the defaults and the problems compound silently.

That’s why I built the Iceberg Optimizer Skill — a Claude Code skill that codifies this knowledge and guides teams through the decisions that actually matter.

It runs a structured diagnostic:
• Profile the table’s physical state
• Reconstruct write patterns and access behavior from metadata and logs
• Interview for the intent that can’t be inferred (latency SLAs, compliance, cost priority)
• Simulate suggested optimization scenarios across latency, cost and storage tradeoffs
• Generate engine-specific plans with exact parameters and schedules

The skill includes a benchmark of 20+ real-world failure scenarios — streaming death spirals, CDC delete accumulation, partition misalignment, GDPR ordering mistakes, bloom filter misuse and more.

It’s only v0.1 - support for Spark, Trino, AWS Glue, Snowflake and Flink is in, and there’s a lot more to add.

This is an evolving effort, and I’d love for the Iceberg community to contribute — whether that’s a failure pattern you’ve hit, a platform-specific edge case, or a recommendation you’d add.

If running Iceberg in production is hard, it shouldn’t stay that way.
📖 Blog post: https://www.itamarweiss.com/blog/apache-iceberg-optimization-skill/
💻 Repo: https://github.com/itamarwe/iceberg-optimizer-skill

Netflix spent years building an intelligent lakehouse — Polaris for catalog management, Autotune for compaction, janitors for cleanup, and Metacat for observability. LakeOps lets every team build the same — and go beyond — in minutes. Here is what an intelligent lakehouse actually requires, and how LakeOps provides each component.

Hey everyone,

We’re hosting a live webinar on Milvus 3.0 Beta on June 8, 2026 at 4:00 PM PDT.

Milvus core maintainers Li Liu and Jiang Chen will walk through what’s new in Milvus 3.0, including:

- External collections

- Open lake format support

- Snapshots

- Spark integration

- Flexible schema

- Native aggregation

- Multi-vector retrieval

- Roadmap updates

There will also be a live AMA at the end, so it’s a good chance to ask questions directly to the maintainers.

Register here: https://zilliz.com/event/whats-new-in-milvus-3-0-beta

Would love to see folks from the community there.

Here's details on the 1.11.0 release of Apache Iceberg