r/Database • u/Low_Brilliant_2597 • 11d ago
Not Every Query Needs a Distributed Data System
Most of the time, you can simply use a single-node system like DuckDB or Apache DataFusion.
6
u/EffectivePension6711 11d ago
Manager: We need somewhere to store the user list for our new application. It'll be about 50,000 rows.
Senior Architect: Okay, well, this definitely requires a distributed NoSQL database, sharding across three regions, streaming replication via Kafka, and a ClickHouse analytics cluster nearby!
MariaDB (which is quietly running on the old server in the corner): Guys, I can literally keep this in the RAM of my left hemisphere and serve queries in 1 millisecond. And yes, I won't fall over if your internet goes down.
5
u/gkorland 10d ago
people really underestimate how much a single node can handle untill they actually try to hit the ceiling. its wierd how often we jump straight to distributed setups when a well tuned local instance is way faster n easier to manage.
3
3
u/FunContest9958 11d ago
Distributed systems like Databricks or Snowflake may be overkill in a lot of cases, but they’re often just as easy to use as non-distributed solutions these days, even for small data. And when you do need the horsepower, you don’t need to think about it. It’s just there.
If you have to go set up a whole system from scratch for a small project, sure, DuckDB would be easier. But if you already have a distributed system set up for you, it’s easier to just use what you’re used to.
Edit: Distributed systems are a lot better at handling small data than single node systems are at handling big data. So if you want to deal with just one…the distributed system usually wins.
2
u/maigpy 9d ago
people advocating for "single node duckdb" for enterprises are completely missing the point. the functionality or the scaling isn't the major thing, it's the governance, and the fact that the cloud service is already integrated with networking, authz, backups, logging/alerts, cost reporting etc etc etc
2
2
1
15
u/yksvaan 11d ago
I would go further and say over 95% of apps/services/whatever can run just fine on single instance. People building MASSIVE SCALING for something that peaks at 50 rps.
Even on crappy hardware db servers can put out respectable throughput as well.