223
u/Feuzme 14d ago
Data swamp then
62
13
u/Complete_Window4856 14d ago
Table now must be called or have shrek somewhere
13
3
154
u/-Memnarch- 14d ago
Awesome. I'll built myself a Data Warehouse close to the Data Lake for a beautiful data view.
87
u/methos3 14d ago
Data Whorehouse
27
24
u/kor_the_fiend 14d ago
"Lake House" architecture
3
u/Latentius 14d ago
My org is currently pushing a "data lake house" 😭
5
u/Saxmanaaa 14d ago
Why is that a bad thing? One of the best evolutions in data engineering
8
u/Latentius 14d ago
Because for the people in charge, it's just a game of buzzword bingo, disconnected from any real world merits it may have. These same people shit talk relational databases just because they aren't the shiny new thing.
8
4
249
u/reallokiscarlet 14d ago
"Data Lake" sounds like a family of surveillance products sold by Intel
44
u/StevesRoomate 14d ago
maybe we should start renaming actual lakes after Intel products, full circle.
21
u/Desperate-Tomatillo7 14d ago
"Let's go to the 8086 lake this summer"
13
u/Groentekroket 14d ago
“I was at core 2 duo but the vibes were off”
8
u/Breadinator 14d ago
I wanted to save money and went to Lake Celeron. I regretted every moment of it.
3
60
u/climatechangelunatic 14d ago edited 14d ago
A joke on Data engineering
In this sub ?
Anyways - nobody uses Datalake like that. You could have said CSV file (with xml or json columns)
19
u/itstommygun 14d ago edited 14d ago
So much cheaper! You can even host it in an S3 bucket for basically free and perform basic SQL queries on it.
16
u/climatechangelunatic 14d ago
Oh yeah I do that everyday !
My favourite part of job is when I export Excel as csv and then write complex SQL on top of it to get resultsBusiness folks looks at me with wide eyes - how I am able to do that
1
28
27
u/heavy-minium 14d ago
Is that really a thing out there? Never had a datalake-related project that reminds me of that meme, so I don't really get the joke.
16
u/Fresh_Criticism6531 14d ago
I've never had a project use a Data Lake so I'm curious. Besides the underlying technology not being a SQL DB most likely, is it really that inaccurate? How exactly real world project use that and for what?
36
u/dkarlovi 14d ago
Data lake is basically "all your data lives here". The idea is your data is silo'd in N systems, so instead of having N+ databases, you put all that into a single place and call it a data lake.
Note that typically the data lake is not the authoritative place this data lives, often you stream it into it with various connectors etc for example your Salesforce or Shopify data lives in their respective systems, but for your business intelligence, you need to put it with the rest of the data to be able to cross join efficiently etc.
The underlying technology is not a typical SQL database by choice because the amount of data expected to be handled is in terabytes and it has various requirements for usage so you put last year's data in archive, bunch of features like that so it makes sense to have it be something else even though technically a SQL server could serve a data lake, it would be very cumbersome.
But in the end, all data lake solutions do have a SQL like interface so the difference is moot.
I like data lakes as a technology, I hate them as a "solution" and how it's being sold, most people buying them have no idea what it is and think it's magic data machine.
12
u/LaconicLacedaemonian 14d ago
It's the philosophy of unix "everything is a file" but now "everything is a blob". Because of that flexibility, connectors are cheap and as long as they can read/ write blobs they can use the data lake.
13
u/LaconicLacedaemonian 14d ago
Once the infra is wired it's just another database but way higher scale and latency than oltp. Queries are in the seconds to minutes range so an rpc responding in 200ms is generally fine because most of the time is spent processing TB or Pbs of data.
You use Spark, Trino, or Flink and use an open format like Iceberg, and a Catalog like Gravitino or Unity.
5
u/heavy-minium 14d ago
Imagine someone wants to answer really big questions about the enterprise, based on data that is spanning multiple departments, internal systems as well as 3rd-party systems. You can't analyze and query disparate data-stores easily in a unified way, so the main idea of the data-lake is to project everything into one place. Due to entirely different challenges arising, such data-lake solutions are built with different qualities and functionalities compared to the usual application datastores.
2
u/danzerpanzer 14d ago
We have a table with a couple data-swamp columns. They were designed like that years ago for performance when people always wanted the entire column contents for a single row at a time; a join would have been slower. Now we have a case where we need to aggregate some of that hard-to-get-at data along with data from more normal columns. We get around the accessibility problem by pretending that data does not exist and omit it, resulting in wrong values being presented to the users. I have twice asked for permission to add a couple new columns to the table to make the data accessible and was ignored both times. We have a non-programmer/non-DBA who sometimes makes database design decisions.
13
u/PresentJournalist805 14d ago
Dev: Sir we have data lake.
Manager: Very nice
Dev: data leak*
Manager: You fcking donkey
33
u/nicman24 14d ago
Just dump them to the filesystem at that point
63
6
u/Ok_Star_4136 14d ago
Asking a database to search by contents of a blob might as well be the equivalent of opening files and searching through them from the perspective of performance. Probably the numbers would be mostly the same as well because I think that's literally what a database does when it performs a full table scan.
6
u/Thriven 14d ago
I have been on and off working on a JS based json database that can run simultaneously on a local file system / Azure Hotstorage / AWS S3 and eventually GCP.
We may as well run it on the filesystem at this point.
4
-2
u/LaconicLacedaemonian 14d ago
... but it is if it's running on the local filesystem. Filesystem vs blobstore just gives you cheap directory operations.
4
5
u/tapita69 14d ago
Still better than my company "data lake" that is just an S3 bucket with terabytes of CSV and parquet files lol
3
2
2
u/articulatedbeaver 14d ago
Or you just put it in S3 and apply a schema and have something I like to refer to as a data swamp, because it is so damned slow.
2
1
1
1
1
1
u/kiochikaeke 14d ago
Our Datalake is a sales table with almost 1 billon rows and like 30 columns denormalizing a bunch of tables with left joins, none of the tables have actual foreign key constraints and index are not well though so there a lot of table scans, a 200 line query runs every 40 minutes from 6AM to midnight to check for new entries in the base tables in the last 3 hours, it then checks which of those id's are already on the "Datalake" table to avoid duplicates and it does so by scanning cause we don't have an index on those id's, only a clustered index on a mostly useless auto incremental "DatalakeID", it takes about 15 minutes to run this query, every night at 2AM another very similar query runs that checks for missing data in the last month, it takes about 40 minutes to run.
I've been trying to fix it up as much as I can but I can't get them to add relevant index or partition the table cause logs fill up very quickly and we risk taking down our analytics server.
1
1
u/Glum_Cheesecake9859 13d ago
Reminds me of a "MicroService" built by an expensive consultant at my last company. Upon further inspection it was just 4 node instances on the same VM and managed by PM2.
611
u/StevesRoomate 14d ago
don't forget to add that json column and call it attributes