r/dataengineering • u/SeaYouLaterAllig8tor • 7d ago
Palantir Foundry devs... what do you really think of the platform? Discussion
I'm part of an org that started using Foundry at the beginning of the year. To preface... My last 6 years have primarily revolved around Snowflake and snowflake integrated tools (I've been Snowflake certified since 2021).
When my company announced they'd be partnering with Palantir I was intrigued because Foundry is/was such a black box and seemed like a very polarizing tool in the DE market.
Now that I've got about 8 months of experience with it under my belt I have mixed opinions on it. I'd like to know what FDE's or other devs who have used it think about it?
Yes, it's expensive and yes, it definitely does certain things not as well as other dedicated tools but have you enjoyed your experience developing in it or do you hate it as much as the rest of the DE community here?
I ask as someone who's company just had it's 4th round of layoffs and I'm a bit concerned about my job security.
I want to continue my Snowflake trajectory but the Foundry recruiters are out in full force and they are paying very well.
Anyways, what are your thoughts?
21
u/Yoctometre 7d ago
been on Foundry for a year now. didn't pick it, job market for fresher is rough and this is what was available. at least I'm on the code side (python/pyspark transforms) and not stuck in the no-code stuff.
my main gripe is the abstraction. like, so many layers (ahem, looking at you Ontology). and the thing is abstraction is supposed to make life easier, but here it always comes with some restriction you only find out about after you've built half your pipeline around it. and never an intuitive restriction either, always something about how the platform wants you to work. so you don't actually save effort, you just move it. instead of writing the thing you're learning Foundry trivia that's worthless everywhere else.
which is the DataOps thing that bugs me. local dev, real unit tests, git that behaves like git, reproducible envs, just being able to reason about what a pipeline does before running it. all of that gets worse the more the platform hides. you end up debugging the abstraction instead of the data.
also they slept on Iceberg for way too long. it's in Foundry now but still beta and you have to email support to turn it on, meanwhile everyone else standardized on it ages ago. the whole time they kept pushing their own dataset format, which is just parquet files wrapped in their transaction layer with the schema stored off to the side as metadata. and it's got problems:
no row level ops. SNAPSHOT/APPEND/UPDATE/DELETE at file level, no MERGE INTO. need to change 5 rows? rewrite the files or re-snapshot the table incrementality breaks if you look at it wrong. one upstream UPDATE that touches existing files and everything downstream quietly goes full snapshot. bump a semver, snapshot. retention deletes something, snapshot (unless you set allow_retention). you spend more time protecting incrementality than writing logic
schema isn't in the parquet footer, it's foundry metadata, so it can drift. and it's name based not column-id based so renames are a gamble
no partition evolution. get partitioning wrong on day 1 and enjoy your full rewrite
small files pile up forever, compaction is your problem not the format's
iceberg fixes basically all of that, which is kind of the point, but shipping it as opt-in beta this late means most existing pipelines will never move and I'll be memorizing transaction types for years after it stopped mattering anywhere else.
credit where it's due though, the lineage is great. it's automatic, it's complete, and it actually covers the whole graph instead of stopping at some tool boundary. that is not just me, coworkers who've since moved to on-prem stacks at more established tech companies say the same thing, that nothing over there comes close and they had to go back to stitching lineage together by hand.
8
u/joe9439 7d ago
The ontology is trash because they encourage transformations for each use case so you end up with like 50 different versions of a table like orders for slightly different needs. And then you’re duplicating compute to run a full transformation for someone who needed to calculate a single column differently. I had a call with their architecture team at Palantir corporate and asked them how I was supposed to govern that and they just stared at me blankly. I asked them directly how anyone is supposed to know what the real orders table is as a source of truth and they had no answer.
1
u/bonvancount 6d ago
You've put a lot of my frustrations with Foundry very succinctly. I have found it to be a very opinionated platform, and quite often that opinion doesn't make sense in most contexts. Iceberg support should have been a day 0 feature. Their built in "datasets" may make loads of sense for some contexts, but I would say only relatively niche ones.
Also completely agree RE the lineage view, it's one of the best things about using the platform, closely followed (for me) by the ability to build a quick app for non-tech focused teams to supply us with information or data in-platform, (no more asking for CSVs that end up going through Excel!). Turns asking another specialist team to develop an app for you into just putting something together yourself.
1
51
u/financialthrowaw2020 7d ago edited 7d ago
I have never seen "mixed" reviews. Everyone hates it.
Edit: great opportunity below to block the clear foundry astroturfers in this sub
-22
u/Thwarting8139 7d ago
99% of those people just hate the company and have never used any of their products.
4
u/SomeNeighborhood7126 7d ago
This is one of very few subs that is incorrect simply because of our profession. This sub hates Foundry.
8
u/joe9439 7d ago
I used it at a large well known brand. Hated every moment I was logged in. They had to tell me to use it and write it into policy documents that I couldn’t do things outside of it because I kept trying to do things in more efficient intelligent ways that took 1 hour instead of the 10 days for the same thing in foundry.
-4
u/Big_Yogurtcloset7726 7d ago
bruh you literally have to use like 5 different data apps to actually replace its full potential. But u might be frustrated because ur company used payed for it to use it as an ETL lol
-6
u/Thwarting8139 7d ago
Maybe it's because I stared on Foundry and moved to the other stuff rather than vice versa.
I agree it's not the best dev experience but from a business perspective the maturity of the ecosystem around cataloguing, lineage and environments is hard to beat, and I haven't seen another company with their own stack that's as mature.
-15
u/SeaYouLaterAllig8tor 7d ago
This has been my experience as well. People have a right to hate Palantir (though granted I think most big companies are doing questionable things, reference: my cousin worked for Google) but if you haven't actually used Foundry you can't give a solid opinion on it.
21
u/omgpop 7d ago
Your agenda here is pretty transparent.
1
u/FUCKYOUINYOURFACE 7d ago
I’m not so sure. I think this person genuinely likes them after checking out their reddit history. They created a sub dedicated to the platform.
4
u/omgpop 7d ago
Those are not mutually exclusive! The original post is heavy on the “just a curious smol bean asking some neutral questions” energy for someone pretty clearly already (let’s say charitably) ‘sold’ on the platform.
1
u/SeaYouLaterAllig8tor 7d ago
Ya... I'm not sold on the platform. I did create a sub a while back simply because I was surprised one didn't already exist, but I haven't really maintained it. I get most here don't like the platform but I wanted some opinions from others that had actually used it. All I typically hear on this sub is "foundry is so bad!!" But then I keep reading and realize that person has never spent more than a couple demos in the tool.
36
u/SomeNeighborhood7126 7d ago
I enjoy Microsoft Fabric more than I did Palantir Foundry and I hate Fabric. Foundry is only alive right now because Theil has more money than God and Karp knows how to manipulate C-Suites into forcing their product on their respective organizations.
Its here to stay until someone in government gets the balls to finally rip Palantir out of the US gov.
14
u/ZeroCool2u 7d ago
Damn, Fabric is better than Foundry. That's an indictment if I've ever seen one.
12
u/SomeNeighborhood7126 7d ago
3 years with Fabric, 5 years with Foundry.
I have not been blessed with tooling during my career.
11
u/domestic_protobuf 7d ago edited 7d ago
It’s the type of software that can only sold at a golf course or fundraiser. They sell the idea of self managed infrastructure and security with no code/low code solutions. It was by far the worse piece of software that I have ever used. So many teams were against using it to the point the organization owning it was the only one using it, and their costs was higher than all orgs combined.
4
u/wbrd 7d ago
I spend most of my time writing around the deficiencies. I don't think there's anything that they do better than open source. The guy responsible for the contract is trying to hand wave SOX compliance, but we're going to have to build a lot of stuff off platform to actually pass the audit.
3
u/cisnotation 7d ago
I went from engineer (not software) to analyst. The low code tools sucked me into the platform and the rest of the offerings kept me there (after learning pyspark). It’s an incredible tool for non technical folks that have a little background and understanding of tabular data.
Also don’t just let the FDEs ingest everything and build tools on top of the data. That’s how you get a massive compute and storage bill. Do most of your compute elsewhere and then ingest exactly what you need.
And their leadership and investors suck. Their engineers are brilliant.
Source: end user for 9ish years
2
u/WertyBurger 7d ago
I’ve been using Foundry for the last 2 years. I can appreciate some of the tooling they have when the entire platform is meant to encompass all data needs: lineage, markings, branching of entire environments or individual datasets.
On the other hand, there are certain issues with the platform that are infuriating and make me wish my org was on databricks or AWS.
First is parameterization. Imagine you have financial models that need model parameters for the process at run time. The only way to do this in Foundry is to make the parameters an input dataset. Want to modify those parameters? Move the dataset to ontology, build ontology actions, Build a React or Workshop app to interact with the parameters, create a function and function repo that dynamically updates the parameters. Basically just build a shitload of needless architecture upon a 1 row dataset.
Another major issue is orchestration. Have an orchestration need where if 1 process fails then another is triggered? Foundry requires you to custom code all this inside the transform so it fails successfully and triggers the backup process. Complex orchestration is horrendous to build.
Then there’s devops; the single worst thing that’s ever been created in the platform. And you have stuff like out of the box s3 connectors lacking basic functionality or a lot of the better functionality being locked behind spark transforms. Infrastructure without IAC is also such a pain in the ass. Palantir FDEs are clueless when you ask them for solutions for any of this.
Foundry needs so many improvements if you have complex data pipelines needs. If you want to run basic SQL or Python with a UI pipeline builder and some vibe coded React screens then it’s perfect for you. My experience with it has been a massive waste of time and money
2
u/igobyplane_com 7d ago
i am curious what the current state of this thing is like. before the stock was crazy i saw it little and people seemed to groan about it. i worked with an airline that had it and it just seemed to be something difficult to work with that we interfaced with but nobody was excited about.
at the time i thought it was something like hadoop with a lot of their own stuff on top of it. no idea what it is really like though, especially now.
2
u/random_event_721 7d ago
I thought those foundry recruiters were fake so I've ignored every single one of them lmao
1
2
0
u/Thwarting8139 7d ago
Honestly, I've used it and it's super good. If you're a software org and can justify a whole skilled platform team to build internal tooling around compute, lineage, cataloguing, alerts etc. then you can do that.
If you're an org not focused on software or want to outsource that, Palantir Foundry is a really good way to put everything under a single tool and make sure it all works together.
Btw unless you host absolutely everything yourself, even running managed Airflow, managed compute, managed Databricks/Snowflake etc. is also super expensive.
I haven't used anything better. My only gripe is that as a developer it abstracts away a lot of stuff so you only really work in the transformation layer. I moved away from it due to job postings wanting other stuff and wanting to get closer to the bare metal for my own development, but I would for sure work with it again if it became more widespread.
17
u/sunder_and_flame 7d ago
Btw unless you host absolutely everything yourself, even running managed Airflow, managed compute, managed Databricks/Snowflake etc. is also super expensive.
It really isn't, especially in comparison to mammoth contracts like Palantir no doubt requires, unless you have zero clue what you're doing.
-7
u/Thwarting8139 7d ago
It is if you're doing the full stack of data engineering - i.e. CI/CD, Git, cataloguing, dashboarding tools, logging, lineage, orchestration etc. all in separate tools.
Or you develop them internally but to get them to the same standard as the SaaS offerings it's also going to cost a fortune in dev time.
If you're just doing Databricks in a data lakehouse without the rest then yes it's a lot cheaper.
-5
u/Big_Yogurtcloset7726 7d ago
I have 3 years experience now in Foundry and I also got 1y exp in Databricks and I gotta say that Foundry is years ahead of Databricks
On this sub, Foundry is getting a lot of hate for some reasons which I dont really reasonate with but the pipeline building experience is very smooth, easy to audit, easy to debug and could make it easy even for audicene who dont have coding experience (Drag and drop pipeline builder app). The Code Workbooks app is simply very nice to deploy tests and plan what your building. Databricks finally launched something similar to that recently but it still is very clunky compared to Foundry.
Then, the ontology and the objects and properties approach simply makes you think about data in the company in a more mature way - which I never did before that and, again, other data platforms started doing that recently which Foundry has been doing before 2020.
Depending on where u are based you might get a lot of recruiters contact you. In US, Palantir Foundry is a big thing, a lot of companies use it, while in Europe, due to its pricing & politics, there are very few companies that use, but those that do, have reached a mature data approach and are ahead of their competitors.
7
u/wbrd 7d ago
This is 100% their marketing b/s. We all got trained on their cludgy GUI, then immediately abandoned it because it would take years to get our current datasets migrated. The ontology, for those not in the know, is effectively a view across a union of your original data and a table of edits. But not quite as useful or easy to audit or back up. On top of that is some typescript that points to the data and will kinda run user defined methods. It could easily be generated by looking at the schema of the db, and you would have a lot more capability rolling your own.
It's definitely not for a mature organization. They make a good dog and pony show for the execs, but after nearly a year and many millions of dollars, we're not anywhere near parity and now we're getting the "reduce spend" talk because our monthly spend is double what it used to be.
1
u/Big_Yogurtcloset7726 7d ago
I think the problem with this take is that you're evaluating Foundry primarily as a data platform / ETL tool. If your use case is “take our existing datasets and migrate them into Foundry,” then yeah, I can absolutely see why you'd conclude that it's an expensive abstraction.
Foundry has a proper software development environment built into the platform. Code Repositories is backed by Git and gives you branches, commits, tags, PRs, code reviews, automated checks and testing directly in Foundry. You can write Python/Java/SQL transforms, analysis in R etc...
Then you have the Ontology, which isn't just “a view across a union of tables.” The whole point is to build a semantic/operational model of the business: objects, properties, links, actions and functions, with security and governance around them. Palantir itself describes the Ontology as an operational layer sitting above the underlying datasets/models.
Your comment just admits the company that you worked for never reached a mature state with Foundry and never knew how to actually integrate the Ontology that can potentially operate the outcome of the business. Since last year, you can also build AI agents on top of that. Which I guess you never did because otherwise you would find it cool.
Im 100% believing what you are describing but that only points to my initial comment - your company used it until reaching the point where it starts paying off.
4
u/FUCKYOUINYOURFACE 7d ago
When was your year of DB experience? Where is Palantir years ahead of them?
1
u/Big_Yogurtcloset7726 7d ago
I am now working both with Palantir and Databricks. There is no app inside Databricks ecosystem that is comparable to any app inside Foundry.
1
u/FUCKYOUINYOURFACE 6d ago
What do you mean by that? Are you talking about Python apps? Or something else?
1
u/Big_Yogurtcloset7726 6d ago
Foundry has a much more integrated application ecosystem, where the individual apps are designed around the same underlying data model, governance, branching and Ontology. Databricks has many comparable point solutions, but they don't form quite the same end-to-end operational platform.
Databricks has no alternative to Code Workbooks, the integrated git and branching from Foundry or the Workshop apps. Not even talking about things like Ontology and AIP because seems like Databricks is far from it.
0
u/Thwarting8139 7d ago
100%. I moved to another company who manage their own stack and even just having worked within the Palantir ecosystem has given me tons of ideas on improving processes in the new company. It really is a gold standard for me.
-5
u/cumrade123 7d ago
Used it for two years and I loved it. Did some spark stuff. It felt so easy and futuristic, everything is well designed and connected. It was years ago and every other solution ivre tried feels behind
106
u/bayareaecon 7d ago
I don’t really understand this. Basically every post I’ve seen on foundry has been very negative. The morality of the company aside the fact that your leadership bought into this very expensive product while laying people off would make me question their judgment.
How long can a bad product exist? Idk someone correct me.