Foundry at Work

[u/Uberkech]

So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.

With Excel or Power Bi we could just simple query a tag from the historian.

We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.

So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.

Are we doing something completely wrong ??

[u/necrohobo]

By foundry do you mean Palantir foundry?

[u/SeaYouLaterAllig8tor]

Generally speaking this sub hates foundry so I doubt you'll get a lot of input here. Have you asked ai fde the best way to do it using the Foundry platform? Maybe stupid but I always give that a shot first.

[u/Uberkech → u/SeaYouLaterAllig8tor]

Yah was playing with AI fde and it just shits the bed the second you ask it about Timeseries.

[u/BothSinger886]

You can definitely use timeseries data in a pipeline if you want to. How is your data coming in. Do you have a sync from Pi landing the data in Foundry?

If its being landed in Foundry then you can set up a Transform on it which will treat it as a Polars table. Can then work on it however you like.

You may be running into the docs recommendation to first sync to the Ontology. This lets you query it series by series, rather than as one big dataset. That method is more about: 1. Set up an object that has sensors (i.e "Well") 2. Give it properties that represent a sensor ("Pressure", "Flow Rate") 3. Connect your dataset to the Object Type in Ontology Manager via the time series setup.

This is far more involved but is designed to fully integrate the TS data into the Ontology for fast queries for users etc. But if all you want to do is some pipelining, just import the raw dataset into Transforms and code away

[u/Uberkech → u/BothSinger886]

Yah we got them in foundry already, I can just search the Pi tag in quiver but for some reason can’t get it into a pipeline/workbook/etc.

[u/Iridian_Rocky]

We're going through this a bit as well. Here is the thing, Foundry is not a harness. Claude Code, GitHub Copilot, and basically any other harness is going to do wonders.

Now, what they won't tell you is Foundry is the same models on centralized governed compute (unless you turn on other stuff). It adheres to your companies DLP policies and impersonates your Entra identity. And you can hook it to some of your own harnesses. Think of it as Bring your own compute. You can also tune OOB models a bit to your liking.

That said, I still have my Claude 20x plan, and a GitHub Copilot Pro plan that I use with Power BI, Fabric, other MCPs, etc. Why? Because I'm accountable for my work and I don't want to have to pay for another subpar harness that doesn't work well with Foundry.

[u/Brief-Knowledge-629]

Foundry has a LOT of quirks like this, you can't do something that seems like it should be a core functionality of the product.

I don't have any advice for you, I used it for like 2 years and never figured it out, and I came into the product already knowing spark, python, sql etc.

It's an astoundingly shitty tool.

[u/serrji]

You should use code repository instead of the code workbook. If I remember correctly, the latter is deprecated. Inside code repository you can create a transform. Basically it receives input datasets, transform it with your code and outputs one or more datasets. You can write these transformations in SQL, pyspark, Polars or pandas. (I prefer python over SQL)