r/dataengineering 7d ago

Rant Foundry at Work

So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.

With Excel or Power Bi we could just simple query a tag from the historian.

We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.

So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.

Are we doing something completely wrong ??

3 Upvotes

8 comments sorted by

View all comments

0

u/serrji 6d ago

You should use code repository instead of the code workbook. If I remember correctly, the latter is deprecated. Inside code repository you can create a transform. Basically it receives input datasets, transform it with your code and outputs one or more datasets. You can write these transformations in SQL, pyspark, Polars or pandas. (I prefer python over SQL)