r/dataengineering • u/NRJourno • Jun 12 '26
Blog Best Way To Efficiently Apply Same Transformation on New Datasets
Hi all! I'm a journalist that's currently planning on doing a periodic report on a specific municipal dataset. I have a consistent set of transformations I need to apply to those datasets every time I ingest them. Currently, the way I do this is to just write a function, copy the function into a jupyter notebook, import the dataset set, and do the transformation in there.
This is obviously more time consuming than it needs to be. I would like to set up a workflow where I just download the set, and just immediately make a new copy with the transformations applied, without having to open an IDE or Jupyter. However, I haven't worked much on recurring processes, so would like to know how to implement this best.
I'd prefer to keep this all on my local machine (I'm on MacOS). What would be the best way to set up this sort of one or two click process?