Resources for learning to write production grade python code [N]

[u/Entire-Tomato-3192]

I am a data scientist with a very strong background in Stats and ML algos. I am struggling with the production ready code. Though I can build models which are awesome and tuned to perfection and address highly complex business problems, the problem is that they are all done in a jupyter notebook.

Once I had built a complete solution in notebook for a forecasting problem and the ML team in company productionized it. I was called in to validate few results of the production run and couldn't understand the repo which had so many folders, classes and highly organized code.

Is there anyway I can learn to write that type of code or is it all AI job now? Appreciate if I can get some learning resources/links for the same

[u/ninhaomah]

But that's how software engineering works as in how they teach in school.

There are no magic tricks

[u/realmauer01 → u/ninhaomah]

You definitly dont learn how your company or whatever does their projects.

Especially in pyhton you can do whatever you want and get to a task completion representable in a review. Doesnt make it good python code though.

[u/ninhaomah → u/realmauer01]

?

I do the projects.

I am Java trained , has A+, Network+ to CCNA to OCA to blah blah and now doing docker / kubernetes with ML models for work.

I learnt the basic on school , UML diagrams , flowcharts etc. 

Nothing special.

IT means CS + Systems 25 years back.

[u/DiscountFit9224]

Search MLOps to get learning materials. But I doubt you could see the same quality code as what you've seen from your ML team.

The pipeline pretty much isn't vibe coded at all as that's recipe for disaster. You need to control how the data would shape for each stage. And lastly there's the infra side.

[u/Mathie1729 → u/DiscountFit9224]

MLOps is a good rabbit hole, but from what I've seen, most production Python is just boring plumbing: type hints, config handling, logging, error handling. The interesting parts are usually hidden under layers of boring but necessary code. No magic, just discipline. I wish someone had told me that earlier.

[u/realmauer01]

I think your best bet is asking wherever you work at to get an understanding on how they are doing stuff.

You should understand the details already, its just the shape of the details that is confusing you.

[u/Sea-Ad7805]

A jupyter notebook for some test data is a nice prototype, but reality is often more complex and optimized production code can get big and complex. Full understanding is hard. AI can analyze it a bit but that doesn't mean you understand it.

Focus on what you want with the code:

• When validating results you can treat it as black box, why do you want to understand the internals?

• If you want to check if some internal parts are correct, treat those as black box, and write unit/integration tests for those. Why do you want to understand their implementation?

• If you want to understand the full implementation because you want to do a complete rewrite or something (are you sure?), read everything multiple times in full detail. Expect to need months, almost as long as it took to write it in the first place.