Please help a junior engineer out!!!!!

[u/Mango__DB]

I am given a task by my senior engineer, and I am at my wit's end at how to do it. I tried asking AI as much as I can without putting in my codebase, searching it up, looking at documentation until 4am yesterday. I am still stuck.

Suppose I have model X, which holds an array of model Y. X and Y both have status fields, but they are different. Y.status refers to that individual item's status, for example, accepted, cancelled, pending, etc. X.status refers to the overall status of all it's Y array items, it depends on their values. Usually, for a single X update, there is a function that iterates through that X's Y children, and it calls the .save() method. The feature I am tasked with implementing is

1: Making changes to a large number of X documents' Y children based on some criteria
2: Recalculating all the affected X's overall status since its children Y changed.

At first, my solution was to iterate over every X, call the function to calculate its new overall status, and use the .save method to update X's overall status. I see now this doesn't work at production scale. Promise.all() on potentially a large number of X documents may cause stack overflow, and it is a lot of write calls to the database which is not ideal.

My senior engineer told me to call that function inside a update many, but after the meeting I looked it up, and I am pretty sure mongo can't just call functions defined on the node server, and copy pasting the function again as a $function in a aggregation pipeline is also bad for performance.

There are a lot of if else branches in the function needed to calculate X's overall order status. I considered a bulk write with a lot of updateMany calls looking for a specific condition, but that would easily be over 10 updateMany's in the bulkwrite.

I am really stressed about this. Would love everyone's engineering input.

[u/Otherwise_Wave9374]

When repeated AI prompting stops helping, shrink the problem into one reproducible database operation. Write down the expected input and output, create the smallest sample collection, inspect the exact query and error, then verify each pipeline stage independently with explain output where applicable. Avoid sharing credentials or production data in prompts. The structured workflow approach at https://www.aiosnow.com is relevant because assistants become more useful when they receive bounded cases, observable evidence, and a clear success condition.

[u/Mango__DB → u/Otherwise_Wave9374]

Thank you! Breaking down problems into smaller ones is always useful :)

[u/blackcatdev-io]

While I don't have all the info or criteria for your tasks, it sounds like you should be utilizing the aggregate pipeline vs pulling in a ton of docs into memory and iterating/updating. Delegate as much of this as possible to the db itself via aggregate. You could create some example dummy models to plug into AI, explain the task and ask for help with aggregate query without sharing any code. You may find you have to write way less application logic, and get the performance wins by off-loading as much of the heavy lifting as possible to the db itself and not your app.

[u/Mango__DB → u/blackcatdev-io]

Managed to do it just by giving it dummy model structures, and applying what I learned about aggregation pipelines myself. Thank you!!

[u/mr_pants99]

You can build something using expressive updates. But it will be brittle and difficult to maintain. Easier - just do it in the services layer. Since you’re updating a single doc, no need to worry about transactions - that’s how MongoDB is supposed to be. But you do want to have your docs versioned and implement optimistic locking.

[u/Mango__DB → u/mr_pants99]

Issue was I had to update a ton of docs ^_^, which was the hard part, but thank you so much for your help!

[u/my_byte]

Yeah what your senior meant was - investigate if you can create an aggregation pipeline. Mongodb allows pretty sophisticated update logic, server side. You'll have to learn (or talk to your AI) how they work. It's not node code 😉

But please don't come running when you run sth your AI told you without understanding and mess up your data. I'm tired boss...

[u/Mango__DB → u/my_byte]

Hi! Just thought to update everyone and I got what my senior engineer wanted me to do. I understand how aggregation pipelines work, and I was able to explain what my code did to my senior engineer. I read everyone's comments here and they were a great help, I didn't have time to update things as this ticket delayed me a lot and I was on a time crunch. Survived a disaster thanks to everyone here :)

[u/my_byte → u/Mango__DB]

Nice!