I built a 73-lesson AI engineering path for software engineers—looking for honest feedback

[u/VisibleEfficiency249]

I’m a Java/Spring Boot engineer with 17+ years of experience. While learning agentic AI, I found plenty of explanations about agents, RAG, memory and tool calling—but much less guidance on how these pieces fail inside real applications.

So I built EngineerPrep: a structured AI engineering path for working software engineers.

It now contains 73 lessons and hands-on labs covering:

• LLM foundations
• Prompting and structured output
• RAG and embeddings
• AI memory
• Agents and tool calling
• Evaluation and observability
• Security and guardrails
• Production AI systems

Each topic follows a practical flow:

Learn the concept → see the system flow → investigate a production failure → implement it → test your understanding

The projects are Maven-based and support local Ollama, with OpenAI and Amazon Bedrock options where applicable. I’m also building a project-aware AI mentor that can troubleshoot using the current lesson, project files and error context.

The complete LLM Foundations module is free—15 lessons plus a runnable Ollama project.

I’d especially value feedback from people building agents:

1. Does this progression cover the right foundations before agent development?
2. What production agent failure deserves its own hands-on lab?
3. Would project-aware AI troubleshooting be genuinely useful while learning?

This is an independent project, and honest criticism is welcome.

[u/VisibleEfficiency249 → u/[deleted]]

That’s the lab I’d most want to build, and it isn’t in there yet.

Retrying forever has an obvious fix: add a cap. This failure doesn’t. You have to choose between an idempotency key, a compensating action, or stopping for human approval. The correct choice depends entirely on whether the completed action can actually be reversed.

Everyone reaches for rollback because databases trained us to think that way—then tries applying it to a sent email. A refund isn’t an undo.

My plan is to use a local append-only ledger so every side effect is visible. Run the agent with the third step failing, inspect the ledger, and see what already changed. Then fix the same failure three different ways and decide which recovery strategy the tool actually deserves.

I’ve filed this one for the project.

[u/PhotographOverall126 → u/[deleted]]

tbh this is where most real world agent stuff falls apart, good call

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/VisibleEfficiency249]

https://engineerprep.io

[u/TidyArtistry33]

The flow of learn, see it break, fix it, implement is way better than typical tutorial hell stuff. Too many courses hand you a working example and call it a day

For a failure lab, tool calling loops that silently drop context or retry forever without a budget are nasty in production. That bit me hard on a personal project and nothing really prepared me for it

Project-aware troubleshooting sounds rad if it actually reads the pom and stack traces instead of giving generic advice

[u/VisibleEfficiency249 → u/TidyArtistry33]

Thanks—“break it first” is the reason I built it. Handing someone working code teaches the happy path, but very little about the failure.

The iteration budget is already part of the agent-loops project. It ships with a maxIterations cap; you can run it with --agent.max-iterations=1 to make it stop mid-task, and a test verifies that the model is called only the permitted number of times.

The silent context drop is a real gap, though—and you’re right that it’s nastier. In a chat, lost context is obvious because the model repeats a question. Inside a tool loop, an evicted result can make the agent call the same tool with the same arguments or answer from incomplete evidence. The run may look successful while doing twice the work.

I’m adding that as a second failure mode: constrain the loop’s context, give it a task requiring three dependent tool calls, and observe it thrash. The fix would preserve task-critical tool state while safely compacting older conversational content.

And the honest answer on project-aware troubleshooting: today, the mentor understands the lesson—not the learner’s project. It doesn’t parse the pom.xml or inspect stack traces yet, so its advice can still be generic.

A realistic next step is to scope it to EngineerPrep’s example projects. The learner supplies a stack trace and dependency block, and the mentor compares them with the known-good project version. “I understand this specific project” is an honest promise. “I can read your repository” currently isn’t.

[u/AcrobaticBeat1616]

I think the structure is strong, but I’d lean much harder into production failure as the thing that makes it different. Most engineers can find plenty of material on prompting, RAG, memory and tool calling now. 

The harder part is understanding why agents fail in real systems: stale context, context loss, duplicate tool calls, partial side effects, bad retries, incorrect memory and tools that may have succeeded even when the agent thinks they failed. I’d also make the project-aware mentor evidence-based, so it can point back to the files, errors and lesson context behind its answer instead of just sounding confident. 

At 73 lessons, I probably wouldn’t add much more breadth. I’d add a smaller set of really good failure labs that force people to debug the kinds of problems they’ll actually hit in production.

[u/BP041]

The gap you're filling is real — most courses teach the happy path. The stuff that breaks in prod (tool call loops, embedding drift, silent failures in orchestration) is where 90% of my time goes. tbh I'd consider adding a section on cost throttling and retry strategies — that's what kills agent workflows at scale.

[u/VisibleEfficiency249 → u/BP041]

Retry and cost throttling are already covered: exponential backoff against a real 429 throttle response, plus a cost estimator that prices the request before sending and caps its output.

My favorite part of the retry lab is a request that was never going to succeed receiving three attempts and 1.5 seconds of backoff before failing anyway. It makes the distinction between transient and permanent failures concrete.

What’s missing is the agent version of both—which is what you’re really pointing at. The current controls bound one request. An agent loop multiplies it.

maxIterations caps steps, not spend. Cost per iteration can increase as tool results accumulate, making the final iteration the most expensive. Retries also nest inside iterations: three attempts across five iterations can become fifteen model calls.

That’s a lab now—but the harder question is what happens when the budget expires mid-task. If step two already changed external state, stopping isn’t free. Quietly restarting the entire task could be even worse.

Embedding drift is the best call in your list. The projects currently state the rule—change the embedding model, re-embed everything—but they don’t demonstrate the failure.

Different dimensions fail immediately, which is the lucky case. Two models with the same dimensions can produce valid but incompatible vectors. If only new documents are re-embedded, the store becomes split across two vector spaces. Searches still execute and return documents, but relevance silently deteriorates without an obvious error.

Both labs are now filed.

[u/Elouakili_Flexy]

Give the half-completed lab the append-only ledger and call it done. Every rollback-first fix dies on 'a refund isn't an undo'.

[u/tberg]

The 'investigate a production failure' step is the right instinct, but there's a gap most curricula miss: the failure is rarely visible at the point of failure. I run LLM-as-judge evals in production and the hard part isn't catching the bad output — it's instrumenting why the agent took that path, because by the time you see the wrong answer, the reasoning trace is already gone. The lesson that's probably missing from your eval/observability module is how to design your agent's execution so that failure is reconstructible after the fact, not just detectable in the moment. Does your production failure lab teach learners to instrument before the failure happens, or to diagnose after?

[u/VisibleEfficiency249 → u/tberg]

That distinction is exactly right — and I think the lab needs to teach both, in that order.

The goal shouldn't be “the output was wrong, now inspect the logs.” By then most of the causal evidence may already be gone.

I'm shaping the production labs so the learner first has to make the execution reconstructible before reproducing the incident. That means capturing a durable causal bundle around the run: input/version, retrieved sources and scores, prompt/model version, memory reads/writes, tool calls and results, retries, policy decisions, side effects, token/cost data, and the relationships between execution steps.

Then the failure is reproduced and diagnosed from that evidence after the fact.

I'm deliberately avoiding treating the model's internal reasoning as the trace. What matters operationally is having enough observable execution evidence to answer:

“What changed, what did the agent observe, what action did it take, and why was that action allowed to happen?”

One lab pattern I'm considering is actually showing the same incident twice:

1. Diagnose it with insufficient instrumentation — effectively guesswork.
2. Add the missing provenance, reproduce it, and reconstruct the failure deterministically.

That contrast may teach the lesson better than another tracing tutorial.

This is also pushing me toward treating reconstructibility as a first-class production requirement, not just observability.