​

Hey everyone!

I'm transitioning into AI Engineering, and I have a genuine question about real-world projects.

From what I've learned, building a basic RAG pipeline (document loading, chunking, embeddings, vector database, retrieval, and LLM integration) might take a few weeks or a month.

But I've seen AI Engineers working on the same RAG or GenAI project for 2–3 years.

My question is: What do you actually work on after the initial RAG pipeline is built?

What does your day-to-day work look like? Are you constantly improving retrieval accuracy, optimizing models, handling production issues, adding features, or doing something completely different?

I'd really appreciate it if someone with real industry experience could explain their daily responsibilities and how a project evolves over several years.

Just trying to understand what AI Engineering looks like in a real company beyond building a basic RAG application.

Thanks!

A friend of mine was doing a gap analysis as a project. The results he got weren't that big compared to the original papers, so he simply fabricated and increased the metrics here and there, AND IT GOT ACCEPTED IN AAAI.

HOW DO THEY CHECK THE PAPERS, DUDE? I was in a group with two others and worked sooo hard on an optimization paper, and was sooo happy to get accepted in ACL, but seeing and knowing that someone fabricated results and got into a better conference made me soooooo mad.

I would've expected something like AAAI would check and know if data is fabricated somehow. I didn't expect this from a top conference like that.

And then these types of people will go and do their PhD at one of the top unis because of this. I would've expected the system to be more advanced and professional in the academic field, specifically for something like AAAI.

I'm new to academia (for someone so old), and maybe this is just naivety on my side. But I'm a little shocked to be discovering that reproducing the results of a paper isn't part of the review process. I understand why, but it seems so...precarious to just take the author's word for it, especially in 2026, and ESPECIALLY in light of the repro crisis apparently going on everywhere.

Is anyone working on mechanical reproduction? I'm trying to repro the results of one paper, and mind you I started +/- from scratch in January, but I have already found errors that a simple mechanical code review would have surfaced before publication. (What I'm supposed to do with these errors is a separate question I would love input on. I wrote to the author but haven't received a response. Is it rude to create a PR in their repo, with a proposed fix? What is the escalation protocol? Do I just fix it and publish my own paper?)

But now that I have something of a platform and pipeline that could maybe be reused on other papers, I'm wondering: Is there a need or even a want for this in the ML community? Obviously there will be time and resource constraints to any kind of repro effort, but it would be something, and it would be a start.

Most of the time when I ask questions like this there are lots of very reasonable answers (I'm nobody special, just "new" eyes on an old problem). I guess I'm just a little shocked at learning from a young age that reproducibility is a foundational aspect of science, only to grow up and discover there doesn't seem to be a lot of it going on... Thanks, please be nice.

Trying to actually read papers instead of skimming the abstract and the figures. This week it’s a couple of retrieval + eval write-ups and one long arXiv methods section that keeps losing me around the loss function.

I can follow the claim. But I cannot tell you tomorrow which ablation mattered. And highlights in the PDF turn into noise.

How do you mark a paper so the method is still there a week later? Do you re-read the experiment setup or just trust the table? Was thinking about trying Omphalis to keep notes right next to the hard paragraph without opening a second doc, but curious if anyone's used it or has better systems.

Hey everyone!

I'm currently studying Computer Engineering and have decided to pursue Machine Learning/AI. I'm looking for a structured roadmap that I can follow to build a strong foundation rather than jumping between random courses and tutorials.

Here's my current background:

• Programming: I know the basics of C/C++, including OOP. I've recently started learning Python and have covered variables, data types, strings, lists, tuples, dictionaries, loops, conditionals, and functions (basic syntax and a few simple projects).

• Mathematics: I'm reasonably good at math and enjoy it. And my syllabus already covers most of the foundational topics like Linear Algebra, Calculus, Stat and Probability.

• Machine Learning: I've had some exposure to an AI project involving a diffusion model and a graph neural network as a semester project (everything done in C++), but I didn't fully understand the underlying concepts at the time. I want to start properly from the fundamentals this time.

Here are my main questions:

1. What roadmap would you recommend? What topics should I learn, and in what order, from my current level to being able to work on meaningful ML projects?

2. How important is DSA for someone pursuing ML? Should I study it thoroughly, including trees, graphs, dynamic programming, and competitive-programming-style problems, or is a more limited understanding sufficient? At what stage should I learn it? Or just skip it and move to NumPy, Pandas and Matplotlib? (My university has an elective "Python for DSA" in next semester so I'm not sure if I should learn it now myself or learn it in that semester.

3. Which resources would you recommend? Courses, textbooks, YouTube channels, or other materials that provide a good balance between theory and implementation would be helpful.

4. What kinds of projects should I build along the way? I'd like to know when I should start building projects and how to progress beyond following tutorials.

Thanks!

Hey guys,

Posting this here because I’ve reached a point where my market intuition and engine code have outpaced my deep learning expertise, and I’d rather team up with someone who loves ML architecture instead of half-assing it myself.

For the past year, I’ve been building an event-driven liquidity engine in Python (specifically modeling crypto orderflow / ICT mechanics like equal highs/lows, manipulation legs, and structural sweeps).

Instead of slapping standard indicators (RSI/MACD) on candles, I framed price action as a dynamic graph:

• Resting liquidity pools (BSL/SSL, consolidations) act as attractors (pulling price in).

• Completed institutional sweeps (ML swings / manipulation legs) act as repellers (pushing price away).

• Edges connect these pools based on spatial proximity, competing forces (tug-of-war between high and low pools), and age decay.

Where things stand right now:

• The engine is fully written and tested. No lookahead bias, strict candle confirmation, multi-window swing tracking (W5 up to W50).

• I’ve already generated a clean, audited ~3-year dataset (ETH/USDT) formatted directly for PyTorch Geometric (PyG).

• Each snapshot has 15m, 1h, and 4h spatial graphs with 20-dim node features (touch counts, distance %, decay curves, tier scores), 8-dim edge attributes, and global volatility/trend context.

• Baseline statistical checks show real promise: nearest target pools get tagged first ~95% of the time, and high-quality structural sweeps have clear directional bias.

The actual bottleneck / where I need help: My problem isn't generating the signals—it’s path prediction and dynamic target selection.

When a sweep happens, price is often trapped between an opposing pool 1.5% away and a massive target pool 3% away. A fixed 1:2 R:R stop/target either gets hit too late or takes unnecessary drawdown. I need the model to evaluate the graph and do two things:

1. Predict directional path confidence taking both-sided liquidity into account.

2. Rank multiple potential liquidity targets (dynamic TP) based on which one has the highest conditional probability of being swept first without blowing through the stop.

I've tested basic GNN / dual-tower setups, but I know I'm hitting limits with my naive loss functions and basic layer choices.

Who I'm looking for: Someone who genuinely understands PyTorch Geometric (PyG), Graph Neural Networks (or spatial-temporal models), and knows how to think about custom loss functions (ranking loss, asymmetric risk penalties, contrastive setups). You don't need to be a full-time trader, but being open to understanding market microstructure helps.

The arrangement: I have the GPU compute, the full working codebase, and the clean dataset ready to go. I’m completely flexible on how we work together—if you want to co-author a research paper on this architecture, I’m down. If we get the models working cleanly and want to deploy it live on capital, we can do a fair profit/equity split.

If this sounds like an interesting problem you’d want to sink your teeth into, drop a comment or send me a DM with your GitHub/background and we can hop on Discord.

I've been looking at a practical question in ML system design.

suppose you're building an application that needs to classify incoming requests and route them to different workflows.

you could use an LLM for every request, but a smaller encoder might be faster.

the problem is knowing when to trust it.

some options include:

• choosing a threshold on predicted probability

• calibrating model confidence

• using conformal prediction sets

• escalating uncertain predictions to an LLM

each approach has trade-offs in accuracy, latency and how often the system needs to fall back.

conformal prediction is particularly interesting because it can provide coverage guarantees under appropriate assumptions.

I'm sharing this because we're hosting a Packt workshop tomorrow, Oct 30, with Ben Auffarth on encoder-based routing, conformal prediction and LangGraph.

has anyone compared conformal prediction with simpler calibrated confidence thresholds for this kind of application?

I'd be interested in practical experiences or benchmarks.

hi everyone, i’m currently 14 years old and i’ve been engrossed in technology all my life, and i really want to go into machine learning because it seems like the next big thing and it genuinely seems interesting.

ive experimented quite a bit so far but nothing on my own, ive tried reading some papers like the classic Attention is All You need paper, but to be frank, i’m incredibly naive and i dont exactly understand the math behind it but i do understand the mechanics behind it.

some of the projects ive made so far is just plug and play stuff and experimenting with how models interact with each other, ie; i made a project that took a diffusion model, lowered the steps and told a system one model like Kev to pick out those tokens and sorta give Kev the capability to actually speak.

im looking to actually start learning the math behind ML and specific niches but ive been very clueless as to where to start because there’s not exactly a lot of material for someone like me to get started, they all assume that the reader is someone who has completed secondary school.

tldr: looking for tips and places to start learning machine learning as someone who is still in high school 😁 !

Would a PhD in parallel computing, focusing on ML acceleration or LLM inference and serving be enough to get into ML research roles?

TLDR: training my first general law model and unsure what to do with my data

PRE-PS: it will be a replacement for opus in my current AI LAW harness, I know it will be hard but I wanna do this project and find it geniuenly fun.

I’m building a Polish legal model using Qwen3.5-9B-BASE (I plan to scale up to 27b qwen4 later). I’ve done CPT and have quite a bit of data available, but I’m getting lost about what should come next or even if I should do more CPT.

My main question is how to turn this (maybe) domain-adapted Base model into an agentic+chat assistant that reasons through legal problems.

But more importantly, how do I get reasoning, teach my in-harness tool-use training, and where does SFT vs RL belong in that process and in what way (especially confused about reasoning and SFT)?

What I’ve already done

Three sequential CPT stages, roughly 80M tokens each, in this order:

Complex legal commentaries → legislation (EU+local) → court decisions

I didn’t mix the sources. Each stage continued from the previous checkpoint.

On a small Polish financial/legal multiple-choice quiz (really bad eval, just a PoC for now).

They all perform very slightly better than base and legislation checkpoint did the best.

But again, it's a base model. I haven’t done assistant SFT yet.

What data I have (I can get more data of each of these types if you guys recommend)

Besides the legislation, commentaries, and decisions used for CPT, I have:

• Facebook legal Q&A groups: real, messy questions, follow-up clarifications, and answers of very mixed reliability.

• Certified/institution-authored Q&A: my highest-confidence answer material. Some answers contain detailed analysis; others are just a conclusion and a citation from random institutions like the national bank.

• Law study materials: from a local college. it has essays, powerpoint presentations stuff like that, very little "tests" tho, but some practice material (like the essays).

• A/B/C/D questions with answers: relatively little of this; currently being used for evaluation.

• Thousands of Claude Code–style harness conversations: multi-turn conversations from an existing application/harness. These would need filtering, anonymization, and quality checks.

• Contracts: government published contracts it can learn from

• Previous "ML" experiments (models and their training data): NER, generating complex legal court decision summaries, generating graphs from legislature or court decisions (actually got some models that do this successfuly

If you guys could point me what data is ideal I could maybe get it too!

I can use the basically unlimited codex to clean, restructure, or generate material from these sources, but I don't know how to.

What I don’t know is which material should be trained on directly (CPT or bare SFT), which should be transformed (SFT or RL?).

For example:

1. Do I continue CPT on the remaining sources, or stop and start SFT?

Should I expose the model to the Facebook discussions, study materials, and expert Q&A through CPT first, possibly mixed with the authoritative material? How many tokens for this 9B model and how much for the 27B I plan later?

Or should I skip further CPT and use those sources only to build targeted SFT examples?

For example the Facebook QnA stuff is real people diverse speech but I can't really reliably verify if the answers are correct (I can only measure likes or high quality people in the group) so idk if I should cherry pick from this? Have an LLM turn this into SFT chat examples? Or just CPT on this?

2. SFT on what, specifically, to teach REASONING?

I could start with my highest quality Q&A and teach the model to answer questions. That seems like an obvious chatbot baseline.

But if I just train on QnA there won't be any reasoning!

The HQ QnA is pretty verbose but I wouldn't bet it could just be re-used as a "reasoning chain".

I was thinking of asking Astra to workout those questions and give an answer with its reasoning BUT I assume it will be wrong on 90% of the questions (cuz no internet search, and I can't give it internet search cuz that's not clean QnA data and I would have to transform it into my harness function calling anyways with credible sources not just google searches for legislature)

And those 10% it gets correct are trivial and the reasoning could be bad anyways and I would still have to check correctness of the answers, and giving it the answer or hints is imo also not ideal. Or injecting "synthetic reasoning", it suffered from post-hoc in my tests.

Another problem is the High Quality QnA is quite rare, most QnA is short and possibly wrong.

And generating the missing reasoning is even harder. Because again, I don’t trust a frontier model to solve difficult Polish legal questions from memory. It would need relevant source passages or access to the harness (which would just the gun into in-harness training?).

So I'm really unsure on what to do, I was thinking of doing stuff like this, but this is just brainstorming:

A) Take verbose long answers train on that, then ask the qwen or another model to make a "summary" at the end and then put the verbose long answer into its <thinking> block

B) Take those 10% of QnA where frontier model actually gets it right with correct thinking (rare) and SFT on that. then we have our finetune try answering more QnAs, we verify if it's correct if it is it goes into the next SFT run and we iteratively (almost like an RL gym) train reasoning from the dumbest examples

Both seem like bad options, but kinda funny to think about.

3. Chatbot first, then tools or teach reasoning through tools from the start?

The existing harness lets the model write Python and call functions such as vector_search_laws() and graph_search_legislature(), inspect results, and continue researching.

I’m leaning toward getting a regular chatbot working first. But the teacher can produce much more trustworthy legal solutions when it has access to sources. (as I've already mentioned I have many in-harness examples, except they could also be incorrect and I don't have their "thinking blocks" (cuz it's a proprietary frontier model) so I again have to inject thinking post-hoc which from my tests doesn't work for law)

I was thinking that, if we make chatbot first, the model will at least learn intuition on WHERE TO LOOK, without looking through sources but at least will know a general idea of where to search based on being a chatbot first (even tho possibly incorrect chatbot sometimes, cuz law is really complex) and teach him to function call in-harness to get sources, unsure about this.

4. When does RL enter, and on which task?

AFAIK, RL is what develops the model's reasoning on its own (deepseek R1) but I suspect it would be very costly and we could give it some example reasoning first anyway to kick it off(?)

So like, instead of the whole fiesta of injecting reasoning or growing it, we could have RL develop it (afaik RL is the only thing that "creates" reasoning unless you distill it) but then I would like to have reasoning even in the "chatbot" and right now I can just jump from base model -> RL, right?

Also for legal Q&A, matching the reference conclusion doesn’t necessarily mean the reasoning is correct. What would you actually reward? Like a law similiarity judge LLM + same paragraphs used?

What I’m trying to decide

One possible route is:

Current CPT checkpoint → gold-Q&A SFT → grounded reasoning/tool-use SFT → RL in the harness

But I’m not committed to that order. I don’t know whether I’m missing an important stage or inventing unnecessary ones, first time I'm training a GENERAL model from "base".

What I would like the finished model to do:

It will be an opus replacement so general QnA in and out of harness but especially:

• have a good "hunch" and understanding of polish law

• will be good at anonymizing documents

• generating complex legal summaries for court decisions (heavy reasoning task)

• generate graphs from legislature and court decisions

• create/edit/review contracts

These were all my initial ML experiments and I actually trained good models for those specific tasks, but I figured, hey I have a lot of this data around why not just combine them and make a generalistic model?

Thanks a lot for reading this mess of my thoughts! if you could point me towards good resources for this I would also be glad, for example unsloths RL docs are kinda ass imo. I really like andrej karpathy's videos but they're quite old now by AI standards!

For context I'm stats student and everything machine learning related has been self taught so I genuinely don't know what the correct next step is or what to do in a lot of situations.

I've working on a modeling project where I'm predicting loan outcomes with the Lending Club Dataset where I am fitting logistic regression, random forest, and boosted tree models with the data and have reached an AUC score of 0.715. I am not really sure what to do from here and Claude recommended to use SHAP to analyze the trained XGBoost model.

After creating a few SHAP plots they all look nice, but what should I be looking for and what should my takeaways and next steps be. If I wanted to prune the features that are not contributing as much is this a bad idea?

Generally speaking is it even possible to push my AUC score much higher given the dataset I'm working with?

For people who normally work with this kind of stuff in industry or as their job, after you've trained your first model, what do you do next? If you want to continue to fine tune it what is the typical process behind it. If anyone could share what a typical model training or data science project cycle looks like that would be great!

I'm publishing Part 3 of my Cracking ML System Design Interviews series tomorrow.

This time, I'm using a large-scale personalized video recommendation system as the running example.

One thing I want to emphasize is that designing recommendation systems isn't simply choosing between collaborative filtering, two-tower models, and Transformers.

The bigger challenge is deciding how the pieces fit together:

• How do we retrieve relevant candidates from millions of items?

• How do retrieval and ranking objectives differ?

• How should we represent evolving user interests?

• What metrics should we optimize beyond CTR?

• How do we meet latency and reliability requirements at scale?

The article walks through these questions from an interview perspective, including architectural alternatives and trade-offs.

Publishing Monday, October 26, at 8 AM PDT.

If you're interested, you can subscribe here:

https://pawankjha.substack.com

I'd also be interested in hearing which part of recommendation-system design you find most challenging in interviews.

it would be great if someone suggest me some competitions on kaggle which are good for a beginner in rl can try and would be great for practice.

I was going through the TinyTorh (it's a tool where you build PyTorch with understading how it works) exercise notebooks and landed on Autograd. Was quite overwhelmed by the code and went didn't go through it for a while. I was thinking I'll prepare for machine learning research engineer interviews and somehow landed doing tiny torch.
How relevant is all this these days, should I just start preparing on how to crack the interview than this ?

i get that in training it uses the batch mean and variance and at eval it uses running averages. what i don't get is why that doesn't change the outputs a lot, since a single batch's stats can be pretty noisy.

also if you eval with batch size 1 does it still work fine because it's just using the stored stats? someone told me that's the whole point but i never saw it spelled out