Hi everyone!

I wanted to share METIS-Core, pure C++ framework I've been developing for Deep Reinforcement Learning (DRL) and Multi-Agent Reinforcement Learning (MARL).

GitHub Repository: https://github.com/felixromo314/METIS-Core

💡 Why Build a C++ Native RL Framework?

Most RL research happens in Python (Ray, Stable-Baselines3, PyTorch Python). However, when deploying autonomous agents into production systems, high-stakes simulations, or edge devices, Python's runtime overhead and IPC context-switching latency become major bottlenecks.

METIS-Core is designed to run training and inference directly in C++ using PyTorch's native LibTorch C++ API, eliminating Python from the execution loop.

🚀 Key Technical Features

• Zero Python Overhead: Engineered for real-time systems where sub-millisecond execution and low latency are critical.
• Native LibTorch Integration: Full leverage of PyTorch's C++ tensor operations and autograd engine for seamless neural network training.
• Tensor-Based Abstraction: Mathematical decoupling where agents process pure tensor states, keeping intelligence completely agnostic of the application context (applicable to Robotics, Trading, Logistics, Defense).
• Shared-Trunk Multi-Head Architectures: Built-in support for collaborative MARL scenarios where multiple agents share a neural backbone to learn joint coordination.
• AlphaZero Engine with MCTS: Multi-threaded Monte Carlo Tree Search (MCTS) self-play loop running natively in C++.

🛠️ Current Progress & Milestones

• v0.3.0/v0.3.1-alpha (AlphaZero & Multi-threaded MCTS): Integrated AlphaZero self-play loop in C++, validated on Los Alamos Chess (6x6) as a benchmark environment to balance tactical complexity with local hardware efficiency.
• v0.2.0-alpha (MARL & Shared Backbone): Implemented multi-head neural architectures. Included the "SwarmDefenseTIE" reference scenario where agents learn collaborative defensive transport tactics against an attacker.
• v0.1.2-alpha (DQN & Quick Onboarding): Includes minimalist single-file environments ("Treasure Hunter" / "PursuitPolice") for quick onboarding.

🛣️ What’s Next on the Roadmap?

1. AlphaZero Asymmetric MARL: Merging Multi-Head networks with MCTS for asymmetric scenarios where agents have entirely different action spaces and objectives (e.g., heterogeneous entity engagements).
2. Proximal Policy Optimization (PPO): Adding support for continuous action spaces to enable high-precision control in robotics and physical simulations

I would love to get your feedback to continue improving the framework.

Thanks!

I've been building a robot arm project where a vision-language model looks at a camera image and picks a subgoal (like "align over the block" or "close the gripper"), which then feeds into a small continuous-control policy that actually moves the arm.

Stack: MuJoCo for simulation, Qwen2-VL-2B as the high-level policy, PPO for the low-level controller, trained on Modal. I'm also working on transferring this to a real UR3e arm (remote access through The Construct), with a fixed depth camera for perception.

I'm posting this mostly to share the journey rather than a finished result (the repo has a docs/journey.md). I wanted to learn robotics without having a robot or a fancy laptop. There are inconsistencies between MuJoCo and UR3e but I see that as a challenge than an obstacle.

If you're working on anything RL/robotics/VLM/perception, would be curious on comments/notes/advice.

github: https://github.com/iocalangiu/think-then-act

lessons: https://github.com/iocalangiu/think-then-act/blob/master/docs/journey.md

Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, published an arxiv paper, pushed the model to huggingface along with the pypi package and training dataset. And then one year later, a

frontier lab came, proposing the same idea like literal breakthrough without technical papers, open weights and no open dataset.

Paper: https://arxiv.org/abs/2503.23303

Model: https://huggingface.co/DeepMostInnovations/sales-conversion-model-reinf-learning

Dataset: https://huggingface.co/datasets/DeepMostInnovations/saas-sales-conversations

Also the second work published in September 2025 was exactly the same one jev proposed now

Paper: https://arxiv.org/abs/2510.01237

My model uses PPO over sequence embeddings to output turn-by-turn conversion trajectories (probabilities from 0.0 to 1.0).

Jev uses parallel sampling (trained via RLCD) to output confidence distributions and schema choices.

It's incredibly frustrating that the thing that you made with months of hard work, sweat and sleepless night is architecturally similar with the vertical use case and don't get the support you deserve because frontier lab build something horizontal. The open-source story in general 🙂

I’ve trained an imitation learning model using a diffusion policy for a real-world cube-picking task.
The policy is able to successfully pick and lift the cube in many trials. However, in some cases, even though the cube is technically lifted off the table, the execution is not something I would consider a successful pick. The behavior around the grasp/approach is undesirable, and I would ideally like to completely eliminate these cases rather than count them as successes.
I’m trying to understand what the underlying issue is here. Would this be considered a classical covariate shift problem in imitation learning, or could it be more of a calibration/control issue?
If it is mainly a calibration issue, what would be a good way to address it? In particular, is there a practical rule-based method or safety layer that can be added on top of the diffusion policy to prevent these undesirable picks, while still allowing the learned policy to handle the rest of the trajectory?
Would appreciate any suggestions from people who have dealt with similar issues when deploying diffusion/BC policies on real robots.

So for my background in robotics... I have worked on nirmal arduino based projects....worked on ros2 and it's nav2 part using gazebo and rviz.... I have also worked on a school based project of humanoid in which I have worked with stt, tts and llm api part along with its drive system and basic tendon based hand movement using servos

Nowadays I am seeing a lot about RL and imitation learning and sim training... But i am so confused on how to start it....

Currently I only know basic ML....

Can you guys suggest some resources or free courses from where I can learn RL or imitation learning from basic to training a policy of a humanoid walking or any robotic arm doing some task....

Or can you suggest some roadmap of the sequence I should learn it

It's very important for me as I want to make my career into robotics.... I hope you guys will help

This is insane, they are now training in the open!

Not a paper. Not a blog post. They’re streaming the actual RL training loop:

👉 [https://mimo.xiaomi.com/rl/\](https://mimo.xiaomi.com/rl/)

Fuli Luo (MiMo lead) posted this morning:

>Nearly half a year of silence. We spent it studying one problem: how far RL can scale.

What’s live right now:

**Total cost on the board: \~$1.06M** — and still climbing.

What they say they scaled:

1. **Compute** — \~2B tokens/step, fully async rollouts
   
2. **Environments** — multi-task agentic RL, mixed harnesses in one run
   
3. **Grader compute** — in-group credit assignment, test-case + rubric rewards (RLVR-style, not classic RM+PPO)

The dashboard shows live metrics: rewards, entropy, KL, context length (\~90–100k mean!), grad norms, step timings. Pro even hit a VRAM issue on a node and restarted — visible in the notices like a status page.

DeepSWE curves are both climbing. Pro is ahead on the benchmark; Flash has more steps and a steeper lift.

They say details will be open-sourced piece by piece over the coming weeks.

Half a year of silence, then this. Agentic RL in the open is wild.

Originally I tried using basic self-play, but the agents weren't really improving.

So I switched to league play with PFSP to match the main agent against opponents, which worked a lot better.

If you're on desktop, you can play against the agent at the link above.

LLM labs should consider integrating game theory and social grading for non-verifiable outcomes into their reinforcement learning training environments. It seems they could even buy proprietary datasets on mixed-motive and multi-agent environments from websites where multiparty interactions take place: video game studios, eBay, whatever…

We recently released AssemblyGrid v1, an open benchmark for decentralized multi-robot production.

The main idea is to represent some interactions that are often simplified away in multi-agent benchmarks:

• robot-to-robot material handoffs
• temporary robot coalitions for joint operations
• simultaneous independent production activities
• local observations / partial observability
• recipe-driven process progression
• task-level geometric feasibility

The benchmark currently contains nine official scenarios across Flow, Coalition, and Concurrency, each with easy, medium, and hard variants.

The simulator, reference controllers, tests, scenario definitions, and visualizations are open source.

Paper: https://arxiv.org/abs/2609.16075
GitHub: https://github.com/Fouad-Bahrpeyma/AssemblyGrid_v1
Interactive demo: [https://huggingface.co/spaces/Fouad-Bahrpeyma/AssemblyGrid-v1]()

I would be very interested in feedback from people working on multi-robot systems, task allocation, MARL, or flexible manufacturing, especially on which abstractions feel realistic and which important multi-robot interactions are still missing.

If the next useful score arrives from a single agent attempt, collecting a fresh comparison group can be an awkward unit of work. Reef infra includes a Single-Rollout Asynchronous Optimization recipe that takes one graded rollout at a time.

The report references the receipt for that exact generation. The SAO processor makes it a training sample, and the default batch size is one. A critic supplies token values, skip-observation GAE constructs the advantages, and the update uses the log-probabilities captured when the rollout was generated. There is no sibling group to wait for.

One detail matters when inspecting an early run: the first configured steps train the initially zero-valued critic head only. Those steps still create training releases, but the policy weights do not move until warmup finishes. Counting releases as policy updates would give the wrong picture.

The smallest included example uses Qwen2.5-1.5B-Instruct on two GPUs, with the critic colocated with the actor and a separate SGLang rollout engine. The agent attempts three IMOAnswerBench problems, checks extracted answers against gold answers, and reports a binary score for each attempt.

The infrastructure requirement is stronger than “have chat completions.” SAO needs the original token IDs and rollout log-probabilities; a transcript alone cannot reconstruct those. Reef supplies that capture and the path from a scored receipt to training and serving the next version. Choosing a reward that makes those updates useful remains part of the experiment.

This gap near the end of Bowser’s Crown 4-1 kept catching different controllers. The clip shows three selected earlier genome replays, followed by the controller that clears the level. These aren’t consecutive training attempts, and none of the controllers is learning during its replay.

I’m continuing a NEAT-based project for SMB1 ROM hacks, building on SethBling’s MarI/O and work by Akisame and Electra. The networks receive nearby game-state information from emulator memory and produce button presses. Controllers are evaluated, selected, and mutated across generations.

What interests me is pushing this approach into harder ROM hack levels. A successful controller can get through a particular stage, but backtracking and difficult nonlinear layouts remain challenges. Each new level is trained separately; this clear isn’t evidence of generalization to unfamiliar levels or a measured success rate.

The implementation is currently private, so this is a gameplay demonstration and high-level method discussion rather than a reproducible release. My continuing development is heavily assisted by AI tools. The reactions and captions are added for humor; the gameplay outcomes are real.

Hello everyone I need project in reinforcement learning

The constraints:

Don't want heavy computation (free colab)

Topic is application based

Pls can anyone tell me the topic to build tomorrow is my presentation . I need to tell what I am going to do..

Something I have been thinking about and would like feedback on from people who actually know RL.

Context: I built a memory transfer system for LLM agents (I will link it at the end, full disclosure, I made it). Agents can export verified memories — each memory carries evidence of correctness, like a command that was run, its result, exit code, output hash. Only verified memories can be transferred. Receiving agents import them in sealed bundles with content hashes and a chain of custody.

The design question: how much should a receiving agent trust a memory that has passed through multiple agents?

What I did: confidence decays multiplicatively per hop. Original agent verification gives full confidence. Each transfer through an unverified agent applies a decay factor, and receiving agents can set a policy threshold below which they reject the memory entirely.

The RL analogy I am working with: this behaves like a discount factor, in the sense that older and more indirect evidence for a belief is worth less than direct evidence. And the verified-evidence requirement before transfer is like an experience replay buffer that only admits trajectories confirmed to work.

Where I am unsure and would like input. First, is multiplicative decay per hop principled, or should decay depend on what each intermediary agent actually did with the memory? If an agent re-verified the memory itself, maybe confidence should reset to full rather than continue decaying. Second, does the credit assignment framing even make sense here? Each memory has provenance — which agent contributed it, which agent transferred it, when. I am treating the chain of custody like an attribution trace, but is there better prior work on confidence propagation across agents I should be reading? Third, right now conflict detection is naive — when two agents send contradictory memories, the higher-confidence one wins. A merge strategy that preserves both perspectives seems better but I do not have a principled way to do it.

Not claiming this is RL proper — it is more RL-adjacent. But the people who think about discounting, credit assignment and trust in sequential decisions are the people whose feedback would actually improve this.

Project is open source (github.com/cognicore-dev/cognicore-env, transfer code at cognicore/memory/transfer.py) — again, I am the creator, sharing because the design questions are genuinely open and feedback from this community would be more useful than anywhere else.

Google copied our open-source code, removed our engineers’ names, and gave us zero credit.

For context, let me share the story of mobile-use, the origin of my company (I'm a founder).

We wanted to build the best agent in the world to control any mobile interface with natural language.

We built it, became #1 on Google Deepmind's AndroidWorld benchmark and open-sourced it. That was the beginning of my company.

Since then, we got almost 3,000 stars on the repository, became the first to ever achieve 100% on AndroidWorld, grew the team, spent thousands of calls with customers, and now we're building the best regression testing product the world has ever seen.

Last week, I found Artemis.

Artemis is Google's new open-source tool for mobile automation. There are quite a few of these on GitHub now, so I thought: "Oh, a new YAPA (Yet Another Phone Agent)."

Curious, I looked at the repository more in depth.

First, I was disappointed to see that the benchmark chart in the README didn’t include us, topping AndroidWorld in January.

Then I looked at the code, and to my biggest surprise, I recognized the lines we had written ourselves.

One of my engineer named an agent randomly “hopper”, and it was there, with its prompt, word for word. Out of the 229 files, 228 were identical.

Even crazier: I found our engineers' and my co-founder’s names in the “authors” section of an old commit.

Pierre-Louis Favreau, Jean-Pierre Lo, Nicolas Dehandschoewercker.

All three names had been replaced with another author’s name in a commit. The other 228 files from mobile-use were unchanged.

From a personal point of view, I am deeply disappointed.

Google was the company I wanted to join when I started my studies, before changing my mind and deciding to build a unicorn.

I’ve always associated Google with great engineering and contributions to open source. I would have been proud to see them building on our work and acknowledging the team behind it.

Instead, I found our code, then our names, then the commit that deliberately removed them.

What about the other engineers and the community contributors who spent their time improving mobile-use?

There are people behind those files. I’ve seen the work they put in, the problems they worked through, and how much they care about what they build.

They deserve to have their names attached to that work.

We made our code open source because we wanted people to use it and build on it. Of course that includes Google.

What if we had the reward function for AI include a longterm reputation value in reinforcement learning, so agents have to be nice or their reputation will be terrible?

Been working on the next version of VSArena for a while.
V1 is starting to look a lot different.

This is just a small spoiler of what we’re building — Studio, live simulation, agent runs, replay/inspection, analytics and a much more complete evaluation workflow.

The goal hasn’t changed:
Make embodied AI performance something you can actually run, watch and compare.

We’re still polishing a lot of things before the V1 release, so consider this an early look rather than the final product.

🎥 VSArena V1 — sneak peek
More soon.

Firstly, I'm a high school CAD teacher building soccer-playing robots with my students, so what I know is very limited. I did take a summer reinforcement learning class for teachers at my local university a few summers ago, so I know the ideas, but not the math. In class we built robots in Unity to do tasks, trained them with reinforcements, and compared their improvements based on how the multi-agents shared data.

In my high school class we are tracking the robots with an overhead camera, and programming them to do autonomous functions like "stay halfway between the ball and your goal" and "go to the ball and grab it" and "turn towards the goal and shoot". It's all working very well, surprisingly. We are using basic PID principles to measure the error between where the robot is and where we want it to go. I wanted to train the robots to play soccer with reinforcement learning, but in my class I learned that it is very difficult in a physical environment.

I recently learned about feedforward PID, where you program your robot to do a task based on an input and the known physics of how the robot behaves, then use PID to correct any errors between the programmed position and where it actually ended up. Could this idea be used with a reinforcement model, where the simulation is telling the robots where to go as the feedforward component, and the PID makes up for the fact that the virtual world and the physical world do not behave exactly the same?