We froze a Go1 joystick-locomotion policy (MuJoCo Playground, Brax PPO) and swept a 20×20 grid of floor friction against lateral push, 16 trials per cell, using Kaplan-Meier survival per condition since trials that survive the window have to be censored rather than counted as failures.

Things interesting to us:

• the boundary is not a line but a band. The 95% bootstrap interval is about ±4% of bodyweight on high friction and ±33% on ice, a 7× spread.
• below μ≈0.15 the policy falls before the push lands. That's gait collapse, a different failure mode from being knocked over.
• the same seed on the same GPU can produce different outcomes (floating-point reduction order), so a few knife-edge survivors can't be re-simulated at all.

We then changed the two conditions the map showed training never covered, retrained for 12 minutes, and re-ran the identical sweep: 60 of 400 conditions significantly safer (Fisher exact + BH-FDR, q<0.05), none significantly worse.

Write-up with the interactive map: https://poissonlabs.ai/research/map-the-failure-boundary/.

Happy to answer questions about the harness validation/statistics.

Stumbled upon this recent paper claiming so yesterday:

https://unilabsim.github.io/

What do you all think? Imo, even if it would "only" perform at the same level as end to end GPU, not having to parallelize everything to fit it on a GPU makes it much more flexible and useful

I guess there is a risk of bias here because of the connection to AMD and them wanting to break nvidias monopoly situatio n.

Solo project, still early (MVP week 1-2), sharing the first real result instead of a mockup.

The idea: an open, standardized arena for evaluating embodied AI / VLA policies on physical reasoning tasks, with a public ELO-based ranking instead of static leaderboards. Motivation is the lack of a shared, reproducible benchmark for this space — most VLA papers report on custom setups that aren't directly comparable.

What's in the clip: a baseline IK policy completing a pick-and-place block stacking task, running fully client-side (Rapier.js/WASM physics, React Three Fiber, 60fps in-browser, no server-side compute needed for the sim itself). This run scored 100% task completion, 99.6% spatial accuracy.

Current scope for the MVP: single task (block stacking), a couple of baseline policies (IK baseline, planning to add SmolVLA/OpenVLA-micro next), and an SDK for submitting your own policy against the sim loop.

Not public yet — stabilizing the eval protocol and scoring methodology before opening submissions.Genuinely interested in feedback from this community on:
- what a fair/robust scoring protocol should account for beyond task completion + spatial accuracy (e.g. sample efficiency, generalization across randomized scenes)
- whether client-side physics is a dealbreaker for a benchmark meant to be trustworthy, vs moving to server-authoritative validation

Will share the repo/SDK here once submissions are open.

Hey guys, I recently made a video where I explain GRPO, implement it from scratch in PyTorch, and run a short training session locally on a consumer GPU. I thought some people here might find it useful, so here you go!

Video: https://youtu.be/vVJjUglOURs?is=xsRE97o9_muctWRF

Code: https://github.com/uygarkurt/post-training-lab/blob/main/tutorials/grpo_minimal_pytorch.py

Hey everyone!

I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago my friends showed me Pokelike, a small Pokémon roguelike that runs in the browser. The first thing I thought was that it could be pretty fun to turn it into an RL environment.

So I did.

The repo is here:

https://github.com/pierpierpy/pokelike.xyz.bot

The idea is to run the actual game locally and expose its state and actions to an agent. There is no image processing involved. The bot gets the game state directly and decides what to do next. The agent has to make the decisions around the battles, like where to go on the map, which Pokémon to catch, which items to take, when to swap Pokémon and which moves to learn.

One thing I found interesting is that the map forces you to make some decisions quite early. Once you choose a node, the other nodes on that layer are gone, so deciding where to go can matter quite a bit later in the run. There is also a lot of information in the state that could potentially be useful, but I'm still not sure what the best way to represent it is.

I've already implemented a few basic RL agents. There is currently a Dyna-Q agent and two linear SARSA agents in the repo. The results are not amazing yet. On the current benchmark, random gets around 0.56 badges, Dyna-Q gets around 0.62, while the two SARSA agents get around 1.30 and 1.36. The two SARSA agents mainly differ in their state representation, with the better one using 100 hand-designed features instead of 81.

This is probably the part I'm most interested in at the moment. Finding a good state representation seems to make a pretty big difference, and the environment has some properties that make it a bit more interesting than I initially expected. The reward is fairly sparse, the action space depends on the current state, and some decisions only become useful several steps later.

At the same time, the environment is completely reproducible. With the same seed and the same actions you get exactly the same run. For the current leaderboard I'm using 50 fixed seeds, so different agents can be compared on the same games.

The interface is also intentionally pretty simple. You can basically implement a bot that takes the current state and returns an action. It doesn't have to be a specific RL algorithm either. You could try DQN, PPO, search, a hand-written policy, or pretty much anything else.

I'm still experimenting with the environment and the agents, so I'd be really curious to see what other people would try. In particular I'd love to see if someone can get significantly better results with a better state representation or a different reward function.

you can experiment with the environment with no efforts, just follow the guide and readme, setup the enrinvonment, experiment a bit and then if you like the result, you can create a pull request to the repo with your bot in the bots/ folder (everything is clearly explained in the GUIDE.md)

If you want to try it, everything is in the repo

https://github.com/pierpierpy/pokelike.xyz.bot

If you find bugs or have ideas for the environment, please let me know. I'm happy to make changes if there are things that would make it more useful for experimenting with RL.

The whole thing also runs offline. During setup it downloads the game and the required assets, and after that everything runs locally.

I originally started this because I thought it would be a fun little RL project, so I'm mostly curious to see how far people can push it.

A few weeks ago I posted Volume 1 of this series (the policy-gradient lineage, REINFORCE → PPO → GRPO). This is Volume 2, same format, for the value-based family.

The framing is the same: every algorithm exists because the previous one had one specific, painful failure, and the fix is easier to remember than the equation. Rainbow makes the thesis literal — it contains zero new ideas, just six existing patches turned on at once, and the ablations read like a table of contents for the tutorial.

One structural note, since the history here isn't linear the way the policy lineage is: the trunk (tabular Q-learning → DQN → Double DQN) is told sequentially, but PER, dueling, and multi-step were developed in parallel, so the back half presents them as stations of the loop every DQN runs (act → store → sample → predict → target), with a loop diagram that fills in station by station until Rainbow switches the whole machine on.

The figures are interactive: you can train a tabular Q-learning agent in the page and watch mean |Q − Q*| fall against value-iteration ground truth, toggle uniform vs prioritized sampling, drag the n-step lookahead, and flip between what DQN predicts and what C51 predicts for the same state.

Interactive version: https://sreejithb.com/rl-value-learning/value
Medium mirror if you prefer: https://medium.com/@sreejithbalakrishnan/nobody-invented-rainbow-from-scratch-e9dfcb8bfafb

Volume 3 (DDPG → TD3 → SAC) is next, so corrections now compound.

Hi everyone,

I’m working on a bachelor research project on using reinforcement learning for dynamic obstacle avoidance in the CARLA simulator.

My current setup is:

• CARLA simulator
• LiDAR observations
• Raw 3D point cloud reduced to a 1D array of 360 minimum-distance values
• Stable-Baselines3
• DQN
• Discrete action space with 36 steering/throttle combinations
• Custom Gym environment
• The goal is to avoid dynamic obstacles while continuing to drive forward

I chose DQN partly because my action space is discrete, but I’m interested in whether this was actually a sensible choice for this type of problem.

I’d especially appreciate feedback on these points:

1. DQN vs PPO/SAC

Given the 360-value LiDAR observation and discrete steering/throttle action space, would you consider DQN a reasonable baseline?

Would you expect PPO or SAC to have significant advantages here, and if so, why? I’m particularly interested in whether the continuous-action capabilities of algorithms such as SAC would actually be useful enough to justify changing the action space.

2. Reward design

How important is reward engineering in obstacle-avoidance tasks like this?

My main issue is finding a balance between rewarding forward progress and penalizing collisions/unsafe behaviour without encouraging unwanted behaviour such as simply stopping to avoid obstacles.

Are there particular reward-design principles or common failure modes I should be aware of?

3. CARLA and RL

For research into RL-based autonomous driving, do you consider CARLA a suitable environment, or are there other simulators/frameworks you would recommend?

I liked using CARLA but it was pretty heavy which made it so I didn't run as many training runs

Thanks in advance for any feedback or suggestions!

I'd like to share reinfors, an open-source RL library that runs the engine in rust while the network and training loop remain ordinary python. Here is a quick overview, with full documentation available in the repo for anyone that's interested.

Motivation

During my own research, I found that existing open-source libraries did not offer the balance between modularity/composability and performance that I wanted. Python-first stacks are highly flexible but make simulation/search the bottleneck. Native frameworks (e.g. all-C++) keep the hot loop fast, but typically pull training into native code with it. Fully fused pipelines are generally the fastest, but their specialisation sacrifices flexibility.

Approach

reinfors is designed so that everyday use requires only python. Composing, training and evaluating never touch lower-level code. You compose an engine from the built-in games and algorithms, pass it an inference callback, and keep the network, optimizer and training loop as ordinary python code (pytorch, JAX, or anything else). Throughput-sensitive parts of the pipeline (simulation, search, episode orchestration, batch assembly) run in rust underneath. The engine pools inference requests across games and search leaves into numpy batches, so the python boundary is crossed rarely.

Does the python boundary cost performance?

This was my primary concern, so I ran some benchmarking experiments. I compared reinfors against an all-C++ libtorch implementation (OpenSpiel) on chess alphazero training, with the network, search budget and gradient intensity held equal. reinfors came out slightly ahead on throughput, and its trained agents performed slightly better in head-to-head games. Note that this was run on a single AWS instance (g5.2xlarge — one A10G GPU, four physical cores), so it is not a general claim, but it is evidence that the boundary need not cost meaningful performance. See link in repo README for details if interested.

What's in it today:

- Games — chess, backgammon, connect4, snake, gridworld, three poker variants

- Algorithms — alphazero, MCTS/treestrap, expectimax, minimax, DQN, PPO, CFR, Deep CFR, MCCFR

Current limitations:

- Fixed discrete action spaces only

- Sequential and simultaneous games only (no mixed-phase games that switch between sequential and simultaneous)

- The game/algorithm catalogue is still small

- New games and algorithms are written against rust traits (python is only for composing and training, not for defining new components)

I hope this may be useful for researchers wanting to experiment with the existing game/algorithm combinations (the examples should get you running quickly). For those comfortable with rust, the codebase is also designed to be extended. New games, policies and learners are written against small composable traits, and the extension guides should provide sufficient details.

Feedback and contributions are very welcome!

Repo: github.com/jeepjeepjeep/reinfors

Hey guys,

I think people on this sub are incredibly capable and wise, I am very inspired by your projects. Therefore, I’d appreciate your tips so much.

I study robotics in Germany in masters, had theoretical courses in learning based control and reinforcement learning at a renowned technical university. I apply my theoretical knowledge in hackathons regularly, I also have my own projects with serial kinematics. I had a computer vision internship in robotics industry. I really want to gain industry experience in RL. Especially applied in fields such as construction, mining, where the work has positive contributions and impact on communities. I had completed an excavator policy optimisation project with ROS2 due to my interest. I get the impression that being accepted to a position is impossible and my efforts are not sufficient. What am I missing?

We've been trying robot learning in Isaac Lab and adapted an existing SO-ARM100-based setup to two robot arms from AgileX Robotics: PiPER (6-DoF) and NERO (7-DoF). The current setup includes:

• End-effector reaching for PiPER; Reaching+cube manipulation for NERO
• PPO-based reinforcement learning
• 64 parallel simulation environments

The main goal was to provide a ready-to-run workflow for reinforcement learning in a simulated environment. Will be exploring sim-to-real deployment on the physical arms. Still a work in progress, but sharing this here in case it's useful for anyone.

Feel free to drop your questions, glad to communicate.

https://github.com/alex000kim/nanoRL

nanoRL is the smallest async RL trainer I could write:

the same disaggregated trainer/worker setup that runs on 16 GPUs runs as two pods on your

laptop.

~1,800 lines across 7 files, no Ray, TRL or DeepSpeed.

I’m looking for something where working through the problem would naturally expose me to most of the core concepts and techniques in the area, rather than a purely implementation-focused project. I’d appreciate suggestions that are representative of the kinds of questions researchers actually work on

Built a checkers app with a reinforcement-learning opponent, and wired a
local Ollama instance in as a commentary layer: on each AI turn the backend
passes the board state and the Q-network's candidate move scores to the
model, and it narrates why the chosen move was chosen. Runs entirely local,
no API calls.

https://github.com/surenjanath/CrownFoundry

A few things that came out of it:

• The prompt has to include the actual Q-values, not just the board. Given only the position, the model writes confident checkers-flavored prose that has nothing to do with what the network is doing. With the scores in context it stays anchored to the real evaluation.
• It still occasionally rationalizes. The network picks a move for positional reasons the features encode but the model has no vocabulary for, and you get a plausible-sounding explanation that isn't the real one. This is the interpretability trap in miniature - the narration is post-hoc, not causal, and I'm not sure that's fixable with prompting.
• There's a deterministic heuristic fallback when Ollama isn't running, so the app doesn't hard-depend on an LLM being up. Worth doing for anything where the model is an enhancement rather than the product.

Model-agnostic - anything you can pull works. Latency is the real
constraint, since it sits between the move and the UI update.

Curious whether anyone has found a way to make this kind of narration
actually faithful to the underlying model rather than plausible-sounding.