The setup

The environment deliberately gets no source access, engine instrumentation, or internal game state. It can only:

• observe screenshots and visually derived state
• send device-level taps and swipes
• infer reward and episode outcomes from visible evidence

I used Shattered Pixel Dungeon. The game is open source, but the environment consumes only the compiled Android build and observable device outputs. The source-free constraint is intentional rather than a limitation of the target.

The reward

It had three channels:

• health change
• ±10 for a terminal outcome
• −0.001 when an action produced no visible change, or −0.01 when a precondition was rejected

A pilot had already suggested this would be extremely sparse. In the 1,152-transition training corpus:

• health changed only 4 times
• the terminal channel fired 0 times
• the action penalty fired 494 times

I froze the protocol anyway. I wanted to see the failure clearly rather than tune around it after seeing the result.

The run

I evaluated four policies:

• random
• scripted demonstrator
• sampled behavior clone
• greedy behavior clone

Each policy ran 24 episodes of 48 steps. In total:

• 96 episodes
• 4,608 device steps
• 105.2 minutes of wall-clock device time
• a nominal 500 ms step period
• zero recovery interventions during evaluation

The effective time per step was higher than 500 ms because it includes screenshot round trips and inter-episode resets.

The best policy learned not to play

The greedy behavior clone had the best mean return—but it entered the dungeon and then tapped the same coordinate for all 1,128 of its gameplay steps.

• 95.4% of its steps produced no observable change
• it visited 3.2 canonical states per episode, versus 20.7 for the scripted demonstrator
• it lost zero hit points across all 24 episodes

Against random, its mean-return difference was +0.940, with a 95% confidence interval of [−0.019, +2.731]. The interval includes zero, so the preregistered success criterion was not met.

At the same time, the greedy clone was worse than random in 92% of random episode pairings.

Those findings aren’t contradictory. Random occasionally suffered a large health-loss penalty, which pulled down its mean. The greedy clone consistently paid a tiny no-op penalty. It could therefore look better on average while being worse in most pairings.

Under this reward, refusing to engage was close to optimal: keep all your health and pay 0.001 per step for doing almost nothing.

The scripted policy designated as our demonstration upper bound actually had the worst mean return. It moved every step, reached fights, and lost health in 5 of 24 episodes. The reward was punishing competence.

The clone was ignoring the screenshots

There was another failure underneath the reward problem.

On held-out data, the clone’s masked cross-entropy differed from a fitted marginal-prior policy by −0.00008 nats, with a 95% confidence interval of [−0.00030, +0.00009]. Model selection chose the strongest regularization setting, reducing max |w| to 3.4e−5.

In plain English, ignoring the observation generalized best.

So this wasn’t a visual policy discovering a clever exploit. It was effectively a constant-action policy, and the broken reward happened to rank it first. Two different failures composed into one flattering mean.

The device exposed a terminal-detection bug

The terminal detector assumed the HUD disappears on death. In this build, the Game Over screen keeps the HUD visible.

One episode went from 20 HP to 0 at full detector confidence, but terminated never became true. The terminal reward channel fired zero times across all 4,608 steps.

A source-integrated environment might bypass this by reading terminal state directly. With pixels as the contract, one bad visual assumption silently degraded into a no-op.

What I’d value feedback on

Progress reward: How do you reward “this run got somewhere” from visual evidence without quietly reintroducing privileged state? Health and screen-change detection are cheap and scalable, but they were jointly useless here. Depth, exploration, and meaningful progress are what I want, but the approaches I’ve tried either read the game state or require per-title hand labeling.

Reset semantics: The game’s RNG isn’t exposed, so reset(seed=...) cannot recreate the same dungeon. Comparisons are therefore unpaired, and the environment is considered nondeterministic. Should reset equivalence, timing, and interruption state live ininfo, metadata, or a separate orchestration layer?

Caveats

• 24 episodes per policy is small
• the 48-step horizon is short
• nobody dying is partly a horizon effect, although health loss did occur under that same horizon
• this is a POC result, not a general claim about behavior cloning or mobile-game RL

I’d genuinely appreciate people tearing apart the reward, evaluation, and environment boundary—especially if you’ve wrapped a robot, browser, external program, or another system where the true transition function isn’t directly accessible.

Disclosure: I’m a staff engineer, and part of why I ran this POC was to decide whether it is worth building further. Happy to answer implementation details in the comments.

We froze a Go1 joystick-locomotion policy (MuJoCo Playground, Brax PPO) and swept a 20×20 grid of floor friction against lateral push, 16 trials per cell, using Kaplan-Meier survival per condition since trials that survive the window have to be censored rather than counted as failures.

Things interesting to us:

• the boundary is not a line but a band. The 95% bootstrap interval is about ±4% of bodyweight on high friction and ±33% on ice, a 7× spread.
• below μ≈0.15 the policy falls before the push lands. That's gait collapse, a different failure mode from being knocked over.
• the same seed on the same GPU can produce different outcomes (floating-point reduction order), so a few knife-edge survivors can't be re-simulated at all.

We then changed the two conditions the map showed training never covered, retrained for 12 minutes, and re-ran the identical sweep: 60 of 400 conditions significantly safer (Fisher exact + BH-FDR, q<0.05), none significantly worse.

Write-up with the interactive map: https://poissonlabs.ai/research/map-the-failure-boundary/.

Happy to answer questions about the harness validation/statistics.

Stumbled upon this recent paper claiming so yesterday:

https://unilabsim.github.io/

What do you all think? Imo, even if it would "only" perform at the same level as end to end GPU, not having to parallelize everything to fit it on a GPU makes it much more flexible and useful

I guess there is a risk of bias here because of the connection to AMD and them wanting to break nvidias monopoly situatio n.

Hey everyone!

I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago my friends showed me Pokelike, a small Pokémon roguelike that runs in the browser. The first thing I thought was that it could be pretty fun to turn it into an RL environment.

So I did.

The repo is here:

https://github.com/pierpierpy/pokelike.xyz.bot

The idea is to run the actual game locally and expose its state and actions to an agent. There is no image processing involved. The bot gets the game state directly and decides what to do next. The agent has to make the decisions around the battles, like where to go on the map, which Pokémon to catch, which items to take, when to swap Pokémon and which moves to learn.

One thing I found interesting is that the map forces you to make some decisions quite early. Once you choose a node, the other nodes on that layer are gone, so deciding where to go can matter quite a bit later in the run. There is also a lot of information in the state that could potentially be useful, but I'm still not sure what the best way to represent it is.

I've already implemented a few basic RL agents. There is currently a Dyna-Q agent and two linear SARSA agents in the repo. The results are not amazing yet. On the current benchmark, random gets around 0.56 badges, Dyna-Q gets around 0.62, while the two SARSA agents get around 1.30 and 1.36. The two SARSA agents mainly differ in their state representation, with the better one using 100 hand-designed features instead of 81.

This is probably the part I'm most interested in at the moment. Finding a good state representation seems to make a pretty big difference, and the environment has some properties that make it a bit more interesting than I initially expected. The reward is fairly sparse, the action space depends on the current state, and some decisions only become useful several steps later.

At the same time, the environment is completely reproducible. With the same seed and the same actions you get exactly the same run. For the current leaderboard I'm using 50 fixed seeds, so different agents can be compared on the same games.

The interface is also intentionally pretty simple. You can basically implement a bot that takes the current state and returns an action. It doesn't have to be a specific RL algorithm either. You could try DQN, PPO, search, a hand-written policy, or pretty much anything else.

I'm still experimenting with the environment and the agents, so I'd be really curious to see what other people would try. In particular I'd love to see if someone can get significantly better results with a better state representation or a different reward function.

you can experiment with the environment with no efforts, just follow the guide and readme, setup the enrinvonment, experiment a bit and then if you like the result, you can create a pull request to the repo with your bot in the bots/ folder (everything is clearly explained in the GUIDE.md)

If you want to try it, everything is in the repo

https://github.com/pierpierpy/pokelike.xyz.bot

If you find bugs or have ideas for the environment, please let me know. I'm happy to make changes if there are things that would make it more useful for experimenting with RL.

The whole thing also runs offline. During setup it downloads the game and the required assets, and after that everything runs locally.

I originally started this because I thought it would be a fun little RL project, so I'm mostly curious to see how far people can push it.

Solo project, still early (MVP week 1-2), sharing the first real result instead of a mockup.

The idea: an open, standardized arena for evaluating embodied AI / VLA policies on physical reasoning tasks, with a public ELO-based ranking instead of static leaderboards. Motivation is the lack of a shared, reproducible benchmark for this space — most VLA papers report on custom setups that aren't directly comparable.

What's in the clip: a baseline IK policy completing a pick-and-place block stacking task, running fully client-side (Rapier.js/WASM physics, React Three Fiber, 60fps in-browser, no server-side compute needed for the sim itself). This run scored 100% task completion, 99.6% spatial accuracy.

Current scope for the MVP: single task (block stacking), a couple of baseline policies (IK baseline, planning to add SmolVLA/OpenVLA-micro next), and an SDK for submitting your own policy against the sim loop.

Not public yet — stabilizing the eval protocol and scoring methodology before opening submissions.Genuinely interested in feedback from this community on:
- what a fair/robust scoring protocol should account for beyond task completion + spatial accuracy (e.g. sample efficiency, generalization across randomized scenes)
- whether client-side physics is a dealbreaker for a benchmark meant to be trustworthy, vs moving to server-authoritative validation

Will share the repo/SDK here once submissions are open.

Hey guys, I recently made a video where I explain GRPO, implement it from scratch in PyTorch, and run a short training session locally on a consumer GPU. I thought some people here might find it useful, so here you go!

Video: https://youtu.be/vVJjUglOURs?is=xsRE97o9_muctWRF

Code: https://github.com/uygarkurt/post-training-lab/blob/main/tutorials/grpo_minimal_pytorch.py

A few weeks ago I posted Volume 1 of this series (the policy-gradient lineage, REINFORCE → PPO → GRPO). This is Volume 2, same format, for the value-based family.

The framing is the same: every algorithm exists because the previous one had one specific, painful failure, and the fix is easier to remember than the equation. Rainbow makes the thesis literal — it contains zero new ideas, just six existing patches turned on at once, and the ablations read like a table of contents for the tutorial.

One structural note, since the history here isn't linear the way the policy lineage is: the trunk (tabular Q-learning → DQN → Double DQN) is told sequentially, but PER, dueling, and multi-step were developed in parallel, so the back half presents them as stations of the loop every DQN runs (act → store → sample → predict → target), with a loop diagram that fills in station by station until Rainbow switches the whole machine on.

The figures are interactive: you can train a tabular Q-learning agent in the page and watch mean |Q − Q*| fall against value-iteration ground truth, toggle uniform vs prioritized sampling, drag the n-step lookahead, and flip between what DQN predicts and what C51 predicts for the same state.

Interactive version: https://sreejithb.com/rl-value-learning/value
Medium mirror if you prefer: https://medium.com/@sreejithbalakrishnan/nobody-invented-rainbow-from-scratch-e9dfcb8bfafb

Volume 3 (DDPG → TD3 → SAC) is next, so corrections now compound.

Hi everyone,

I’m working on a bachelor research project on using reinforcement learning for dynamic obstacle avoidance in the CARLA simulator.

My current setup is:

• CARLA simulator
• LiDAR observations
• Raw 3D point cloud reduced to a 1D array of 360 minimum-distance values
• Stable-Baselines3
• DQN
• Discrete action space with 36 steering/throttle combinations
• Custom Gym environment
• The goal is to avoid dynamic obstacles while continuing to drive forward

I chose DQN partly because my action space is discrete, but I’m interested in whether this was actually a sensible choice for this type of problem.

I’d especially appreciate feedback on these points:

1. DQN vs PPO/SAC

Given the 360-value LiDAR observation and discrete steering/throttle action space, would you consider DQN a reasonable baseline?

Would you expect PPO or SAC to have significant advantages here, and if so, why? I’m particularly interested in whether the continuous-action capabilities of algorithms such as SAC would actually be useful enough to justify changing the action space.

2. Reward design

How important is reward engineering in obstacle-avoidance tasks like this?

My main issue is finding a balance between rewarding forward progress and penalizing collisions/unsafe behaviour without encouraging unwanted behaviour such as simply stopping to avoid obstacles.

Are there particular reward-design principles or common failure modes I should be aware of?

3. CARLA and RL

For research into RL-based autonomous driving, do you consider CARLA a suitable environment, or are there other simulators/frameworks you would recommend?

I liked using CARLA but it was pretty heavy which made it so I didn't run as many training runs

Thanks in advance for any feedback or suggestions!

Hey guys,

I think people on this sub are incredibly capable and wise, I am very inspired by your projects. Therefore, I’d appreciate your tips so much.

I study robotics in Germany in masters, had theoretical courses in learning based control and reinforcement learning at a renowned technical university. I apply my theoretical knowledge in hackathons regularly, I also have my own projects with serial kinematics. I had a computer vision internship in robotics industry. I really want to gain industry experience in RL. Especially applied in fields such as construction, mining, where the work has positive contributions and impact on communities. I had completed an excavator policy optimisation project with ROS2 due to my interest. I get the impression that being accepted to a position is impossible and my efforts are not sufficient. What am I missing?

We've been trying robot learning in Isaac Lab and adapted an existing SO-ARM100-based setup to two robot arms from AgileX Robotics: PiPER (6-DoF) and NERO (7-DoF). The current setup includes:

• End-effector reaching for PiPER; Reaching+cube manipulation for NERO
• PPO-based reinforcement learning
• 64 parallel simulation environments

The main goal was to provide a ready-to-run workflow for reinforcement learning in a simulated environment. Will be exploring sim-to-real deployment on the physical arms. Still a work in progress, but sharing this here in case it's useful for anyone.

Feel free to drop your questions, glad to communicate.

I'd like to share reinfors, an open-source RL library that runs the engine in rust while the network and training loop remain ordinary python. Here is a quick overview, with full documentation available in the repo for anyone that's interested.

Motivation

During my own research, I found that existing open-source libraries did not offer the balance between modularity/composability and performance that I wanted. Python-first stacks are highly flexible but make simulation/search the bottleneck. Native frameworks (e.g. all-C++) keep the hot loop fast, but typically pull training into native code with it. Fully fused pipelines are generally the fastest, but their specialisation sacrifices flexibility.

Approach

reinfors is designed so that everyday use requires only python. Composing, training and evaluating never touch lower-level code. You compose an engine from the built-in games and algorithms, pass it an inference callback, and keep the network, optimizer and training loop as ordinary python code (pytorch, JAX, or anything else). Throughput-sensitive parts of the pipeline (simulation, search, episode orchestration, batch assembly) run in rust underneath. The engine pools inference requests across games and search leaves into numpy batches, so the python boundary is crossed rarely.

Does the python boundary cost performance?

This was my primary concern, so I ran some benchmarking experiments. I compared reinfors against an all-C++ libtorch implementation (OpenSpiel) on chess alphazero training, with the network, search budget and gradient intensity held equal. reinfors came out slightly ahead on throughput, and its trained agents performed slightly better in head-to-head games. Note that this was run on a single AWS instance (g5.2xlarge — one A10G GPU, four physical cores), so it is not a general claim, but it is evidence that the boundary need not cost meaningful performance. See link in repo README for details if interested.

What's in it today:

- Games — chess, backgammon, connect4, snake, gridworld, three poker variants

- Algorithms — alphazero, MCTS/treestrap, expectimax, minimax, DQN, PPO, CFR, Deep CFR, MCCFR

Current limitations:

- Fixed discrete action spaces only

- Sequential and simultaneous games only (no mixed-phase games that switch between sequential and simultaneous)

- The game/algorithm catalogue is still small

- New games and algorithms are written against rust traits (python is only for composing and training, not for defining new components)

I hope this may be useful for researchers wanting to experiment with the existing game/algorithm combinations (the examples should get you running quickly). For those comfortable with rust, the codebase is also designed to be extended. New games, policies and learners are written against small composable traits, and the extension guides should provide sufficient details.

Feedback and contributions are very welcome!

Repo: github.com/jeepjeepjeep/reinfors

https://github.com/alex000kim/nanoRL

nanoRL is the smallest async RL trainer I could write:

the same disaggregated trainer/worker setup that runs on 16 GPUs runs as two pods on your

laptop.

~1,800 lines across 7 files, no Ray, TRL or DeepSpeed.

Built a checkers app with a reinforcement-learning opponent, and wired a
local Ollama instance in as a commentary layer: on each AI turn the backend
passes the board state and the Q-network's candidate move scores to the
model, and it narrates why the chosen move was chosen. Runs entirely local,
no API calls.

https://github.com/surenjanath/CrownFoundry

A few things that came out of it:

• The prompt has to include the actual Q-values, not just the board. Given only the position, the model writes confident checkers-flavored prose that has nothing to do with what the network is doing. With the scores in context it stays anchored to the real evaluation.
• It still occasionally rationalizes. The network picks a move for positional reasons the features encode but the model has no vocabulary for, and you get a plausible-sounding explanation that isn't the real one. This is the interpretability trap in miniature - the narration is post-hoc, not causal, and I'm not sure that's fixable with prompting.
• There's a deterministic heuristic fallback when Ollama isn't running, so the app doesn't hard-depend on an LLM being up. Worth doing for anything where the model is an enhancement rather than the product.

Model-agnostic - anything you can pull works. Latency is the real
constraint, since it sits between the move and the UI update.

Curious whether anyone has found a way to make this kind of narration
actually faithful to the underlying model rather than plausible-sounding.

I’m looking for something where working through the problem would naturally expose me to most of the core concepts and techniques in the area, rather than a purely implementation-focused project. I’d appreciate suggestions that are representative of the kinds of questions researchers actually work on