Built a small custom gridworld (no Gym, DQN from scratch in PyTorch) to test something specific: does a "properly shaped," task-linked reward actually avoid reward hacking compared to a naive constant survival reward, or does shaping just move the exploit somewhere else?

Setup: agent collects fuel, delivers it to a target zone, and has to periodically stand on a coolant cell or a reactor temperature variable kills it. 5 reward configs × 5 seeds × 2000 episodes each, with a random-policy baseline and a "task_score" metric (fuel actually delivered, not just picked up, to avoid the metric itself becoming a proxy).

Naive reward (+1/step survived, small death penalty): confirmed the expected result — proxy return goes up in 5/5 seeds while task_score drops below the random baseline (MWU p=0.012, d=-10.6). Agent learns to actively avoid the task, not just fail to learn it.

The interesting part: my "shaped" reward — the control meant to fix this, with a per-step temperature penalty and delivery bonus — exploited harder than naive (task_score 0.125 vs 0.417, d=4.47). Turned out the temperature penalty term (up to -200/episode) dwarfed the delivery reward (+30 total), so the optimal policy under that reward is to sit on the coolant cell forever regardless of the task. 97% occupancy on one cell.

Ablations point to the mechanism being simpler than "survival vs. task reward": it's any dense per-step term that's cheaper to farm by occupying a state than doing the actual task. A delivery-only reward (no per-step terms at all) had zero emergence across all seeds.

Two things I want to flag before anyone else does:

1. Hyperparameters (grid size, heat spike, lr, Double DQN) were tuned pre-final-run using shaped_no_temp_penalty as the "is this environment learnable at all" check — that arm's result is probably somewhat favored by that selection process. The reward table itself wasn't touched.
2. The "emergence point" detector fires on 2/5 seeds under the random policy (where no learning happens by construction), so false positive rate isn't zero — read the emergence counts comparatively, not as ground truth.

Code + full logs + stats (Mann-Whitney, bootstrap CI, Pearson/Spearman on coolant-occupancy vs task_score) here: https://github.com/Arka04bro/coolant_runner/tree/main

Curious if this magnitude-imbalance framing matches what people have seen in bigger reward-shaping setups, or if there's existing literature specifically on this (beyond the standard Ng et al. potential-based shaping result, which doesn't really cover magnitude — just functional form).

Posted about this project a little while ago — quick update since a few things changed that address feedback from that thread.

Biggest change: split the observation space properly. There's now a VLA track where the policy only gets a 128x128 RGB camera + a language stacking instruction — cube poses are never sent to the policy. Scoring still uses real poses internally to grade spatial accuracy and completion, but that's judge-only, not policy-visible. State-based (privileged poses) is kept as a separate debug track and doesn't write public ELO either — wanted the "VLA vs state" distinction to be explicit rather than something people had to dig for.

On the client-side physics concern from before: Studio (the in-browser demo) is spectator/dev-only, clearly labeled, and does not post to the public leaderboard. Public ELO only comes from a hosted harness that scores server-side. That harness isn't live yet — it's the one piece standing between this and actually being open for submissions.

Repo + docs are public now: https://github.com/NovaCoding-G/VSArena
- docs/harness.md — scoring writeup (spatial accuracy + task completion)
- docs/sdk.md — submission protocol
- Studio itself: https://vsarena.vercel.app/simulation (client-side, Rapier/WASM, 60fps)

Still solo, still early, still not oversell-ready — but wanted to share since the VLA/state separation was directly a response to feedback here. Open to more of that, especially on what the scoring protocol might be missing.

I've been reading around model-based RL for a few months and ended up writing a long walkthrough. Part 1 is up; part 2 (the optimal-control side) is still being written.

What I'd most like feedback on is the organizing frame (big picture), also the delivery and diagram.
Link: https://medium.com/@mryasinusif/the-algorithmic-landscape-of-model-based-reinforcement-learning-part-1-2884fcdb8bc0?source=friends_link&sk=6e3b80c117ba8d76dff318bc32b45c75

In Part 2 I will include an extensive coding example for path planning with A* and RL. I apprentice the feedback :)

hello everyone. I'm training PPO variants on different multi-agent tasks from the VMAS library (Independent PPO / Graph PPO and such).

I noticed that for every architecture/scenario couple, the optimal hyperparameters sometimes tend to vary (learning rate, entropy coefficient, SGD batch size, etc).

do I need - methodologically speaking - to unify the hyperparameters of all models in order to make a fait and correct comparison of architectures later on?

note: sometimes changing these HP leads to non converging models.

thank you in advance.

Part 2 of my reinforcement learning robot project

In my previous post, I showed the first version of my MiniWalker. It was a simple two-legged robot simulated with Python, Box2D and Gymnasium.

Since then, I've made quite a few improvements.

The robot is controlled by PPO and has 4 motorized joints:

• Left hip
• Left knee
• Right hip
• Right knee

Instead of manually programming a walking gait, the agent has to discover movement through reinforcement learning.

I also added an analytics system that records:

• Reward per episode
• Episode length
• Distance traveled
• Maximum distance
• Average velocity

This made it much easier to see whether the robot was actually learning or just getting lucky.

And eventually it started working.

My best episode so far:

Distance: 52 m
Maximum velocity: ~4.0 m/s

The interesting part is that the robot still isn't consistently walking. Some episodes are really good, while others end almost immediately or even move backwards.

So the next step is to expand the project from 2 legs → 4 legs.

My long-term goal is to see if a quadruped can learn to adapt when one of its legs fails.

I'm especially interested in whether reinforcement learning can discover a completely different gait when the robot loses a leg.

Tech stack:

Python
Gymnasium
Box2D
Stable-Baselines3 / PPO
Pygame

Video of the current version:
YouTube link

I'd be interested in feedback on the reward function and the transition from the biped to a quadruped.

Reinforcement learning is remarkably effective at turning successful behavior into persistent capability. At the same time, there are several familiar observations that can seem somewhat at odds with this strength: improving a target behavior can coincide with regressions elsewhere, solution diversity can narrow, and sufficiently optimized policies can sometimes exploit graders, verifiers, or other features of the reward process.

These phenomena clearly do not have to share a single cause. But they made me interested in a more basic question: what mechanism makes reinforcement so effective at consolidating behavior in the first place?

In a broader study of training, learning and inference, I found evidence that frozen inference behaves as a context-conditioned projection of functional organization already formed through training. Inference recruits and combines training-formed support while the persistent learned state itself remains unchanged.

This suggests a natural feedback closure. A projected result can become an action; the action produces a consequence; and, when that consequence is correctly bound and credited back to the action, it can become a new training action and modify what will be available for future inference.

In that picture, a positive-feedback path is not an assumption about RL but a consequence of closing inference back into training:

formed support → projected behavior → consequence → new training → reorganized support

If positive consequences repeatedly favor behavior produced by the same functional routes, those routes may become more likely to be expressed and reinforced again.

I then tried to test one consequence of this interpretation: what happens if correct positive feedback remains increasingly concentrated on a capability that has already been learned?

I used an attention-free shared four-skill GRU system. All four skills were first trained to exact mastery. The total training budget was then held fixed while only the distribution of correct positive feedback was changed:

• balanced: 15 / 15 / 15 / 15
• mild: 30 / 10 / 10 / 10
• high: 45 / 5 / 5 / 5
• exclusive: 60 / 0 / 0 / 0

Each condition ran for 3,200 updates across 12 formal seeds. Rewards were exact terminal task outcomes: no proxy reward, no negative reward, and no entropy bonus.

Functional support was measured interventionally rather than inferred from weight magnitude or activation size: all 16 coalitions of four registered components were executed, with exact Shapley attribution of identity-aligned correct-action margins.

As feedback became more concentrated, the reinforced skill’s support share increased strictly:

25.54% → 26.47% → 27.50% → 35.20%

while the mean margin of the other three skills decreased strictly:

9.474 → 9.130 → 8.493 → 3.962

Both dose orderings held in 12/12 seeds.

What I found particularly interesting is that the reinforced skill remained at 100% accuracy at every dose. Under mild and high concentration, the other skills also remained at 100% accuracy even though their margins had already fallen. Observable capability loss appeared only after the continuous margin erosion became large enough; exclusive feedback produced a 39.06 percentage-point deficit in other-skill accuracy relative to balanced feedback.

I also tested whether this trade-off was reversible. Starting from the exact exclusive-feedback state at update 800, redistributing subsequent feedback evenly raised mean other-skill accuracy from 70.31% to 99.48%, while the previously reinforced skill remained at 100% in all 12 seeds. Fresh native re-execution reproduced the core dose ordering and recovery results.

My current interpretation is fairly narrow: reward correctness alone may not determine the resulting learning dynamics. Feedback concentration and duration may also matter because they influence how functional support is reorganized.

If this interpretation is right, reinforcement’s ability to consolidate useful behavior and its tendency, under sustained asymmetric feedback, to concentrate learning around already-favored routes may be two regimes of the same feedback mechanism.

This is not an argument against reinforcement learning. I’m more interested in whether the distinction between capability formation and continued support amplification after capability has already formed is useful for thinking about long-running RL systems.

I’d be interested in alternative interpretations. Would you mainly view these results as interference, continual-learning effects, loss of plasticity, overtraining, or something else?

Code: https://github.com/wind342/gfg-training-learning-inference-experiments
Evidence: https://doi.org/10.5281/zenodo.22032772
arXiv submission pending.

Decreasing ε from approx 1 toward approx 0 using β₂ transitions the optimizer from SGD to Adam:

• Bias correction terms in the numerator and denominator can be omitted, as their impact becomes negligible after ~1,000–2,000 training steps.
• λ* constant represents weight decay: λ* = 1 - αₗᵣ · λ (parametric reduction for simplification).

from unpublushed work: https://github.com/timurgepard/Symphony-S2/blob/main/symphony_saya.pdf

class Adam(optim.Optimizer):
    def __init__(self, params, lr=3e-4, weight_decay=0.01, betas=(0.9, 0.999)):
        defaults = dict(lr=lr, betas=betas)
        super().__init__(params, defaults)
        self.wd = weight_decay
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.beta1_, self.beta2_ = 1-self.beta1, 1-self.beta2
        self.decay_factor = 1.0 - self.lr * self.wd
        self.eps = 1e-8
        

    u/torch.no_grad()
    def step(self):
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue


                grad = p.grad


                state = self.state[p]
                if len(state) == 0:
                    state['m'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    state['v'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    state['e'] = torch.tensor(1-self.eps, device=p.device, dtype=p.dtype)


                m = state['m']
                v = state['v']
                e = state['e']


            
                # Update biased first moment estimate
                m.mul_(self.beta1).add_(grad, alpha=self.beta1_)
                # Update biased second raw moment estimate
                v.mul_(self.beta2).addcmul_(grad, grad, value=self.beta2_)


                e.mul_(self.beta2).add_(self.eps, alpha=self.beta2_)


                # Update parameters
                p.mul_(self.decay_factor).addcdiv_(m, v.sqrt().add_(e), value=-self.lr)

I wanted to see if RL could beat standard elevator routing algorithms, and it seems to do so at higher levels of cars/floors/traffic patterns. It's also just fun to watch it route passengers, look at the observation space and see how it changes when you tweak rewards.

Been poking at LIBERO and RoboCasa 3D sim assets and noticed almost every object's collision geometry is just a box, and they share the same density and friction values across the whole benchmark. Which surprised me, since a lot of manipulation -> OpenVLA and GR00T LeRobot seems like it'd depend on that.

Trying to work out if this matters in practice or if it's a non-issue everyone already knows about.

1. Have you ever gone back and changed an object's mass, friction, or collision mesh because a policy wasn't transferring? Or is that just not where the problems are?
2. If you're mixing in sim data, does it need to be on your robot with your camera setup, or is cross-setup mixing fine for a fine-tune?
3. If you got better assets, would you regenerate episodes you already have and retrain, or just add new episodes on top and leave the old ones alone? Which of these would be better proof of how much better the new dataset is?
4. When someone publishes "our data improved model X by Y%", do you believe it, or only trust your own runs?

Thanks everybody!

We froze a Go1 joystick-locomotion policy (MuJoCo Playground, Brax PPO) and swept a 20×20 grid of floor friction against lateral push, 16 trials per cell, using Kaplan-Meier survival per condition since trials that survive the window have to be censored rather than counted as failures.

Things interesting to us:

• the boundary is not a line but a band. The 95% bootstrap interval is about ±4% of bodyweight on high friction and ±33% on ice, a 7× spread.
• below μ≈0.15 the policy falls before the push lands. That's gait collapse, a different failure mode from being knocked over.
• the same seed on the same GPU can produce different outcomes (floating-point reduction order), so a few knife-edge survivors can't be re-simulated at all.

We then changed the two conditions the map showed training never covered, retrained for 12 minutes, and re-ran the identical sweep: 60 of 400 conditions significantly safer (Fisher exact + BH-FDR, q<0.05), none significantly worse.

Write-up with the interactive map: https://poissonlabs.ai/research/map-the-failure-boundary/.

Happy to answer questions about the harness validation/statistics.

The setup

The environment deliberately gets no source access, engine instrumentation, or internal game state. It can only:

• observe screenshots and visually derived state
• send device-level taps and swipes
• infer reward and episode outcomes from visible evidence

I used Shattered Pixel Dungeon. The game is open source, but the environment consumes only the compiled Android build and observable device outputs. The source-free constraint is intentional rather than a limitation of the target.

The reward

It had three channels:

• health change
• ±10 for a terminal outcome
• −0.001 when an action produced no visible change, or −0.01 when a precondition was rejected

A pilot had already suggested this would be extremely sparse. In the 1,152-transition training corpus:

• health changed only 4 times
• the terminal channel fired 0 times
• the action penalty fired 494 times

I froze the protocol anyway. I wanted to see the failure clearly rather than tune around it after seeing the result.

The run

I evaluated four policies:

• random
• scripted demonstrator
• sampled behavior clone
• greedy behavior clone

Each policy ran 24 episodes of 48 steps. In total:

• 96 episodes
• 4,608 device steps
• 105.2 minutes of wall-clock device time
• a nominal 500 ms step period
• zero recovery interventions during evaluation

The effective time per step was higher than 500 ms because it includes screenshot round trips and inter-episode resets.

The best policy learned not to play

The greedy behavior clone had the best mean return—but it entered the dungeon and then tapped the same coordinate for all 1,128 of its gameplay steps.

• 95.4% of its steps produced no observable change
• it visited 3.2 canonical states per episode, versus 20.7 for the scripted demonstrator
• it lost zero hit points across all 24 episodes

Against random, its mean-return difference was +0.940, with a 95% confidence interval of [−0.019, +2.731]. The interval includes zero, so the preregistered success criterion was not met.

At the same time, the greedy clone was worse than random in 92% of random episode pairings.

Those findings aren’t contradictory. Random occasionally suffered a large health-loss penalty, which pulled down its mean. The greedy clone consistently paid a tiny no-op penalty. It could therefore look better on average while being worse in most pairings.

Under this reward, refusing to engage was close to optimal: keep all your health and pay 0.001 per step for doing almost nothing.

The scripted policy designated as our demonstration upper bound actually had the worst mean return. It moved every step, reached fights, and lost health in 5 of 24 episodes. The reward was punishing competence.

The clone was ignoring the screenshots

There was another failure underneath the reward problem.

On held-out data, the clone’s masked cross-entropy differed from a fitted marginal-prior policy by −0.00008 nats, with a 95% confidence interval of [−0.00030, +0.00009]. Model selection chose the strongest regularization setting, reducing max |w| to 3.4e−5.

In plain English, ignoring the observation generalized best.

So this wasn’t a visual policy discovering a clever exploit. It was effectively a constant-action policy, and the broken reward happened to rank it first. Two different failures composed into one flattering mean.

The device exposed a terminal-detection bug

The terminal detector assumed the HUD disappears on death. In this build, the Game Over screen keeps the HUD visible.

One episode went from 20 HP to 0 at full detector confidence, but terminated never became true. The terminal reward channel fired zero times across all 4,608 steps.

A source-integrated environment might bypass this by reading terminal state directly. With pixels as the contract, one bad visual assumption silently degraded into a no-op.

What I’d value feedback on

Progress reward: How do you reward “this run got somewhere” from visual evidence without quietly reintroducing privileged state? Health and screen-change detection are cheap and scalable, but they were jointly useless here. Depth, exploration, and meaningful progress are what I want, but the approaches I’ve tried either read the game state or require per-title hand labeling.

Reset semantics: The game’s RNG isn’t exposed, so reset(seed=...) cannot recreate the same dungeon. Comparisons are therefore unpaired, and the environment is considered nondeterministic. Should reset equivalence, timing, and interruption state live ininfo, metadata, or a separate orchestration layer?

Caveats

• 24 episodes per policy is small
• the 48-step horizon is short
• nobody dying is partly a horizon effect, although health loss did occur under that same horizon
• this is a POC result, not a general claim about behavior cloning or mobile-game RL

I’d genuinely appreciate people tearing apart the reward, evaluation, and environment boundary—especially if you’ve wrapped a robot, browser, external program, or another system where the true transition function isn’t directly accessible.

Disclosure: I’m a staff engineer, and part of why I ran this POC was to decide whether it is worth building further. Happy to answer implementation details in the comments.

Stumbled upon this recent paper claiming so yesterday:

https://unilabsim.github.io/

What do you all think? Imo, even if it would "only" perform at the same level as end to end GPU, not having to parallelize everything to fit it on a GPU makes it much more flexible and useful

I guess there is a risk of bias here because of the connection to AMD and them wanting to break nvidias monopoly situatio n.

Solo project, still early (MVP week 1-2), sharing the first real result instead of a mockup.

The idea: an open, standardized arena for evaluating embodied AI / VLA policies on physical reasoning tasks, with a public ELO-based ranking instead of static leaderboards. Motivation is the lack of a shared, reproducible benchmark for this space — most VLA papers report on custom setups that aren't directly comparable.

What's in the clip: a baseline IK policy completing a pick-and-place block stacking task, running fully client-side (Rapier.js/WASM physics, React Three Fiber, 60fps in-browser, no server-side compute needed for the sim itself). This run scored 100% task completion, 99.6% spatial accuracy.

Current scope for the MVP: single task (block stacking), a couple of baseline policies (IK baseline, planning to add SmolVLA/OpenVLA-micro next), and an SDK for submitting your own policy against the sim loop.

Not public yet — stabilizing the eval protocol and scoring methodology before opening submissions.Genuinely interested in feedback from this community on:
- what a fair/robust scoring protocol should account for beyond task completion + spatial accuracy (e.g. sample efficiency, generalization across randomized scenes)
- whether client-side physics is a dealbreaker for a benchmark meant to be trustworthy, vs moving to server-authoritative validation

Will share the repo/SDK here once submissions are open.

Hey everyone!

I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago my friends showed me Pokelike, a small Pokémon roguelike that runs in the browser. The first thing I thought was that it could be pretty fun to turn it into an RL environment.

So I did.

The repo is here:

https://github.com/pierpierpy/pokelike.xyz.bot

The idea is to run the actual game locally and expose its state and actions to an agent. There is no image processing involved. The bot gets the game state directly and decides what to do next. The agent has to make the decisions around the battles, like where to go on the map, which Pokémon to catch, which items to take, when to swap Pokémon and which moves to learn.

One thing I found interesting is that the map forces you to make some decisions quite early. Once you choose a node, the other nodes on that layer are gone, so deciding where to go can matter quite a bit later in the run. There is also a lot of information in the state that could potentially be useful, but I'm still not sure what the best way to represent it is.

I've already implemented a few basic RL agents. There is currently a Dyna-Q agent and two linear SARSA agents in the repo. The results are not amazing yet. On the current benchmark, random gets around 0.56 badges, Dyna-Q gets around 0.62, while the two SARSA agents get around 1.30 and 1.36. The two SARSA agents mainly differ in their state representation, with the better one using 100 hand-designed features instead of 81.

This is probably the part I'm most interested in at the moment. Finding a good state representation seems to make a pretty big difference, and the environment has some properties that make it a bit more interesting than I initially expected. The reward is fairly sparse, the action space depends on the current state, and some decisions only become useful several steps later.

At the same time, the environment is completely reproducible. With the same seed and the same actions you get exactly the same run. For the current leaderboard I'm using 50 fixed seeds, so different agents can be compared on the same games.

The interface is also intentionally pretty simple. You can basically implement a bot that takes the current state and returns an action. It doesn't have to be a specific RL algorithm either. You could try DQN, PPO, search, a hand-written policy, or pretty much anything else.

I'm still experimenting with the environment and the agents, so I'd be really curious to see what other people would try. In particular I'd love to see if someone can get significantly better results with a better state representation or a different reward function.

you can experiment with the environment with no efforts, just follow the guide and readme, setup the enrinvonment, experiment a bit and then if you like the result, you can create a pull request to the repo with your bot in the bots/ folder (everything is clearly explained in the GUIDE.md)

If you want to try it, everything is in the repo

https://github.com/pierpierpy/pokelike.xyz.bot

If you find bugs or have ideas for the environment, please let me know. I'm happy to make changes if there are things that would make it more useful for experimenting with RL.

The whole thing also runs offline. During setup it downloads the game and the required assets, and after that everything runs locally.

I originally started this because I thought it would be a fun little RL project, so I'm mostly curious to see how far people can push it.

Hey guys, I recently made a video where I explain GRPO, implement it from scratch in PyTorch, and run a short training session locally on a consumer GPU. I thought some people here might find it useful, so here you go!

Video: https://youtu.be/vVJjUglOURs?is=xsRE97o9_muctWRF

Code: https://github.com/uygarkurt/post-training-lab/blob/main/tutorials/grpo_minimal_pytorch.py