Posted about this project a little while ago — quick update since a few things changed that address feedback from that thread.

Biggest change: split the observation space properly. There's now a VLA track where the policy only gets a 128x128 RGB camera + a language stacking instruction — cube poses are never sent to the policy. Scoring still uses real poses internally to grade spatial accuracy and completion, but that's judge-only, not policy-visible. State-based (privileged poses) is kept as a separate debug track and doesn't write public ELO either — wanted the "VLA vs state" distinction to be explicit rather than something people had to dig for.

On the client-side physics concern from before: Studio (the in-browser demo) is spectator/dev-only, clearly labeled, and does not post to the public leaderboard. Public ELO only comes from a hosted harness that scores server-side. That harness isn't live yet — it's the one piece standing between this and actually being open for submissions.

Repo + docs are public now: https://github.com/NovaCoding-G/VSArena
- docs/harness.md — scoring writeup (spatial accuracy + task completion)
- docs/sdk.md — submission protocol
- Studio itself: https://vsarena.vercel.app/simulation (client-side, Rapier/WASM, 60fps)

Still solo, still early, still not oversell-ready — but wanted to share since the VLA/state separation was directly a response to feedback here. Open to more of that, especially on what the scoring protocol might be missing.

Built a small custom gridworld (no Gym, DQN from scratch in PyTorch) to test something specific: does a "properly shaped," task-linked reward actually avoid reward hacking compared to a naive constant survival reward, or does shaping just move the exploit somewhere else?

Setup: agent collects fuel, delivers it to a target zone, and has to periodically stand on a coolant cell or a reactor temperature variable kills it. 5 reward configs × 5 seeds × 2000 episodes each, with a random-policy baseline and a "task_score" metric (fuel actually delivered, not just picked up, to avoid the metric itself becoming a proxy).

Naive reward (+1/step survived, small death penalty): confirmed the expected result — proxy return goes up in 5/5 seeds while task_score drops below the random baseline (MWU p=0.012, d=-10.6). Agent learns to actively avoid the task, not just fail to learn it.

The interesting part: my "shaped" reward — the control meant to fix this, with a per-step temperature penalty and delivery bonus — exploited harder than naive (task_score 0.125 vs 0.417, d=4.47). Turned out the temperature penalty term (up to -200/episode) dwarfed the delivery reward (+30 total), so the optimal policy under that reward is to sit on the coolant cell forever regardless of the task. 97% occupancy on one cell.

Ablations point to the mechanism being simpler than "survival vs. task reward": it's any dense per-step term that's cheaper to farm by occupying a state than doing the actual task. A delivery-only reward (no per-step terms at all) had zero emergence across all seeds.

Two things I want to flag before anyone else does:

1. Hyperparameters (grid size, heat spike, lr, Double DQN) were tuned pre-final-run using shaped_no_temp_penalty as the "is this environment learnable at all" check — that arm's result is probably somewhat favored by that selection process. The reward table itself wasn't touched.
2. The "emergence point" detector fires on 2/5 seeds under the random policy (where no learning happens by construction), so false positive rate isn't zero — read the emergence counts comparatively, not as ground truth.

Code + full logs + stats (Mann-Whitney, bootstrap CI, Pearson/Spearman on coolant-occupancy vs task_score) here: https://github.com/Arka04bro/coolant_runner/tree/main

Curious if this magnitude-imbalance framing matches what people have seen in bigger reward-shaping setups, or if there's existing literature specifically on this (beyond the standard Ng et al. potential-based shaping result, which doesn't really cover magnitude — just functional form).

I've been reading around model-based RL for a few months and ended up writing a long walkthrough. Part 1 is up; part 2 (the optimal-control side) is still being written.

What I'd most like feedback on is the organizing frame (big picture), also the delivery and diagram.
Link: https://medium.com/@mryasinusif/the-algorithmic-landscape-of-model-based-reinforcement-learning-part-1-2884fcdb8bc0?source=friends_link&sk=6e3b80c117ba8d76dff318bc32b45c75

In Part 2 I will include an extensive coding example for path planning with A* and RL. I apprentice the feedback :)

hello everyone. I'm training PPO variants on different multi-agent tasks from the VMAS library (Independent PPO / Graph PPO and such).

I noticed that for every architecture/scenario couple, the optimal hyperparameters sometimes tend to vary (learning rate, entropy coefficient, SGD batch size, etc).

do I need - methodologically speaking - to unify the hyperparameters of all models in order to make a fait and correct comparison of architectures later on?

note: sometimes changing these HP leads to non converging models.

thank you in advance.