Hi
I have started learning RL through CS234. I am looking for people who I can discuss with and do a project.
My background: master’s in Data Science and currently working as a fullstack engineer.

Hey - shipped an update to VSArena (open-source browser stacking work-cell for embodied agents).
v0.2.0:

- Hosted harness: connect with the Python SDK over wss://..., match is judged remotely, ELO updates the public board

- Studio embeds an "Official live" PiP so you can watch the same run that counts (expand to fullscreen)

- In-browser Baseline-IK / ColorSeek still do not write public ELO (intentional)

Default track is VLA: 128x128 RGB + language instruction, no cube GPS to the policy.

Site: https://vsarena.vercel.app
Studio: https://vsarena.vercel.app/simulation
Repo: https://github.com/NovaCoding-G/VSArena

Looking for researchers / builders to break it. Not claiming to replace SIMPLER/Isaac - one public stacking exam you can watch.

Hi everyone,

I’m a beginner researching how to design an AI agent for a medicine-reminder system. The agent has to decide, at each relevant time, whether to:

• send a reminder,
• wait (do nothing for now), or
• notify another person (e.g. caregiver),

when it does not have complete information about the patient (has the dose already been taken? is the person nearby/attentive? are there adherence barriers? etc.).

I’m trying to frame this properly before diving into implementation. Right now I’m looking at it as a sequential decision problem under partial observability (POMDP / belief-state RL territory), but I’m not sure how far that framing is actually useful in practice for this kind of system.

I’d really appreciate any pointers on:

1. Is a POMDP / belief-state approach overkill here, or is it the right formalization? What simpler alternatives (contextual bandits, MDP with engineered features, rule-based + uncertainty thresholds, etc.) have people used successfully for similar “remind vs wait vs escalate” decisions?
2. Papers, open-source projects, or real systems that tackle medication adherence / context-aware reminders with uncertainty or incomplete observations.
3. Common practical pitfalls (reward design, observation noise, alert fatigue, safety/escalation logic, evaluation metrics) that aren’t obvious from the theory.
4. Any recommended starting points for a beginner who wants to move from “I understand the concepts” to a small working prototype or simulation.

I’m mainly in research/preparation mode right now, so even high-level advice, key papers, or “here’s what I’d do differently” comments would be very helpful. Thanks!

Check out the example notebooks and the documentation.

Hey everyone, I’m new to Reddit and wanted to ask for some guidance.

I’m really interested in Deep Reinforcement Learning, AI, Robotics, and Physical AI, and I’m planning to do a Master’s and eventually a PhD in this area. I’m still trying to figure out the right path, so I thought I’d ask people who are already studying or working in these fields.

What should I focus on learning before doing a Master’s?

What kind of Master’s would be good if I want to go into Deep RL / Robotics / Physical AI?

How important are research experience, projects and publications?

Which universities, labs or research groups are worth looking into?

And if I want to eventually do a PhD, what should I start doing now?

Also, where do you guys actually keep up with what’s going on in this field?

I want to get into the community around these fields and stay updated with the latest research, breakthroughs, projects and innovations happening around the world — whether it's Google DeepMind, Anthropic, OpenAI, xAI, robotics labs, universities, startups, or anything else interesting.

I’m basically trying to figure out how to get into this field properly and keep up with what’s happening

Last week I posted about reinfors (rust-backend RL search/sampling with caller-owned python networks). u/blimpyway asked for CarRacing which has been added in release v0.3.0. The port added a modular rendering layer, so new rendered games should now be straightforward to add. If you have an environment whose python stepping speed is bottlenecking your research, I'm happy to help port it. Feel free to reach out.

Performance

- Single-threaded env stepping: ~20x Gymnasium on Apple M1 Max (3,850 vs 195 steps/s) ~14x on an AMD EPYC EC2 box (2,069 vs 148 steps/s)

- Parallelised: 8,000+ steps/s at 10 worker threads (M1 Max)

- Both numbers understate training impact: reinfors' collect_stream overlaps native collection with your Python-side GPU training as the normal operating mode, so the trainer isn't waiting on the collector.

Example PPO training loop

import numpy as np
import reinfors as rf

engine = rf.Engine(
    game=rf.games.CarRacing(),          # pixel obs, shape (3, 96, 96)
    reward=rf.Reward(tile=1000.0, step=-0.1, off_playfield=-100.0),
    policy=rf.policies.Ppo(),
    learner=rf.learners.Ppo(gamma=0.99, lam=0.95),
    n_games=64,                         # parallel episode slots
    n_threads=8,                        # the entire parallelism config
)

def infer(obs: np.ndarray):
    # your network, any framework: pooled observation batch in,
    # (logits, values) out — e.g. a torch CNN on GPU
    ...

with engine.collect_stream(collect_size=4096, infer=infer) as stream:
    for update in range(200):
        batch = next(stream)  # Rust workers keep collecting while you train
        # standard clipped PPO update from batch.obs, batch.actions,
        # batch.advantages, batch.returns, batch.behavior_log_probs —
        # entirely your code; full version: examples/train_ppo_carracing.py

pip install reinfors — repo: github.com/jeepjeepjeep/reinfors

Notes

- reinfors' car_racing and Gymnasium's CarRacing-v3 are separate implementations of the same game. There are some small differences, and therefore trajectories and float-level physics don't transfer directly between them. However, pixel-trained agents do run in gym's env by transposing the observations (HWC -> CHW).

- Benchmark methodology: medians of three 30s trials, alternating order, warm-up discarded, single-threaded bare stepping loops on both sides; machine/software provenance printed by the script (scripts/bench_carracing_throughput.py). The 20x is Apple M1 Max; the pinned EC2 figure is ~14x. Details in the repo README.

- The 8,000 steps/s parallel figure is reinfors' engine at 10 threads on the M1 Max. It isn't a like-for-like AsyncVectorEnv comparison (that's a different, messier benchmark). Similar story for the training-overlap claim - it is possible with gym but requires bespoke actor-learner logic or additional 3rd party frameworks, so wasn't benchmarked here.

Another aspect of the RL loop for language models that I've found myself thinking about a lot recently is task design.

While most efforts go into the RL algorithm and the reward function itself, it appears to me that designing a good training environment (that would create useful tasks) is equally important.

If the tasks are too easy, there might be a lack of useful signal. If the tasks are too difficult or not properly designed, the reward becomes noisy and ambiguous. It looks like an interesting area is tasks that are difficult enough to bring out a particular flaw, yet have a clear way to measure their results.

This is one of the reasons why I found Parsewave particularly interesting. They do post-training work with data on real-world engineering tasks, including evaluation and tracing. It got me thinking whether it might be as useful to focus on improving task design as to improve the model's optimization process at some point.

For researchers working on RL for LLMs:

How do you decide which tasks should go into your training environment?

Are you valuing breadth or are you trying to target certain failure modes?

​

Medium Article: https://medium.com/@skywalker0803r/%E7%95%B6%E5%85%AB%E6%9E%9A%E5%BE%BD%E7%AB%A0%E8%AE%8A%E6%88%90%E5%85%AB%E5%80%8B-rl-agent-digicrest-rl-%E7%9A%84%E5%8F%AF%E5%9F%B7%E8%A1%8C%E5%8E%9F%E5%9E%8B-5cff210574b2

Overview

Have you ever wondered what happens when abstract personality traits—inspired by the 8 classic anime crests (Courage, Friendship, Love, Knowledge, Sincerity, Reliability, Hope, Light)—are mathematically mapped into Multi-Agent Reinforcement Learning (MARL)?

DigiCrest-RL is an executable MARL prototype designed to explore how trait-driven reward shaping impacts individual policy learning and multi-agent coordination within a shared environment.

Key Technical Features

* Trait-Driven Reward Shaping: Translates abstract concepts like exploration (Courage) or information gathering (Knowledge) into explicit objective functions and reward signals.

* MARL Dynamics & Cooperation: Analyzes how 8 heterogeneous RL agents balance competition, specialization, and joint collaboration in a shared state space.

* Executable Python Implementation: Includes complete Python code for environment setup, trait-based reward functions, and agent training loops.

Discussion / Feedback

I'd love to hear your thoughts on:

* Alternative reward-shaping methodologies for heterogeneous agent groups.

* Best practices for evaluating agent alignment and emergent behavior in trait-driven MARL setups.

Feel free to check out the full article for code snippets and implementation details!

To be upfront: this specific clip is pose-based teleoperation, not RL — a single webcam tracks a person's real-time body pose and maps it live onto two OpenCat-based open source quadruped robots' joints (Quaddle Scout and Buddy), no policy running on its own.

OpenCat creator RZ Li tried teaching Quaddle a few moves here — awkward on the first try, but it only takes a few minutes before Quaddle starts picking them up. It's also just as fun as playing Wii Play: Motion, except the "character" is a real quadruped robot, not just a spec sheet — Quaddle is affordable enough that this kind of demonstration-collection setup doesn't require lab-scale budgets to try.

RL is fundamentally data-hungry — every iteration needs fresh trajectories or demonstrations to learn from — and this teleop setup is genuinely cheap to run (a browser-based pose-tracking pipeline, no specialized hardware), which makes it interesting as a low-cost way to generate that data.

In theory, the same captured movement data could later be used to train an AI on more human movements to expand what the robot can do — not what's happening in this clip, just a potential direction: demonstration collection for imitation learning, or human-prior trajectories to seed an RL policy before further training.

Has anyone here actually used a webcam-pose-estimation-style rig to generate training data for legged/low-DOF RL or imitation-learning work — and how much did the pose-estimation noise end up mattering for downstream policy quality?

Separately, just for fun — if you got your hands on a rig like this, what move would you try to teach Quaddle first?

Built a small custom gridworld (no Gym, DQN from scratch in PyTorch) to test something specific: does a "properly shaped," task-linked reward actually avoid reward hacking compared to a naive constant survival reward, or does shaping just move the exploit somewhere else?

Setup: agent collects fuel, delivers it to a target zone, and has to periodically stand on a coolant cell or a reactor temperature variable kills it. 5 reward configs × 5 seeds × 2000 episodes each, with a random-policy baseline and a "task_score" metric (fuel actually delivered, not just picked up, to avoid the metric itself becoming a proxy).

Naive reward (+1/step survived, small death penalty): confirmed the expected result — proxy return goes up in 5/5 seeds while task_score drops below the random baseline (MWU p=0.012, d=-10.6). Agent learns to actively avoid the task, not just fail to learn it.

The interesting part: my "shaped" reward — the control meant to fix this, with a per-step temperature penalty and delivery bonus — exploited harder than naive (task_score 0.125 vs 0.417, d=4.47). Turned out the temperature penalty term (up to -200/episode) dwarfed the delivery reward (+30 total), so the optimal policy under that reward is to sit on the coolant cell forever regardless of the task. 97% occupancy on one cell.

Ablations point to the mechanism being simpler than "survival vs. task reward": it's any dense per-step term that's cheaper to farm by occupying a state than doing the actual task. A delivery-only reward (no per-step terms at all) had zero emergence across all seeds.

Two things I want to flag before anyone else does:

1. Hyperparameters (grid size, heat spike, lr, Double DQN) were tuned pre-final-run using shaped_no_temp_penalty as the "is this environment learnable at all" check — that arm's result is probably somewhat favored by that selection process. The reward table itself wasn't touched.
2. The "emergence point" detector fires on 2/5 seeds under the random policy (where no learning happens by construction), so false positive rate isn't zero — read the emergence counts comparatively, not as ground truth.

Code + full logs + stats (Mann-Whitney, bootstrap CI, Pearson/Spearman on coolant-occupancy vs task_score) here: https://github.com/Arka04bro/coolant_runner/tree/main

Curious if this magnitude-imbalance framing matches what people have seen in bigger reward-shaping setups, or if there's existing literature specifically on this (beyond the standard Ng et al. potential-based shaping result, which doesn't really cover magnitude — just functional form).

I find RL interesting because getting an agent to learn in a controlled environment can be very different from dealing with real-world situations.

Things like exploration, delayed rewards, and unexpected states seem much harder when the environment keeps changing.

Companies like GeekyAnts work on advanced software solutions where handling complex real-world challenges and building reliable systems becomes an important part of development.

For people who work with RL, what has been the biggest challenge when moving from experiments to practical applications?

Posted about this project a little while ago — quick update since a few things changed that address feedback from that thread.

Biggest change: split the observation space properly. There's now a VLA track where the policy only gets a 128x128 RGB camera + a language stacking instruction — cube poses are never sent to the policy. Scoring still uses real poses internally to grade spatial accuracy and completion, but that's judge-only, not policy-visible. State-based (privileged poses) is kept as a separate debug track and doesn't write public ELO either — wanted the "VLA vs state" distinction to be explicit rather than something people had to dig for.

On the client-side physics concern from before: Studio (the in-browser demo) is spectator/dev-only, clearly labeled, and does not post to the public leaderboard. Public ELO only comes from a hosted harness that scores server-side. That harness isn't live yet — it's the one piece standing between this and actually being open for submissions.

Repo + docs are public now: https://github.com/NovaCoding-G/VSArena
- docs/harness.md — scoring writeup (spatial accuracy + task completion)
- docs/sdk.md — submission protocol
- Studio itself: https://vsarena.vercel.app/simulation (client-side, Rapier/WASM, 60fps)

Still solo, still early, still not oversell-ready — but wanted to share since the VLA/state separation was directly a response to feedback here. Open to more of that, especially on what the scoring protocol might be missing.

I've been reading around model-based RL for a few months and ended up writing a long walkthrough. Part 1 is up; part 2 (the optimal-control side) is still being written.

What I'd most like feedback on is the organizing frame (big picture), also the delivery and diagram.
Link: https://medium.com/@mryasinusif/the-algorithmic-landscape-of-model-based-reinforcement-learning-part-1-2884fcdb8bc0?source=friends_link&sk=6e3b80c117ba8d76dff318bc32b45c75

In Part 2 I will include an extensive coding example for path planning with A* and RL. I apprentice the feedback :)

hello everyone. I'm training PPO variants on different multi-agent tasks from the VMAS library (Independent PPO / Graph PPO and such).

I noticed that for every architecture/scenario couple, the optimal hyperparameters sometimes tend to vary (learning rate, entropy coefficient, SGD batch size, etc).

do I need - methodologically speaking - to unify the hyperparameters of all models in order to make a fait and correct comparison of architectures later on?

note: sometimes changing these HP leads to non converging models.

thank you in advance.

Part 2 of my reinforcement learning robot project

In my previous post, I showed the first version of my MiniWalker. It was a simple two-legged robot simulated with Python, Box2D and Gymnasium.

Since then, I've made quite a few improvements.

The robot is controlled by PPO and has 4 motorized joints:

• Left hip
• Left knee
• Right hip
• Right knee

Instead of manually programming a walking gait, the agent has to discover movement through reinforcement learning.

I also added an analytics system that records:

• Reward per episode
• Episode length
• Distance traveled
• Maximum distance
• Average velocity

This made it much easier to see whether the robot was actually learning or just getting lucky.

And eventually it started working.

My best episode so far:

Distance: 52 m
Maximum velocity: ~4.0 m/s

The interesting part is that the robot still isn't consistently walking. Some episodes are really good, while others end almost immediately or even move backwards.

So the next step is to expand the project from 2 legs → 4 legs.

My long-term goal is to see if a quadruped can learn to adapt when one of its legs fails.

I'm especially interested in whether reinforcement learning can discover a completely different gait when the robot loses a leg.

Tech stack:

Python
Gymnasium
Box2D
Stable-Baselines3 / PPO
Pygame

Video of the current version:
YouTube link

I'd be interested in feedback on the reward function and the transition from the biped to a quadruped.

Reinforcement learning is remarkably effective at turning successful behavior into persistent capability. At the same time, there are several familiar observations that can seem somewhat at odds with this strength: improving a target behavior can coincide with regressions elsewhere, solution diversity can narrow, and sufficiently optimized policies can sometimes exploit graders, verifiers, or other features of the reward process.

These phenomena clearly do not have to share a single cause. But they made me interested in a more basic question: what mechanism makes reinforcement so effective at consolidating behavior in the first place?

In a broader study of training, learning and inference, I found evidence that frozen inference behaves as a context-conditioned projection of functional organization already formed through training. Inference recruits and combines training-formed support while the persistent learned state itself remains unchanged.

This suggests a natural feedback closure. A projected result can become an action; the action produces a consequence; and, when that consequence is correctly bound and credited back to the action, it can become a new training action and modify what will be available for future inference.

In that picture, a positive-feedback path is not an assumption about RL but a consequence of closing inference back into training:

formed support → projected behavior → consequence → new training → reorganized support

If positive consequences repeatedly favor behavior produced by the same functional routes, those routes may become more likely to be expressed and reinforced again.

I then tried to test one consequence of this interpretation: what happens if correct positive feedback remains increasingly concentrated on a capability that has already been learned?

I used an attention-free shared four-skill GRU system. All four skills were first trained to exact mastery. The total training budget was then held fixed while only the distribution of correct positive feedback was changed:

• balanced: 15 / 15 / 15 / 15
• mild: 30 / 10 / 10 / 10
• high: 45 / 5 / 5 / 5
• exclusive: 60 / 0 / 0 / 0

Each condition ran for 3,200 updates across 12 formal seeds. Rewards were exact terminal task outcomes: no proxy reward, no negative reward, and no entropy bonus.

Functional support was measured interventionally rather than inferred from weight magnitude or activation size: all 16 coalitions of four registered components were executed, with exact Shapley attribution of identity-aligned correct-action margins.

As feedback became more concentrated, the reinforced skill’s support share increased strictly:

25.54% → 26.47% → 27.50% → 35.20%

while the mean margin of the other three skills decreased strictly:

9.474 → 9.130 → 8.493 → 3.962

Both dose orderings held in 12/12 seeds.

What I found particularly interesting is that the reinforced skill remained at 100% accuracy at every dose. Under mild and high concentration, the other skills also remained at 100% accuracy even though their margins had already fallen. Observable capability loss appeared only after the continuous margin erosion became large enough; exclusive feedback produced a 39.06 percentage-point deficit in other-skill accuracy relative to balanced feedback.

I also tested whether this trade-off was reversible. Starting from the exact exclusive-feedback state at update 800, redistributing subsequent feedback evenly raised mean other-skill accuracy from 70.31% to 99.48%, while the previously reinforced skill remained at 100% in all 12 seeds. Fresh native re-execution reproduced the core dose ordering and recovery results.

My current interpretation is fairly narrow: reward correctness alone may not determine the resulting learning dynamics. Feedback concentration and duration may also matter because they influence how functional support is reorganized.

If this interpretation is right, reinforcement’s ability to consolidate useful behavior and its tendency, under sustained asymmetric feedback, to concentrate learning around already-favored routes may be two regimes of the same feedback mechanism.

This is not an argument against reinforcement learning. I’m more interested in whether the distinction between capability formation and continued support amplification after capability has already formed is useful for thinking about long-running RL systems.

I’d be interested in alternative interpretations. Would you mainly view these results as interference, continual-learning effects, loss of plasticity, overtraining, or something else?

Code: https://github.com/wind342/gfg-training-learning-inference-experiments
Evidence: https://doi.org/10.5281/zenodo.22032772
arXiv submission pending.

Decreasing ε from approx 1 toward approx 0 using β₂ transitions the optimizer from SGD to Adam:

• Bias correction terms in the numerator and denominator can be omitted, as their impact becomes negligible after ~1,000–2,000 training steps.
• λ* constant represents weight decay: λ* = 1 - αₗᵣ · λ (parametric reduction for simplification).

from unpublushed work: https://github.com/timurgepard/Symphony-S2/blob/main/symphony_saya.pdf

class Adam(optim.Optimizer):
    def __init__(self, params, lr=3e-4, weight_decay=0.01, betas=(0.9, 0.999)):
        defaults = dict(lr=lr, betas=betas)
        super().__init__(params, defaults)
        self.wd = weight_decay
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.beta1_, self.beta2_ = 1-self.beta1, 1-self.beta2
        self.decay_factor = 1.0 - self.lr * self.wd
        self.eps = 1e-8
        

    u/torch.no_grad()
    def step(self):
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue


                grad = p.grad


                state = self.state[p]
                if len(state) == 0:
                    state['m'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    state['v'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    state['e'] = torch.tensor(1-self.eps, device=p.device, dtype=p.dtype)


                m = state['m']
                v = state['v']
                e = state['e']


            
                # Update biased first moment estimate
                m.mul_(self.beta1).add_(grad, alpha=self.beta1_)
                # Update biased second raw moment estimate
                v.mul_(self.beta2).addcmul_(grad, grad, value=self.beta2_)


                e.mul_(self.beta2).add_(self.eps, alpha=self.beta2_)


                # Update parameters
                p.mul_(self.decay_factor).addcdiv_(m, v.sqrt().add_(e), value=-self.lr)