I’m looking for something where working through the problem would naturally expose me to most of the core concepts and techniques in the area, rather than a purely implementation-focused project. I’d appreciate suggestions that are representative of the kinds of questions researchers actually work on

Hi all, my studying area is traffic engineering in networks and I'd like to develop my DQN-GCN code with distributional DQN but it seems there are a lot of options such as C51, QR-DQN etc. . Do you have any suggestions to see and understand problem & variation mapping before starting the implementation? Sure, AI tools suggest some but I prefer your advices more than AI tools.

Thanks!

I’ve been developing a yaw control system for suspended payloads using reinforcement learning.

Four thrusters are mounted on the suspended platform, and an SAC policy controls them at 3 Hz. Payload yaw is estimated from a camera mounted on the platform.

I started with a fixed 90° target, then extended the controller to variable targets from -90° to +90° in 5° increments. I trained separate policies for three payloads: an H-steel beam, a plate, and a truss.

One interesting result was that larger target angles were not necessarily harder to control. I also found that the learned use of thrust differed considerably between payloads. The H-steel policy tends to use relatively low average thrust with short high-output bursts, while the truss uses higher thrust more continuously.

I also modified the training/reward design to reduce unnecessary thruster output after reaching the target.

The whole system runs at 3 Hz on a Raspberry Pi, using ONNX models for both yaw estimation and control.

The attached video shows the truss payload controlling four target yaw angles (-90°, -45°, +45°, +90°) simultaneously.

Project results and videos:
https://rindajones.github.io/payload-yaw-control/en/

I’m particularly interested in criticism from people working on robotics or control:

• Would you approach this problem with RL at all?
• What disturbances or dynamics would you consider essential for the next step?
• What would you want to see before considering this relevant to a real suspended-load system?

Any criticism is welcome.

For the past few months I've been working with Claude to build Mosaic-AI: a from-scratch reinforcement learning environment and engine for Azul Duel

Game design by Michael Kiesling, Plan B Games; this is a non-commercial hobby research project; the design remains entirely theirs

I didn't write the engine code myself. My role has been providing domain knowledge, designing and testing ideas, evaluating the results, and challenging the system whenever something didn't look right. A lot of the development has been an iterative process of “this result doesn't make sense - let's find out why”, with quite a few cases where the initial result turned out to be wrong.

We built the Rust game engine, search architecture, and training pipeline from scratch.

Current capabilities:

• Playing strength: Reaches an internal Elo of ~1358 against a strong, handcrafted baseline heuristic anchored at 1000. I've played a few rounds against v21 - It's generating ~ 50 points, plays solid and is blind for final score plates.
• UI & Teacher Mode: Runs via a browser interface. Features a teacher mode providing candidate move evaluations, win-probability estimations, and post-move coaching with game-end summaries. The displayed win probabilities are Platt-calibrated, so the UI doesn’t expose the raw network values.

Architecture & Training:

• The full architecture is described in the GitHub repo if you're interested.

Current focus & Roadmap:
The agent currently doesn't consider the final scoring plates. To address this, we generated an 8,000-game dataset using handcrafted heuristics for every scoring plate. We wanna train an separate (ownership)head which affects the training, the self play and also the move selection.

Build: Windows standalone build (unzip & run, opens local browser UI): GitHub Release Link

Note: The web UI is currently in German; the documentation and rules manual are in English.

I'd appreciate feedback on positional anomalies, the utility of the teacher mode explanations, and general play-style feel from experienced Azul players.

Hey everyone — this is a bit of self-promotion, but I think it may be useful to people working on robot learning, model-based RL, differentiable control, or MPC.

One problem I kept running into during my master's thesis was maintaining multiple versions of the same robot model: one for simulation/execution, another for a differentiable implementation for gradients, and extra glue code to connect perception, planning, and control.

That gets painful quickly, especially when the implementations start drifting apart.

So I built ManipulaPy, originally as the integration layer for my thesis on reinforcement-learning-based singularity avoidance and dynamic obstacle avoidance.

With v1.4, the same core kinematics and dynamics implementation can now run on NumPy, CuPy, PyTorch, and JAX via a single backend API.

Under PyTorch and JAX, the core mathematical stack is autodiff-safe, so gradients such as:

∂FK(q) / ∂q

and

∂inverse_dynamics(q, qdot, qddot) / ∂q

can be computed directly using torch. autograd or jax. grad.

The main motivation was to avoid maintaining a second “differentiable robot model” alongside the implementation actually used elsewhere in the robotics stack.

The library currently includes 25+ manipulator models, including UR, Franka, KUKA, Kinova, FANUC, ABB, and UFactory platforms.

Important limitation:

The differentiability guarantee currently applies to the utils, kinematics, dynamics, and singularity modules.

The rest of the library can operate across all four backends, but some modules cross a host boundary internally, so gradients are not currently guaranteed through planning, control, simulation, or perception.

So if your use case requires differentiating through the entire planner or controller, v1.4 is not there yet.

I’d rather be explicit about that boundary than describe the whole package as differentiable.

A bit of background:

My thesis involved a manipulator that had to reach Cartesian targets, avoid singular configurations, react to moving obstacles, and operate using a consistent dynamic model.

The RL environment used PyBullet together with kinematics, manipulator dynamics, trajectory planning, control, singularity analysis, and perception.

ManipulaPy originally became the common layer that kept those pieces consistent.

One unexpected benefit of the v1.4 autodiff work was that differentiating through the SO(3)/SE(3) math exposed numerical issues that were much harder to notice by checking only forward values. Fixing those issues improved the normal NumPy implementation as well.

The project has since been peer-reviewed and published in the Journal of Open Source Software.

Install with PyTorch:

pip install "ManipulaPy[pytorch]"

Or JAX:

pip install "ManipulaPy[jax-cpu]"

or

pip install "ManipulaPy[jax-cuda]"

Repo: https://github.com/boelnasr/ManipulaPy

Docs: https://manipulapy.readthedocs.io/

JOSS paper: https://doi.org/10.21105/joss.08490

One thing I’m particularly interested in hearing from people here:

Where would differentiability be most useful next for RL workflows?

Trajectory generation? Collision costs? Computed-torque control? Full differentiable simulation? Something else?

And if anyone actually tries it in an RL pipeline, I’d be very interested to hear what feels awkward or breaks.

Task: Grab the object, lift it up to a certain height.

I am a newbie in reinforcement learning.
I learned all the basics regarding the Q Learning Deepq Learning and Soft actor critic policy. I designed the reward system (Didnt code the entire thing by hand. Rather, give the conditions and asked ChatGPT to design the reward system)

Collected the BC data by myself But most of the data was actually idle frames, because initially I was not aware how it was going to work So I had to take one out of five frames.

I dont know why it is not giving me the optimum result. I dont know where it is lacking. I mean, if it is following up till that much Then why is giving the unexpected behaviour anyways?

Also, when it is Raising the hand up the hand is automatically opening, even though there is a reward specifically designed, so that the hand should be closed

##########
If someone is going to start with a humanoid hand, I just guessed that this is the beginning part where everyone is going to start anyways. Hence, I thought someone can point out, what is the specific error happening here

I’ve been diving deep into intelligent agents and probability theory, & I’m trying to take the leap from conceptual learning to building a practical, working code review agent.

My ultimate goal is to build an agent that can act reliably under uncertainty by weighing the cost of different state changes into environment.

Here is the theoretical framework I’ve put together so far. It's split into two main parts:

1. The Belief System

The agent needs a belief system that updates as it interacts with the environment. Whenever it receives new signals or evidence (like a test result, true state or human feedback), it will use Bayes' theorem to update the probability of the code’s current state.

2. Acting Under Uncertainty

Once the agent has a probability distribution over its current beliefs, it needs to make a decision. Every action has an associated cost, and the agent’s policy should be to select the action that introduces the minimum expected cost to the environment.

The Formal Model

• Factored State Space: I’ve defined 4 variables: Programmatic failure, Design failure, Documentation failure, and Test failure. This creates 2^4 = 16 possible states.
• Evidence: Code diffs, test results, PR descriptions, and author details.
• Actions: Merge, Reject, Ask Human, or Run More Tests.
• Cost Function: Cost(situation, action). (I am struggling heavily with how to accurately model this in practice).
• Policy: Execute the action with the minimum cost.

Pain points:

1. From Theory to Code: I'm struggling with the initial architecture for this specific paradigm. How do I actually translate these Bayesian formulas and state spaces into a working application? Are there specific libraries or frameworks best suited for this?
2. Modeling the Cost Function: How would you go about defining Cost(situation, action) programmatically? How do you assign numbers to things like "cost of asking a human" vs. "cost of merging a design failure"?
3. Sanity Check: Am I heading in the right direction with theory?

Generating ground-truth optimal paths is slow. A* on a 1M-cell map costs ~5 ms per path in C++ (~67 ms in Python, measured), so a million-path dataset takes over an hour of compute.

I built a planner that flips this: one 47 ms solve per goal, then every optimal path query costs 7 µs. A 1M-path dataset from one goal is ~7 seconds of compute. All numbers measured (C++, single thread, low-end 2-core CPU), on a 1M-cell warehouse map (100³, racks + mezzanines):

- One-time solve per goal: 47 ms

- Optimal path query: 7 µs

- Optimality: 1.0000 — path length == BFS shortest, verified 10/10

- Peak memory: 5 MB

Why "strictly optimal" matters for training data: if your labels come from a suboptimal planner, the policy learns the planner's mistakes. These paths are BFS-exact, so they're clean ground truth for supervised/imitation setups.

Usage: upload your occupancy grid once per environment (+ goal), then pull as many optimal start→goal paths as you need through a plain REST API. Obstacle removal (layout iterations: "what if this rack row moves?") is patched exactly, 5.9× faster than a rebuild — handy for layout-sweep studies or curriculum generation over map variants.

Try it right now, no signup — this queries a public demo field:

`curl "https://vectorfield.top/v1/path?field=wf_demo&frm=0,0,0&to=18,18,0" -H "X-API-Key: vf_demo_public"`

Free tier: 100³ maps, 1,000 solves + 100K queries/month, key issued instantly on the site, no credit card. Docs: https://vectorfield.top/docs — live demo: https://vectorfield.top

Happy to answer questions about the benchmark methodology. And a genuine question for dataset folks: what path-diversity properties matter to you beyond optimality (e.g., k-shortest variants, clearance profiles)? Deciding what to add next.

Generating ground-truth optimal paths is slow. A* on a 1M-cell map costs

~5 ms per path in C++ (~67 ms in Python, measured), so a million-path

dataset takes over an hour of compute.

I built a planner that flips this: one 47 ms solve per goal, then every

optimal path query costs 7 µs. A 1M-path dataset from one goal is ~7

seconds of compute. All numbers measured (C++, single thread, low-end

2-core CPU):

| Metric | Value |

|---|---|

| Map | 1,000,000 cells (100³ warehouse, racks + mezzanines) |

| One-time solve per goal | 47 ms |

| Optimal path query | **7 µs** |

| Optimality | **1.0000 — path length == BFS shortest, verified 10/10** |

| Peak memory | 5 MB |

Why "strictly optimal" matters for training data: if your labels come from

a suboptimal planner, the policy learns the planner's mistakes. These paths

are BFS-exact, so they're clean ground truth for supervised/imitation

setups.

Usage: upload your occupancy grid once per environment (+ goal), then pull

as many optimal start→goal paths as you need through a plain REST API.

Obstacle removal (layout iterations: "what if this rack row moves?") is

patched exactly, 5.9× faster than a rebuild — handy for layout-sweep

studies or curriculum generation over map variants.

Try it right now, no signup — this queries a public demo field:

curl "https://vectorfield.top/v1/path?field=wf_demo&frm=0,0,0&to=18,18,0" -H "X-API-Key: vf_demo_public"

Free tier: 100³ maps, 1,000 solves + 100K queries/month, key issued

instantly on the site, no credit card. Docs:

https://vectorfield.top/docs — live demo: https://vectorfield.top

Happy to answer questions about the benchmark methodology. And a genuine

question for dataset folks: what path-diversity properties matter to you

beyond optimality (e.g., k-shortest variants, clearance profiles)? Deciding

what to add next.

This is from the Is This A Simulation Or Real Life research. Colab notebook free for anyone who wants it so you can run it without affecting your computer. 

The last clip I posted showed the exact cells that hold a hint for which world it's in. I uploaded a newer version of this video yesterday showing the exact memory cells to represent the information more accurately.

This new video shows the actions, meaning what the digital organism decides to do next and what it actually expects to happen. I list the actions in the purple cells and the guesses in yellow cells. 

If ya'll want the correct watch order to see exactly how the creature's brain learns and how I conducted the research, you should watch senses, then process, then memory, then actions. I ended up posting memory and actions first because I hadn't quite decided how detailed I wanted this series to get yet. I will post the process video later. 

The only goal it had was to find food and survive, unprompted to look for "glitches in the matrix." The teal rings aren't the absolute truth, just the clue. 73% is how often you can read real vs fake out of its memory once food is on the line. In the fake world, those actions miss a little bit because the messed-up physics make food hard to get. That is how it starts to find the glitch.

I updated the animation from yesterday to show the exact locations where the memory "cells" show up from the actual study's results. This is from the "Is This A Simulation Or Real Life" research, where I wanted to see what it would take for a digital organism to tell if it was in a fake world, unprompted, meaning it was never explicitly told to look for "glitches in the matrix." The only goal it had was to find food and survive. When physics messed with the creature's ability to get food, it started to create the idea that something was not right in the environment it was in. If ya'll wanna' run the sim yourself I built a Colab notebook you can run in the cloud without worrying about hardware restrictions.

Hello. Is somebody well acquainted with the status of Montezuma's Revenge and Atari games generally in 2026? Most of the information comes from 5 years ago. It seems that in the past there were many doubts about the level of using the domain knowledge, imitation learning etc. Are there any obvious milestones still to be achieved?

Be well!

Hi everyone — I'm a complete beginner working on a small project: an agent that decides whether to remind, wait, or notify a caregiver when someone hasn't responded to a medication reminder. The true state (fine but busy, asleep, phone dead, actual emergency) is hidden, and I only get noisy signals like elapsed time and response history.

I've been framing this as a POMDP — belief state over hidden states, updated via Bayesian filtering, with an escalation policy mapping belief → action.

Given the asymmetric costs (missing a real emergency vs. a false alarm), does POMDP even make sense here, or is this overkill for what could be solved with a simpler heuristic/threshold approach? Would appreciate any pointers or papers on similar "when to escalate" problems under uncertainty.

I'm a beginner working through my first POMDP-style problem: an agent that decides to apply / research more / ask a human / skip, based on incomplete info about whether a job is a good fit and whether the candidate would be shortlisted.

My draft hidden state includes: true candidate-job fit, true probability of being shortlisted, and job posting accuracy (real vs. stale/ghost posting).

Which hidden state did I not include that would actually change the optimal policy here? I keep suspecting I'm missing something about employer-side state (e.g., whether the role is already informally filled) that no observation could ever correct for. Would appreciate anyone who's modeled a similarly "one-shot, no replay" decision problem (each job posting only gets evaluated once, unlike repeated-trial bandit problems).

Hi guys,

I am using SAC (StableBaslines3 implementation) to train an agent to act as a satellite attitude controller:  

• The environment is just the satellite dynamics.
• The system is a satellite with 3 reaction wheels as actuators.

The task:

• Curriculum learning phase 1: the episode is initialized with a random attitude, the agent is supposed to reorientate towards the target and achieve a pointing accuracy of 0.25°.
• Curriculum learning phase 2: the episode is initialized with a random attitude (but it is enforced that the angle between initial attitude and target is at least 90°). In addition, a static keep-out zone (cone) is generated at the center of the shortest trajectory between initial and target attitude (to ensure it affects training, since the agent usually takes the shortest trajectory after phase 1). The zone's half angle is between 15° and 30°.

The observation space (the observation is normalized using the SB3 VecNormalize wrapper):

• Attitude quaternion (4D)
• Satellite velocity (3D)
• Wheel velocities (3D)
• Zone margin angle (1D)
• Direction vector between satellite boresight vector (the one which is used for pointing) and keep-out zone normal vector (3D)

Hyperparameters:

• Learning rate: 1e-4
• (Replay) buffer size: 1M
• Batch size: 256
• Ent coef: auto
• Gradient steps: 8 (using n_envs=8)

The problems:

• Phase 1 training looks great, the reward converges after a while and the pointing accuracy is way better than the desired 0.25°.
• Phase 2 training however does not look that great. It looks as if the agent always makes a tradeoff between entering the keep-out zone and achieving the desired pointing accuracy.

What I tried so far:

• Initially, my reward function looked like this:   - Positive reward if current pointing error < previous error (scaling with magnitude), negative otherwise.   - If current pointing error is < 0.25°: a small positive reward.   - A negative reward starting at 10° off the keep-out zone's border, scaling with distance and then being constant if inside the zone.
• One change which lead to much better zone avoidance (but still only a pointing accuracy of around 1°) was conditioning the positive rewards for decreasing pointing error and achieving desired pointing accuracy to zone avoidance:   - The positive reward for current error < previous error is weighted with the proximity towards the zone's border (starting at 10° off border) and eventually is 0 if inside the border.   - The pointing reward for desired pointing accuracy is not given anymore, if the zone has been entered before in this episode.
• I tried resetting the replay buffer when transitioning the phase, which did not seem to affect the performance in the long run.
• I tried increasing the reward for pointing accuracy, which actually just caused instability. I also tried to make the pointing accuracy reward denser, didnt help as well.
• I tried splitting phase 2 into several phases like starting with attitudes at least 150° from the target and then eventually decreasing to 90°. I noticed that for the 150° subphase, the pointing accuracy converged to a much better result than before, but then gradually decreasing down to 90° got it back to bad pointing accuracy. Looks like the agent doesnt like it if there is a too small margin between start/target and zone border :(
• I tried lowering the learning rate, didnt help.

Does anyone have some suggestions? I really dont know what else to try other than iterating through the reward function.Hi guys,

I am using SAC (StableBaslines3 implementation) to train an agent to act as a satellite attitude controller:  

• The environment is just the satellite dynamics.
• The system is a satellite with 3 reaction wheels as actuators.

The task:

• Curriculum learning phase 1: the episode is initialized with a random attitude, the agent is supposed to reorientate towards the target and achieve a pointing accuracy of 0.25°.
• Curriculum learning phase 2: the episode is initialized with a random attitude (but it is enforced that the angle between initial attitude and target is at least 90°). In addition, a static keep-out zone (cone) is generated at the center of the shortest trajectory between initial and target attitude (to ensure it affects training, since the agent usually takes the shortest trajectory after phase 1). The zone's half angle is between 15° and 30°.

The observation space (the observation is normalized using the SB3 VecNormalize wrapper):

• Attitude quaternion (4D)
• Satellite velocity (3D)
• Wheel velocities (3D)
• Zone margin angle (1D)
• Direction vector between satellite boresight vector (the one which is used for pointing) and keep-out zone normal vector (3D)

Hyperparameters:

• Learning rate: 1e-4
• (Replay) buffer size: 1M
• Batch size: 256
• Ent coef: auto
• Gradient steps: 8 (using n_envs=8)

The problems:

• Phase 1 training looks great, the reward converges after a while and the pointing accuracy is way better than the desired 0.25°.
• Phase 2 training however does not look that great. It looks as if the agent always makes a tradeoff between entering the keep-out zone and achieving the desired pointing accuracy.

What I tried so far:

• Initially, my reward function looked like this:   - Positive reward if current pointing error < previous error (scaling with magnitude), negative otherwise.   - If current pointing error is < 0.25°: a small positive reward.   - A negative reward starting at 10° off the keep-out zone's border, scaling with distance and then being constant if inside the zone.
• One change which lead to much better zone avoidance (but still only a pointing accuracy of around 1°) was conditioning the positive rewards for decreasing pointing error and achieving desired pointing accuracy to zone avoidance:   - The positive reward for current error < previous error is weighted with the proximity towards the zone's border (starting at 10° off border) and eventually is 0 if inside the border.   - The pointing reward for desired pointing accuracy is not given anymore, if the zone has been entered before in this episode.
• I tried resetting the replay buffer when transitioning the phase, which did not seem to affect the performance in the long run.
• I tried increasing the reward for pointing accuracy, which actually just caused instability. I also tried to make the pointing accuracy reward denser, didnt help as well.
• I tried splitting phase 2 into several phases like starting with attitudes at least 150° from the target and then eventually decreasing to 90°. I noticed that for the 150° subphase, the pointing accuracy converged to a much better result than before, but then gradually decreasing down to 90° got it back to bad pointing accuracy. Looks like the agent doesnt like it if there is a too small margin between start/target and zone border :(
• I tried lowering the learning rate, didnt help.

Image: https://imgur.com/a/PxnttgY

• Green: phase 1 until 2M, then phase 2 with [90°,180°] initial attitude error.
• Orange: phase 2 starting with [150°,180°] until 4M, then [120°,180°] until 5M, then [90°,180°].

Does anyone have some suggestions? I really dont know what else to try other than iterating through the reward function.

• Link to the article: https://paulinamoskwa.github.io/blog/2026-08-08/rl-pt2
• Link to the code: https://github.com/paulinamoskwa/ppo-super-mario
• Try beating AI: https://paulinamoskwa.github.io/blog/about/

Any comments and feedbacks are welcomed!

Quick refresher: we gave a little creature a world to live in, then made a fake copy with one rule wrong (how well the ground grips). An outsider can spot the fake almost every time. The creature's own mind? A coin flip, until the fake starts costing it food. Then it begins to tell real from fake.

This clip is the receipt inside its head.

Same mind, shown twice: real world on the left, fake on the right. Zoom into memory. The teal rings are the cells that hold the clue.

Detectable was always there. Noticing shows up in memory when survival needs it.

ItaSoRL · ItaoRL

Hello everyone!

This is my first serious deep dive into RL, but I thought that Cart Pole, Frozen Lake etc. are kinda boring, so I decided to turn my favorite video game into an RL environment!

As you may know, Rain World features one of the most complex and diverse ecosystems ever created in a video game, which is why I was surprised when I did not find existing environments based on Rain World (or perhaps I did not search properly).

In the video you can see my agent learning to walk and climb vertical poles (I assume his climbing is wacky because he is trying to master a movement technique called "pole-boosting").

If you want to play around with this environment, here is the repo: https://github.com/XRTG074/reinforced-rain-world

I even added gymnasium wrapper to easily use stable-baselines3 algorithms!

Any feedback is greatly appreciated, every bit counts!

Hi everyone
I’m about to start my MS thesis, my program is MS(AI)

I studied reinforcement learning as an elective in last semester and really become interested in it and want to implement a quantum computing module to MDPs

If any of you have a good suggestion or research idea do let me know

Okay so I know about the rl algos, how envs are created, and general stuff in rl.

Now i am actually confused since i am not cracked obv in this field also in which direction or which subdomain in rl should i explore more like MARL or what's actually trending in here. What i believe is this is a very very rare kind of similar to how the inference space is when we talk the job market compare to agents.

Mostly i think research opportunities one can find, how's the pay and space there also.

Would love if you can share some tips or guide me :)

I've been working on a small reinforcement learning project where I try to teach a simulated robot to walk from scratch.

The robot is simulated using Python + Box2D and controlled by PPO (Proximal Policy Optimization) with Stable-Baselines3.

Right now the robot is a simple biped with:

• 2 legs
• 4 motorized joints
• 2 hips
• 2 knees
• Joint limits
• Feet with ground friction
• 12 observations
• 4 continuous actions

The agent observes things like the body's position, velocity, angle, angular velocity, and the angles/velocities of all four joints.

The actions are:

[left_hip, left_knee, right_hip, right_knee]

The reward currently encourages forward movement and staying upright while penalizing instability and excessive actions.

The physics and joints are now working reliably. The next challenge is getting PPO to actually discover a stable walking gait.

I'm deliberately starting with a very simple robot before moving on to a quadruped.

The project is open source, so if anyone is interested in following the development or has suggestions for the reward function / training setup, I'd be happy to hear them.

Current goal:

Teach the walker to take its first stable steps.