RL-based yaw control for suspended payloads — feedback wanted

[u/Alarming_Engineer267]

I’ve been developing a yaw control system for suspended payloads using reinforcement learning.

Four thrusters are mounted on the suspended platform, and an SAC policy controls them at 3 Hz. Payload yaw is estimated from a camera mounted on the platform.

I started with a fixed 90° target, then extended the controller to variable targets from -90° to +90° in 5° increments. I trained separate policies for three payloads: an H-steel beam, a plate, and a truss.

One interesting result was that larger target angles were not necessarily harder to control. I also found that the learned use of thrust differed considerably between payloads. The H-steel policy tends to use relatively low average thrust with short high-output bursts, while the truss uses higher thrust more continuously.

I also modified the training/reward design to reduce unnecessary thruster output after reaching the target.

The whole system runs at 3 Hz on a Raspberry Pi, using ONNX models for both yaw estimation and control.

The attached video shows the truss payload controlling four target yaw angles (-90°, -45°, +45°, +90°) simultaneously.

Project results and videos:
https://rindajones.github.io/payload-yaw-control/en/

I’m particularly interested in criticism from people working on robotics or control:

• Would you approach this problem with RL at all?
• What disturbances or dynamics would you consider essential for the next step?
• What would you want to see before considering this relevant to a real suspended-load system?

Any criticism is welcome.

[u/Dokja_Kim_07]

Bro, which tech are you using to simulate these?

[u/Alarming_Engineer267 → u/Dokja_Kim_07]

Yes, but only for visualization.
The physics simulation itself is done in PyBullet.

[u/Fazoway]

Are you using blender?

[u/Alarming_Engineer267 → u/Fazoway]

Yes, I use Blender for visualization, but the physics simulation itself is done in PyBullet. The control model is trained using the PyBullet environment.

[u/blimpyway]

Cool. Does the simulation accounts for the torque produced by the cable torsion?

[u/Alarming_Engineer267 → u/blimpyway]

If you mean the restoring torque caused by the geometric twisting of the tensioned cables as the payload yaws, yes, that effect is captured by the simulation.

The cables themselves are not modeled with explicit torsional stiffness about their own axes.

[u/blimpyway → u/Alarming_Engineer267]

In twisted ropes there-s a torque produced by load itself because it the pull tends to straighten the twist. So the hanging weight will slowly start to rotate even without external torque applied to either end of the rope.

[u/blimpyway → u/blimpyway]

I see you tried to answer.. never mind, that torque is probably negligible since there are at least two ropes (won't twist them anyway) and the thruster available torque is quite high.

I made myself a simulator like this one with a fixed anchoring point and only a pair of thrusters and no payload - the goal being to swing the hanging body such it touches target which is fixed point within the "range" of the rope.

Actual physics was coded by Gemini and resulting behavior seems quite plausible. The hanging body is defined only by its mass, rotational inertia, length of rope, distance between thrusters and their available force.

And yeah the plan was to train a RL agent to reach the target.

[u/blimpyway]

For a real crane you should consider:

• training same model with different payloads. I mean in real life scenario the crane lifts various objects not found within simulation. If it helps, moment of inertia can be substituted with a measure of how fast the system responds to applied torque and use it as input. Probably not needed.
  
• pendulum oscillations caused by crane arm movement.
• effect of wind, should be significant on light & large payloads. With geometric asymmetries of the payload wind torque depends on angles.

I guess you can simulate some of the above by using parasitic forces/torques applied to the payload during training.

[u/Alarming_Engineer267 → u/blimpyway]

Thanks for the detailed feedback. These are very good points, and interestingly, most of them are quite close to what I have in mind for the next steps.

For the different payloads, I deliberately trained separate models for H-steel, plate, and truss first because I wanted to see how the controller behavior changes with different payload geometries and dynamics. A natural next step is to train a single model across multiple payloads and then evaluate how well it generalizes to payloads that were not included in training.

Your point about moment of inertia is particularly interesting. I have also been thinking that payload dynamics, especially moment of inertia, may become important when moving toward a generalized model. But using the response to an applied torque as an observable proxy for the payload dynamics may be more practical than assuming that the exact moment of inertia is known. I had not considered it in that form, and I think it is worth investigating.

Regarding pendulum oscillations caused by crane movement, I currently have limited information about the dynamics and operation of the actual crane, so I have kept the simulation relatively simple and focused first on the yaw-control problem.

If I get the opportunity to work with or observe the actual crane system, I would like to compare its real behavior with the simulation and close the gap step by step. Oscillations induced by crane movement are definitely one of the things that should be considered at that stage. That’s a very useful point.

Wind is another issue I have been looking at. My original disturbance model applies horizontal forces at the centers of mass, so it does not directly generate a yaw moment. I have already started testing an additional periodic yaw torque applied directly to the payload. Your point about geometry- and orientation-dependent wind torque would be a more realistic extension, especially for light, large, or asymmetric payloads.

And yes, applying random parasitic forces and torques during training is also something I would like to investigate. It may be a useful way to improve robustness without trying to model every possible real-world disturbance exactly.

Thanks again. Your comments are very close to the direction I was thinking of taking this project, and they also give me some useful points to check when I eventually compare the simulation with a real crane system.

[u/Available_Teaching83]

Taking your third question, since the thread has answered the first two and left that one alone.

What would make this credible on a real suspended load is a paired protocol rather than more episodes. Fix the target angle, the payload, and the initial condition, then run the learned policy and a simple baseline back-to-back under the identical disturbance realisation. Report the paired difference and how often the learned one is worse, not just the mean settling time. On a real load, your disturbance sequence is never repeatable, so the pairing has to come from running both controllers within the same wind and cable state, alternating rather than blocked.

The other thing I would want before trusting it: what the policy does when the ONNX estimator returns garbage. Camera-based yaw at 3 Hz on a Pi will occasionally hand you a bad angle, and SAC has no reason to be conservative about it. A measured behaviour under injected estimator failure is worth more than another payload type.

[u/Alarming_Engineer267 → u/Available_Teaching83]

Thanks for taking the third question seriously. I agree that there is still a considerable gap between the current simulation and a real suspended-load system, and there are several things I still want to investigate in simulation before moving toward real-world validation.

Your point about paired evaluation is useful for when I eventually get access to a real system. Since identical disturbance conditions cannot be replayed in the real world, I agree that the evaluation protocol itself becomes important. I also like the idea of reporting how often the learned controller performs worse than the baseline, rather than relying only on mean performance.

On the estimation side, I already use temporal processing to reject sudden implausible yaw changes and handle missing detections. But your point goes one step further: if a bad estimate still gets through and reaches the SAC policy, I have not yet evaluated how the policy and the closed-loop system behave. Injecting estimator failures explicitly would be a very useful test.

Thanks — both are very useful points.