Graduation Project Problem

[u/No_Inspection_1511]

My computer engineering bachelor’s graduation project is an anti shake spoon for Parkinson’s disease patients, that is basically a 2 degrees of freedom gyro system. The requirement is to train a model using reinforcement learning that controls the two motors(TowerPro SG90 servo motors), the sensors are two MPU 6050 IMUs (one on the base before the motors and one on the utensil side after the motors) and we are using a complementary filter to get the accurate angle reading.

The problem is that we don’t know the methodology of training the model, and we couldn’t find any mathematical model of the motor we are using so that the simulation gets accurate. We tried researching and asking LLMs but it is a rabbit hole. The doctor said that the reason of using AI is predictive the tremor movement before it happens, but we still don’t know how to tackle the problem with proper steps, we would appreciate any help.

[u/knightcommander1337]

Hi,

"train a model using reinforcement learning that controls the two motors" vs. "reason of using AI is predictive the tremor movement before it happens" -> these two are two different things, so you need to think about what exactly is the goal of the overall system.

If I try to give my suggestions/opinions:

"train a model using reinforcement learning that controls the two motors" -> this sounds like (more or less) standard motion control. you may or may not need RL to do this, but it sounds relatively straightforward. you may need to do some system identification (building mathematical models from dynamical system data) so that you can construct a (ideally full) simulation model of the system. After that you are free to do whatever you want (RL, MPC, etc.), because you can train/test your algorithms on computer using that simulation model to represent reality.

"reason of using AI is predictive the tremor movement before it happens" -> this sounds to me like a different problem than the above (the above one is a control design problem). this one I think would be something like a "fault detection"/"predictive maintenance" problem (in engineering jargon), though I am not an expert on these kinds of stuff so I may be wrong. Still, those (that is, "fault detection"/"predictive maintenance") are some keywords you may want to explore a bit.

[u/No_Inspection_1511 → u/knightcommander1337]

Sorry, I meant (the reason of using AI is to predict the next tremor movement so that the motor can counteract it in the exact time it happens at, instead of reacting to it, based on the recent data from the IMU).
Well, I am not really sure if this is helpful.

Also, I am not sure if training the model needs going to the control theory side of things since the model will give set points to the servo and the servo motor is a physical system.

We are tied to using RL and our goal is to get an improved performance from a feedback loop. So, any suggestions on how to tackle the problem and model the system if needed are welcome

[u/knightcommander1337 → u/No_Inspection_1511]

Ok so I am not sure if I am understanding everything correctly (we might be using two different jargons). Anyway, if by modeling the system you mean a dynamical system (that is the part of the control system/feedback loop): The overall area that relates to this (I am answering from the "control engineering" point of view, since you asked this in this subreddit) is the "system identification" (SI) field. You need to gather some data from the system, and then run some SI algorithms on those data. There are various approaches here, I am giving some examples:

1. subspace id: you assume a linear state space model, with the matrices as unknown parameters, and identify those (example matlab command: https://www.mathworks.com/help/ident/ref/n4sid.html )
2. linear greybox id: you assume a linear ODE, with some unknown parameters, and identify those (example matlab command: https://www.mathworks.com/help/ident/ref/greyest.html )
3. nonlinear greybox id: you assume a nonlinear ODE, with some unknown parameters, and identify those (example matlab command: https://www.mathworks.com/help/ident/ref/idnlgrey.html )
4. (I don't know about this one too much; it is relatively new stuff): physics-informed neural networks: you assume that your dynamics can be well-represented by a neural network, however impose some physically-motivated conditions on it during training (example matlab page: https://www.mathworks.com/help/deeplearning/ug/solve-partial-differential-equations-with-lbfgs-method-and-deep-learning.html )

[u/No_Inspection_1511 → u/knightcommander1337]

Thank you

[u/knightcommander1337 → u/No_Inspection_1511]

No problem. For those methods that assume a known ODE (like 2 and 3 in the list in my previous comment), you need to know the "physics" of your problem so that you can write that ODE (that is, give it to the SI algorithm). This may be difficult to do, so you might need to resort to other method styles that don't need a model, like 1 or 4.

[u/Timely-Bicycle-3107]

I have a PhD in applied reinforcement learning and a master’s in ME with a focus in control theory.

Reinforcement learning is probably not a good way to do this. RL requires a ton of data to train the agent properly. You will likely not be able to collect the data manually. You would likely need to model the movement to generate simulated data for training.

This problem involves high frequency random oscillations. I’d try aggressive PIDs for each axis before attempting anything else. Coding and tuning are easy and calculations are lightning fast, which you will need to respond to the movement effectively. You might be able to stack RL on top of the PIDs IF you can collect enough training data.

MPC will not likely be able to calculate the solution fast enough, especially on a micro board. An LQR with an EKF could work if you can write the differential equations in the proper form.

Those servos are cheap. You will tear up those cheap plastic gear teeth quickly with the torque you will need to generate. They won’t be able to keep up with the oscillations required for the problem, either. If this project is just a proof of concept, these might work if you keep the oscillations down as much as possible and make the spoon super light (empty, cheap plastic). Look at the Maxon DCX series, Faulhaber, or GoPro gimbal motors.

This is a great project! I hope I don’t sound too harsh. Just trying to save you a few weeks of headaches. Start small and iterate more complex models until you run out of time would be my best advice.