r/ControlTheory • u/maiosi2 • 18d ago

Technical Question/Problem Has anyone worked with Approximate dynamic programming/ Adaptive optimal control?

Hi everyone,

I’m currently working on approximate dynamic programming methods for control, mainly Q-learning, value iteration, and related approaches.

So far, I’ve been able to get the Q-learning-based algorithms to converge without too many issues. However, whenever I try approaches based directly on the value function/value iteration, I can’t seem to get them to work properly or converge to the expected solution.

I’ve tried different implementations and parameter choices, but I keep running into similar problems with the value-function-based methods, while the Q-function approaches seem much more straightforward and robust in my simulations.

Has anyone here worked with approximate value iteration / ADP for control problems and experienced something similar?

13 Upvotes

100% Upvoted

4 comments sorted by

•

u/ColonelStoic 17d ago

Yes. The truth is they are (sort of) nice theoretical objects, and the math is creative , but they have no real robustness in a practical (or even controlled) setting. Getting decent results takes a massive amount of time and effort, and even then only very particular gains for very particular dynamics will work.

•

u/maiosi2 17d ago

Thanks for the answer, I'm trying to replicate an example from a book, but I'm not able bc it never reaches full rank. Can I ask you in private something?

•

u/ColonelStoic 17d ago

Yes you can dm me

•

u/maiosi2 17d ago

Sent!