Е-Публикации
Технически университет - София

Abstract: This article presents the synthesis, real-time implementation, and experimental validation of an approximated adaptive dynamic programming (AADP) actor–critic controller for precise flow rate regulation of a variable-displacement axial-piston pump designed for open-circuit hydraulic systems. Replacing the conventional hydro-mechanical regulator with an electrohydraulic proportional spool valve, the model-free controller employs two compact two-layer neural networks: the actor generates valve PWM signals from the flow tracking error, its integral, and measured discharge pressure, while the critic approximates the infinite-horizon quadratic cost-to-go via the online solution of the Bellman equation through gradient descent on Bellman residuals. Lyapunov analysis establishes closed-loop stability under bounded learning rates, with initial weights tuned via nominal plant simulation to ensure convergence from feasible starting policies. After extensive laboratory testing across four fixed loading conditions and dynamic load variations, the adaptive controller demonstrated superior performance compared with a proportional-integral (PI) controller, a Lyapunov model-reference adaptive controller (LMRAC), and an H∞ controller (Hinf). Real-time metrics confirm bounded critic signals and near-zero Bellman errors, validating optimal policy convergence amid unmodeled hydraulic nonlinearities.

References

1. Findeisen D. Helduser S. Ölhydraulik Springer Berlin, Germany 2015
2. Ivantysyn J. Ivantysynova M. Hydrostatic Pumps and Motors: Principles, Design, Performance, Modelling, Analysis, Control and Testing Academia Books International New Delhi, India 2001
3. Manring N. Fluid Power Pumps and Motors: Analysis, Design, and Control McGraw-Hill Education Columbus, OH, USA 2013
4. Frankenfield T. Using Industrial Hydraulics Rexroth Worldwide Hydraulics Penton Publishing Inc. New York, NY, USA 1984
5. Tonyan M. Electronically Controlled Proportional Valves Marcel Dekker, Inc. New York, NY, USA 1985
6. Skarpetis M.G. Automatic Control of Hydraulic Systems Nova Science Publishers, Inc. New York, NY, USA 2023
7. Zhou K. Doyle J. Robust and Optimal Control Prentice Hall International Upper Saddle River, NJ, USA 1996
8. Åström K.J. Wittenmark B. Adaptive Control Courier Corporation North Chelmsford, MA, USA 2008
9. Park S. Lee J. Kim J. Robust control of the pressure in a control-cylinder with direct drive valve for the variable displacement axial piston pump Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng. 2009 223 455 465 10.1243/09596518JSCE677
10. Zeiger G. Akers A. The application of linear optimal control techniques to axial piston pump controller design SAE Tech. Pap. 1989 890953 10.4271/890953
11. Lin S. Akers A. Optimal control theory applied to pressure-controlled axial piston pump design J. Dyn. Syst. Meas. Control Trans. ASME 1990 112 475 481 10.1115/1.2896167
12. Berg H. Ivantysynova M. Design and testing of a robust linear controller for secondary controlled hydraulic drive Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng. 1999 213 375 385 10.1243/0959651991540223
13. Zeiger G. Akers A. Dynamic Analysis of an Axial Piston Pump Swashplate Control Proc. Inst. Mech. Eng. Part C J. Mech. Eng. Sci. 1986 1 49 58 10.1243/PIME_PROC_1986_200_093_02
14. Berg H. Ivantysynova M. Robust closed loop speed and angular position control for variable displacement hydraulic motors supplied from a constant pressure mains system Olhydraulik Pneum. 1999 43 405 410
15. Zhang R. Alleyne A. Prasetiawan E. Modeling and H 2/H∞MIMO control of an earthmoving vehicle powertrain J. Dyn. Syst. Meas. Control Trans. ASME 2002 124 625 636 10.1115/1.1515326
16. Lennevi J. Palmberg J.-O. Application and implementation of LQ design method for the velocity control of hydrostatic transmissions Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng. 1995 209 255 268 10.1243/PIME_PROC_1995_209_393_02
17. Heybroek K. Larsson J. Palmberg J.-O. Open Circuit Solution for Pump Controlled. Actuators Proceedings of the 4th FPNI-PhD Symposium Sarasota, FL, USA 13–17 June 2006 27 40
18. Zhang P. Li Y. Research on Control Methods for the Pressure Continuous Regulation Electrohydraulic Proportional Axial Piston Pump of an Aircraft Hydraulic System Appl. Sci. 2019 9 1376 10.3390/app9071376
19. Kemmetmüller W. Fuchshumer F. Kugi A. Nonlinear pressure control of self-supplied variable displacement axial piston pumps Control Eng. Pract. 2010 18 84 93 10.1016/j.conengprac.2009.09.006
20. Wei J. Guo K. Fang J. Tian Q. Nonlinear supply pressure control for a variable displacement axial piston pump Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng. 2015 229 614 624 10.1177/0959651815577546
21. Helian B. Mustalahti P. Mattila J. Chen Z. Yao B. Adaptive robust pressure control of variable displacement axial piston pumps with a modified reduced-order dynamic model Mechatronics 2022 87 102879 10.1016/j.mechatronics.2022.102879
22. Feng Y. Jian Z. Li J. Tao Z. Wang Y. Xue J. Advanced Control Systems for Axial Piston Pumps Enhancing Variable Mechanisms and Robust Piston Positioning Appl. Sci. 2023 13 9658 10.3390/app13179658
23. Busquets E. Ivantysynova M. Handroos H. Discontinuous projection-based adaptive robust control for displacement-controlled actuators J. Dyn. Syst. Meas. Control Trans. ASME 2015 137 8 10.1115/1.4030064
24. Haggag S.A. Robust control and modelling of a heavy equipment variable displacement pump hydraulic system Int. J. Heavy Veh. Syst. 2011 18 288 302 10.1504/IJHVS.2011.041595
25. Guo K. Wei J. Adaptive robust control of variable displacement pumps Proceedings of the American Control Conference Washington, DC, USA 17–19 June 2013
26. Koivumäki J. Mattila J. Adaptive and nonlinear control of discharge pressure for variable displacement axial piston pumps J. Dyn. Syst. Meas. Control Trans. ASME 2017 139 101008 10.1115/1.4036537
27. Mitov A. Slavov T. Kralev J. Comparison of Advanced Multivariable Control Techniques for Axial-Piston Pump Processes 2024 12 1797 10.3390/pr12091797
28. Slavov T. Mitov A. Kralev J. Novel Approach for Robust Control of Axial Piston Pump Mathematics 2025 13 643 10.3390/math13040643
29. Slavov T. Mitov A. Kralev J. Lyapunov-Based Two-Degree-of-Freedom Model Reference Adaptive Control of Axial-Piston Pump Mathematics 2025 13 3513 10.3390/math13213513
30. Mitov A. Kralev J. Slavov T. Angelov I. Design of Embedded Control System for Open Circuit Axial Piston Pump Proceedings of the 22nd International Symposium on Electrical Apparatus and Technologies, SIELA 2022 Bourgas, Bulgaria 1–4 June 2022
31. Mitov A. Slavov T. Kralev J. Rapid Prototyping of H∞ Algorithm for Real-Time Displacement Volume Control of Axial Piston Pumps Algorithms 2023 16 120 10.3390/a16020120
32. Ljung L. System Identification: Theory for the User 2nd ed. Prentice Hall Wilmington, DE, USA 1999
33. Mitov A. Kralev J. Slavov T. Identification of Variable Displacement Axial-Piston Pump with Proportional Valve Control Proceedings of the 14th International Scientific Conference on Aeronautics, Automotive, and Railway Engineering and Technologies Sozopol, Bulgaria 10–13 September 2022
34. Zhou J. Zhang T. Zhang H. Zhang Z. Hong J. Yang J. Energy management strategy for electro-hydraulic hybrid electric vehicles considering optimal mode switching: A soft actor-critic approach trained on a multi-modal driving cycle Energy 2024 305 132172 10.1016/j.energy.2024.132172
35. Krishnakumar K. Limes G. Gundy-Burlet K. Bryant D. An adaptive critic approach to reference model adaptation Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, USA San Francisco, CA, USA 11–14 August 2003
36. Akraminia M. Tatari M. Fard M. Jazar R.N. Designing active vehicle suspension system using critic-based control strategy Nonlinear Eng. 2015 4 141 154 10.1515/nleng-2015-0004
37. Agvik R. Vännman S. Adaptive Control of Hydraulic Drive System Chalmers University of Technology Gothenburg, Sweden 2023
38. Han T. Nie X. Que N. Lu J. Yao J. Yu X. Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning Actuators 2026 15 9 10.3390/act15010009
39. Li Y. Qi X. Research on Motion Control of Hydraulic Manipulator Based on Prescribed Performance and Reinforcement Learning Actuators 2026 15 39 10.3390/act15010039
40. Wei X. Ye J. Xu J. Tang Z. Adaptive Dynamic Programming-Based Cross-Scale Control of a Hydraulic-Driven Flexible Robotic Manipulator Appl. Sci. 2023 13 2890 10.3390/app13052890
41. Su Q. Pei Z. Tang Z. Tracking Control for a Lower Extremity Exoskeleton Based on Adaptive Dynamic Programing Biomimetics 2023 8 353 10.3390/biomimetics8040353
42. He J. Zhou L. Li C. Li T. Huang J. Su S. Control Strategy of Hydraulic Servo Control Systems Based on the Integration of Soft Actor-Critic and Adaptive Robust Control IEEE Access 2024 12 63629 63643 10.1109/ACCESS.2024.3396283
43. Teng W. Wang G. Adaptive Optimal Control Based on Critic-Actor Architecture for Hydraulic Support Cylinder System with Asymmetric Output Error Constraints Eng. Lett. 2025 33 3535 3542
44. Kong Y. Wang Y. Wang Y. Zhu S. Zhang R. Wang L. Deep Reinforcement Learning Trajectory Tracking Control for a Six-Degree-of-Freedom Electro-Hydraulic Stewart Parallel Mechanism Eng 2025 6 212 10.3390/eng6090212
45. Yuan X. Wang Y. Zhang R. Gao Q. Zhou Z. Zhou R. Yin F. Reinforcement Learning Control of Hydraulic Servo System Based on TD3 Algorithm Machines 2022 10 1244 10.3390/machines10121244
46. Singh Sidhu H. Siddhamshetty P. Kwon J.S. Approximate Dynamic Programming Based Control of Proppant Concentration in Hydraulic Fracturing Mathematics 2018 6 132 10.3390/math6080132
47. Rout R. Kumawat A.K. Reinforcement Learning-Based Position Tracking Control for Proportional Directional Control Valve-Based Electro-Hydraulic System IEEE Access 2025 13 159597 159609 10.1109/ACCESS.2025.3608253
48. Jia C. Yu T. Song Z. Robust reinforcement learning with augmented state for leveling control of multi-cylinder hydraulic system J. Supercomput. 2025 81 1 10.1007/s11227-024-06681-x
49. Khater A.A. Fekry M. El-Bardini M. El-Nagar A.M. Deep reinforcement learning-based adaptive fuzzy control for electro-hydraulic servo system Neural Comput. Appl. 2025 37 24607 24624 10.1007/s00521-024-10741-x
50. Hao X. Xin Z. Huang W. Wan S. Qiu G. Wang T. Wang Z. Deep reinforcement learning enhanced PID control for hydraulic servo systems in injection molding machines Sci. Rep. 2025 15 1 10.1038/s41598-025-05904-2
51. Hu P. Wen T. Zhang D. Bayesian reinforcement learning for adaptive control of energy recuperation in hydraulic excavator arms Sci. Rep. 2026 16 6195 10.1038/s41598-026-35391-y
52. Rexroth Bosch Group Pressure and Flow Control System Technical Data Sheet, RE 30630 Rexroth Bosch Group Lohr am Main, Germany 2015
53. Rexroth Bosch Group Proportional Directional Valves, Direct Operated, with Electrical Position Feedback as Pilot Control Valve for Control Systems SY(H)DFE Technical Data Sheet, RE 29016 Rexroth Bosch Group Lohr am Main, Germany 2019
54. Kordak R. Hydrostatic Drives with Control of the Secondary Unit The Hydraulic Trainer Vol.6 Mannesmann Rexroth GmbH Lohr am Main, Germany 1996
55. Danfoss Plus+1 Controllers MC012-020 and 022 Data Sheet, 11077167, Rev DA Danfoss Nordborg, Denmark 2013

Issue