Optimizing Automatic Voltage Regulation: A Deep Reinforcement Learning Approach
Abstract
This paper presents a novel approach to enhance the performance of Automatic Voltage Regulator (AVR) systems in power systems using Deep Reinforcement Learning (DRL). The AVR plays a critical role in maintaining voltage stability and ensuring reliable power delivery. However, conventional control strategies, such as PID controllers, have limitations in handling complex and nonlinear power system dynamics. In this study, the application of DRL techniques is explored, particularly the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm, to AVR control. This algorithm offers the advantage of handling continuous action spaces and enable the controller to learn optimal control policies directly from the system's state information. The results show that the DRL approach outperforms the traditional PID and Neural Network-based control approaches, with the shortest time response and the best voltage regulation performance. The use of DRL in AVR system control shows promising potential for improving the efficiency and accuracy of power system control. This research provides insights into the advantages of DRL for process control and highlights its potential for future applications in power system control.
References
1. Chatterjee, V. Mukherjee, and S. P. Ghoshal, Velocity relaxed and craziness-based swarm optimized intelligent PID and PSS controlled AVR system, International Journal of Electrical Power & Energy Systems, Vol. 31, No. 7–8, pp. 323–333, 2009.
2. M. R. ESTAKHROUIEH and A. L. I. A. GHARAVEISI, Optimal iterative learning control design for generator voltage regulation system, Turkish Journal of Electrical Engineering and Computer Sciences, Vol. 21, No. 7, pp. 1909–1919, 2013.
3. G. Bal, O. Kaplan, and S. S. Yalcin, Artificial neural network based automatic voltage regulator for a stand-alone synchronous generator, in 2019 8th International Conference on Renewable Energy Research and Applications (ICRERA), IEEE, pp. 1032–1037, 2019.
4. Tripathi, R. L. Verma, and M. S. Alam, Design of Ziegler Nichols tuning controller for AVR System, International Journal of Research in Electronics & Communication Technology, Vol. 1, No. 2, pp. 154–158, 2013.
5. S. F. M. Khedr, M. E. Ammar, and M. A. M. Hassan, Multi objective genetic algorithm controller’s tuning for non-linear automatic voltage regulator, in 2013 International Conference on Control, Decision and Information Technologies (CoDIT), IEEE, pp. 857–863, 2013.
6. Z.-L. Gaing, A particle swarm optimization approach for optimum design of PID controller in AVR system, IEEE Transactions on Energy Conversion, Vol. 19, No. 2, pp. 384–391, 2004.
7. P. Memon, A. S. Memon, A. A. Akhund, and R. H. Memon, Multilayer perceptrons neural network automatic voltage regulator with applicability and improvement in power system transient stability, International Journal of Emerging Trends in Electrical and Electronics (IJETEE ISSN: 2320-9569), IRET publication, Vol. 9, No. 1, pp. 30–38, 2013.
8. A. Bhutto, F. A. Chachar, M. Hussain, D. K. Bhutto, and S. E. Bakhsh, Implementation of probabilistic neural network (PNN) based automatic voltage regulator (AVR) for excitation control system in Matlab, in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), IEEE, pp. 1–5, 2019.
9. R. Meléndez-Pérez, F. Ortiz-Rodríguez, and D. Ruiz-Vega, Design of an ANFIS Automatic Voltage Regulator of a Synchronous Generator, in 2020 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC), IEEE, pp. 1–6, 2020.
10. M. L. Amer, H. H. Hassan, and H. M. Youssef, Modified evolutionary particle swarm optimization for AVR-PID tuning, in Communications and Information Technology, Systems and Signals 2008, pp. 164–173, 2008.
11. K. Yavarian, F. Hashemi, and A. Mohammadian, Design of intelligent PID controller for AVR system using an adaptive neuro fuzzy inference system, International Journal of Electrical and Computer Engineering (IJECE), Vol. 4, No. 5, pp. 703–718, 2014.
12. T. Gupta and D. K. Sambariya, Optimal design of fuzzy logic controller for automatic voltage regulator, in 2017 International Conference on Information, Communication, Instrumentation and Control (ICICIC), IEEE, pp. 1–6, 2017.
13. M. I. Solihin, L. F. Tack, and M. L. Kean, Tuning of PID controller using particle swarm optimization (PSO), in Proceeding of the International Conference on Advanced Science, Engineering and Information Technology, pp. 458–461, 2011.
14. N. K. Bahgaat and M. A. Moustafa Hassan, Swarm intelligence PID controller tuning for AVR system, Advances in Chaos Theory and Intelligent Control, pp. 791–804, 2016.
15. P. Engelbrecht, Computational Intelligence: An Introduction. John Wiley & Sons, 2007.
16. T. J. Stray, Application of deep reinforcement learning for control problems, NTNU, 2019.
17. L. N. Magangane and K. A. Folly, Neural networks for designing an automatic voltage regulator of a synchronous generator, in 2013 Africon, IEEE, pp. 1–5, 2013.
18. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 2018.
19. Welcome to Spinning Up in Deep RL!-Spinning Up documentation. https://spinningup.openai.com/en/latest/ (accessed Jan. 26, 2023).
20. T. P. Lillicrap et al., Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971, 2015.
21. S. Fujimoto, H. Hoof, and D. Meger, Addressing function approximation error in actor-critic methods, in International Conference on Machine Learning, PMLR, pp. 1587–1596, 2018.
22. S. Priyambada, B. K. Sahu, and P. K. Mohanty, Fuzzy-PID controller optimized TLBO approach on automatic voltage regulator, in 2015 International Conference on Energy, Power and Environment: Towards Sustainable Growth (ICEPE), IEEE, pp. 1–6, 2015.
