An Improved Deep Q-Learning Approach for Navigation of an Autonomous UAV Agent in 3D Obstacle-Cluttered Environment

Ghulam Farid, Muhammad Bilal, Lanyong Zhang, Ayman Alharbi, Ishaq Ahmed, Muhammad Azhar

The performance of the UAVs while executing various mission profiles greatly depends on the selection of planning algorithms. Reinforcement learning (RL) algorithms can effectively be utilized for robot path planning. Due to random action selection in case of action ties, the traditional Q-learning algorithm and its other variants face the issues of slow convergence and suboptimal path planning in high-dimensional navigational environments. To solve these problems, we propose an improved deep Q-network (DQN), incorporating an efficient tie-breaking mechanism, prioritized experience replay (PER), and L2-regularization. The adopted tie-breaking mechanism improves the action selection and ultimately helps in generating an optimal trajectory for the UAV in a 3D cluttered environment. To improve the convergence speed of the traditional Q-algorithm, prioritized experience replay is used, which learns from experiences with high temporal difference (TD) error and avoids uniform sampling of stored transitions during training. This also allows the prioritization of high-reward experiences (e.g., reaching a goal), which helps the agent to rediscover these valuable states and improve learning. Moreover, L2-regularization is adopted that encourages smaller weights for more stable and smoother Q-values to reduce the erratic action selections and promote smoother UAV flight paths. Finally, the performance of the proposed method is presented and thoroughly compared against the traditional DQN, demonstrating its superior effectiveness.