We study the variance of Q-function estimators in finite-horizon, finite-state Markov decision process (MDP) tree search. We show that the variance decomposes into three components attributed to the immediate reward collected, probabilistic state transitions, and uncertainty in future state value function estimates. Using this decomposition, we show that the sample variance estimator based on the assumption of i.i.d. paths is biased, underestimating the true variance, and the bias does not vanish in the limit. We then propose a recursive variance estimator that is consistent. To enable efficient storage and computation, we derive an equivalent implementation of the recursive estimator using only node-local statistics that can be iteratively updated. This consistent variance estimator is integrated into two Monte Carlo Tree Search (MCTS) sampling procedures for finite-horizon MDPs. In numerical examples from inventory control and kidney paired donation matching, the new estimator improves the performance of the MCTS algorithm relative to a baseline that uses the i.i.d.-based sample variance estimator.
Flight test is shifting toward a data-centric approach in which data contribute to model refinement, reducing reliance on pre-scripted test points. An open problem is how to sequence maneuvers within a sortie to maximize uncertainty reduction under resource limits. We present a r…
We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, w…
This paper investigates the control of discrete-time linear time-invariant (LTI) systems subject to incomplete and corrupted measurements. Specifically, we focus on designing a Linear Quadratic Gaussian (LQG) controller without relying on explicit state estimation. By leveraging…
The Subspace-based Model Identification algorithm using a modified Iterative Principal Component Analysis (SMI-IPCA) is a theoretically rigorous method for identifying a linear state-space model of a multi-input multi-output (MIMO) process, in an errors-in-variables (EIV) setting…
Accurate and reliable environmental mapping is a fundamental requirement for multi-robot autonomy. While continuous mapping techniques like Gaussian Process Occupancy Mapping (GPOM) provide rich spatial correlation and uncertainty estimates, they lack formal, finite-sample guaran…
Control barrier functions (CBFs) provide formal safety guarantees for constrained nonlinear systems, but their effectiveness relies on accurate system dynamics. In high-dimensional manipulators subject to unknown disturbances and model uncertainties, fixed safety constraints cons…