A fully adaptive framework for continuous-state stochastic dynamic programming

Approximate dynamic programming (ADP) carries out approximation of the future value function (FVF) to enable numerical solutions to dynamic programming (DP). Recent ADP methodologies often employ the design and analysis of computer experiment (DACE) techniques for the FVF approximation. Use of DACE-...

Full description

Saved in:
Bibliographic Details
Published in:Computers & operations research Vol. 183; p. 107160
Main Authors: Fan, Huiyuan, Tarun, Prashant K., Viswanatha, Amith, Chen, Victoria C.P.
Format: Journal Article
Language:English
Published: Elsevier Ltd 01.11.2025
Subjects:
Approximate dynamic programming
Continuous state space
Sequential exploration
Value function approximation
Sequential exploration
Approximate dynamic programming
Value function approximation
Continuous state space
ISSN:0305-0548
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Approximate dynamic programming (ADP) carries out approximation of the future value function (FVF) to enable numerical solutions to dynamic programming (DP). Recent ADP methodologies often employ the design and analysis of computer experiment (DACE) techniques for the FVF approximation. Use of DACE-based ADP approach, however, creates a “chicken and egg” situation where we cannot collect the data for statistical modeling until we know the state space region, but we do not know the state space region until we collect the data. To overcome this dilemma, this paper introduces a sequential state space exploration (SSSE) approach to adaptively identify the state space region for the experimental design while also sampling useful data for the statistical model. In the proposed methodology, the SSSE approach works in tandem with an adaptive value function approximation (AVFA) algorithm that gradually grows the complexity of the statistical model as more data are observed. This novel SSSE-AVFA approach features a “fully adaptive dynamic programming” algorithm, which can automatically and appropriately identify the three critical components (state space region, sample size of the data, and statistical model structure) for FVF approximation, thereby eliminating the need for time-consuming trial-and-error computational runs that were previously required. The SSSE-AVFA approach is examined with a nine-dimensional inventory forecasting problem and is compared with fixed structure runs in which the state space region, sample size of the data, and statistical model structure are assumed in advance. Our proposed methodology ensured either that the established solutions could be more reasonable or that the modeling process could effectively save the computational effort. With its full adaptiveness in determining those critical components, the SSSE-AVFA approach has the potential to be more effective and efficient than the traditional methods in handling a wide range of real-world continuous-state DP problems. •Proposes a novel adaptive dynamic programming methodology to solve a high-dimensional, continuous-state, multistage, stochastic dynamic programming (SDP) problem.•Presents a methodology with a unique ability to automatically and adaptively identify the state space, sample size, and statistical model structure for future value function approximation for an SDP problem.•Demonstrates the efficiency and efficacy of the proposed methodology using a nine- dimensional inventory forecasting problem.
ISSN:0305-0548
DOI:10.1016/j.cor.2025.107160