Text

Learning in Structured MDPs with Convex Cost Functions : Improved Regret Bounds for Inventory Management

We consider a stochastic inventory control problem under censored demand, lost sales, and positive lead times. This is a fundamental problem in inventory management, with significant literature establishing near optimality of a simple class of policies called “base-stock policies” as well as the convexity of long-run average cost under those policies. We consider a relatively less studied problem of designing a learning algorithm for this problem when the underlying demand distribution is unknown. The goal is to bound the regret of the algorithm when compared with the best base-stock policy. Our main contribution is a learning algorithm with a regret bound of O˜((L+1)T−−√+D) for the inventory control problem. Here, L≥0 is the fixed and known lead time, and D is an unknown parameter of the demand distribution described roughly as the expected number of time steps needed to generate enough demand to deplete one unit of inventory. Notably, our regret bounds depend linearly on L, which significantly improves the previously best-known regret bounds for this problem where the dependence on L was exponential. Our techniques utilize the convexity of the long-run average cost and a newly derived bound on the “bias” of base-stock policies to establish an almost black box connection between the problem of learning in Markov decision processes (MDPs) with these properties and the stochastic convex bandit problem. The techniques presented here may be of independent interest for other settings that involve large structured MDPs but with convex asymptotic average cost functions.

Ketersediaan

Informasi Detail

Judul Seri

OPERATIONS RESEARCH; Vol.70 No.3 May-June 2022

No. Panggil

-

Penerbit

: .,

Deskripsi Fisik

p. 1646-1664

Bahasa

English

ISBN/ISSN

-

Klasifikasi

NONE

Tipe Isi

-

Tipe Media

-

Tipe Pembawa

-

Edisi

-

Subjek

REGRET BOUNDS
REINFORCEMENT LEARNING
ONLINE CONVEX OPTIMIZATION
CENSORED DEMAND
INVENTORY CONTROL PROBLEM

Info Detail Spesifik

https://doi.org/10.1287/opre.2022.2263

Pernyataan Tanggungjawab

Shipra Agrawal, Randy Jia

Versi lain/terkait

Lampiran Berkas

Komentar