imitation learning

Expert demonstrations are recordings of an expert performing the desired task, often collected as state-action pairs $(o\_t^*,\; a\_t^*)$.

Behavior Cloning (BC) is the most basic form of imitation learning. Essentially, it uses supervised learning to train a policy $\backslash pi\_\backslash theta$ such that, given an observation $o\_t$, it would output an action distribution $\backslash pi\_\backslash theta(\backslash cdot\; |\; o\_t)$ that is approximately the same as the action distribution of the experts.[https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/slides/lec-2.pdf CS 285 at UC Berkeley: Deep Reinforcement Learning. Lecture 2: Supervised Learning of Behaviors]

BC is susceptible to distribution shift. Specifically, if the trained policy differs from the expert policy, it might find itself straying from expert trajectory into observations that would have never occurred in expert trajectories.

This was already noted by ALVINN, where they trained a neural network to drive a van using human demonstrations. They noticed that because a human driver never strays far from the path, the network would never be trained on what action to take if it ever finds itself straying far from the path.

Dagger (Dataset Aggregation){{Cite journal |last1=Ross |first1=Stephane |last2=Gordon |first2=Geoffrey |last3=Bagnell |first3=Drew |date=2011-06-14 |title=A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning |url=https://proceedings.mlr.press/v15/ross11a.html |journal=Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics |language=en |publisher=JMLR Workshop and Conference Proceedings |pages=627–635}} improves on behavior cloning by iteratively training on a dataset of expert demonstrations. In each iteration, the algorithm first collects data by rolling out the learned policy $\backslash pi\_\backslash theta$. Then, it queries the expert for the optimal action $a\_t^*$ on each observation $o\_t$ encountered during the rollout. Finally, it aggregates the new data into the dataset$$D\; \backslash leftarrow\; D\; \backslash cup\; \backslash \{\; (o\_1,\; a\_1^*),\; (o\_2,\; a\_2^*),\; ...,\; (o\_T,\; a\_T^*)\; \backslash \}$$and trains a new policy on the aggregated dataset.

File:Decision_Transformer_architecture.png

The Decision Transformer approach models reinforcement learning as a sequence modelling problem.{{Cite journal |last1=Chen |first1=Lili |last2=Lu |first2=Kevin |last3=Rajeswaran |first3=Aravind |last4=Lee |first4=Kimin |last5=Grover |first5=Aditya |last6=Laskin |first6=Misha |last7=Abbeel |first7=Pieter |last8=Srinivas |first8=Aravind |last9=Mordatch |first9=Igor |date=2021 |title=Decision Transformer: Reinforcement Learning via Sequence Modeling |url=https://proceedings.neurips.cc/paper/2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=34 |pages=15084–15097|arxiv=2106.01345 }} Similar to Behavior Cloning, it trains a sequence model, such as a Transformer, that models rollout sequences $$(R\_1,\; o\_1,\; a\_1),\; (R\_2,\; o\_2,\; a\_2),\; \backslash dots,\; (R\_t,\; o\_t,\; a\_t),$$where $R\_t\; =\; r\_t\; +\; r\_\{t+1\}\; +\; \backslash dots\; +\; r\_T$ is the sum of future reward in the rollout. During training time, the sequence model is trained to predict each action $a\_t$, given the previous rollout as context:$$(R\_1,\; o\_1,\; a\_1),\; (R\_2,\; o\_2,\; a\_2),\; \backslash dots,\; (R\_t,\; o\_t)$$During inference time, to use the sequence model as an effective controller, it is simply given a very high reward prediction $R$, and it would generalize by predicting an action that would result in the high reward. This was shown to scale predictably to a Transformer with 1 billion parameters that is superhuman on 41 Atari games.{{Citation |last1=Lee |first1=Kuang-Huei |title=Multi-Game Decision Transformers |date=2022-10-15 |url=https://arxiv.org/abs/2205.15241 |access-date=2024-10-22 |arxiv=2205.15241 |last2=Nachum |first2=Ofir |last3=Yang |first3=Mengjiao |last4=Lee |first4=Lisa |last5=Freeman |first5=Daniel |last6=Xu |first6=Winnie |last7=Guadarrama |first7=Sergio |last8=Fischer |first8=Ian |last9=Jang |first9=Eric}}

See {{Cite arXiv |last1=Hester |first1=Todd |last2=Vecerik |first2=Matej |last3=Pietquin |first3=Olivier |last4=Lanctot |first4=Marc |last5=Schaul |first5=Tom |last6=Piot |first6=Bilal |last7=Horgan |first7=Dan |last8=Quan |first8=John |last9=Sendonaris |first9=Andrew |date=2017-04-12 |title=Deep Q-learning from Demonstrations |class=cs.AI |eprint=1704.03732v4 |language=en}}{{Cite journal |last1=Duan |first1=Yan |last2=Andrychowicz |first2=Marcin |last3=Stadie |first3=Bradly |last4=Jonathan Ho |first4=OpenAI |last5=Schneider |first5=Jonas |last6=Sutskever |first6=Ilya |last7=Abbeel |first7=Pieter |last8=Zaremba |first8=Wojciech |date=2017 |title=One-Shot Imitation Learning |url=https://proceedings.neurips.cc/paper_files/paper/2017/hash/ba3866600c3540f67c1e9575e213be0a-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=30}} for more examples.

Inverse Reinforcement Learning (IRL) learns a reward function that explains the expert's behavior and then uses reinforcement learning to find a policy that maximizes this reward.{{Cite journal |last=A |first=Ng |date=2000 |title=Algorithms for Inverse Reinforcement Learning |url=https://cir.nii.ac.jp/crid/1570854174323564160 |journal=Proc. Of 17th International Conference on Machine Learning, 2000 |pages=663–670}} Recent works have also explored multi-agent extensions of IRL in networked systems.V. S. Donge, B. Lian, F. L. Lewis and A. Davoudi, "Multiagent Graphical Games With Inverse Reinforcement Learning," in IEEE Transactions on Control of Network Systems, vol. 10, no. 2, pp. 841-852, June 2023, doi:10.1109/TCNS.2022.3210856.

Generative Adversarial Imitation Learning (GAIL) uses generative adversarial networks (GANs) to match the distribution of agent behavior to the distribution of expert demonstrations.{{Cite journal |last1=Ho |first1=Jonathan |last2=Ermon |first2=Stefano |date=2016 |title=Generative Adversarial Imitation Learning |url=https://proceedings.neurips.cc/paper_files/paper/2016/hash/cc7e2b878868cbae992d1fb743995d8f-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=29|arxiv=1606.03476 }} It extends a previous approach using game theory.{{Cite journal |last1=Syed |first1=Umar |last2=Schapire |first2=Robert E |date=2007 |title=A Game-Theoretic Approach to Apprenticeship Learning |url=https://proceedings.neurips.cc/paper_files/paper/2007/hash/ca3ec598002d2e7662e2ef4bdd58278b-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=20}}

• Reinforcement learning
• Supervised learning
• Inverse reinforcement learning

• {{Cite journal |last1=Hussein |first1=Ahmed |last2=Gaber |first2=Mohamed Medhat |last3=Elyan |first3=Eyad |last4=Jayne |first4=Chrisina |date=2018-03-31 |title=Imitation Learning: A Survey of Learning Methods |url=https://dl.acm.org/doi/10.1145/3054912 |journal=ACM Computing Surveys |language=en |volume=50 |issue=2 |pages=1–35 |doi=10.1145/3054912 |hdl=10059/2298 |issn=0360-0300|hdl-access=free }}

{{Artificial intelligence navbox}}

Category:Supervised learning