action model learning
{{Machine learning bar}}
Action model learning (sometimes abbreviated action learning) is an area of machine learning concerned with the creation and modification of a software agent's knowledge about the effects and preconditions of the actions that can be executed within its environment. This knowledge is usually represented in a logic-based action description language and used as input for automated planners.
Learning action models is important when goals change. When an agent acted for a while, it can use its accumulated knowledge about actions in the domain to make better decisions. Thus, learning action models differs from reinforcement learning. It enables reasoning about actions instead of expensive trials in the world.
{{cite journal
| last1 = Amir | first1 = Eyal
| last2 = Chang | first2 = Allen
| title = Learning Partially Observable Deterministic Action Models
| journal = Journal of Artificial Intelligence Research
| volume = 33
| pages = 349–402
| year = 2008
| url = http://dl.acm.org/citation.cfm?id=1622708
| doi = 10.1613/jair.2575
| arxiv = 1401.3437
| s2cid = 9432224
}}
Action model learning is a form of inductive reasoning, where new knowledge is generated based on the agent's observations.
The usual motivation for action model learning is the fact that manual specification of action models for planners is often a difficult, time consuming, and error-prone task (especially in complex environments).
Action models
Given a training set consisting of examples , where are observations of a world state from two consecutive time steps and is an action instance observed in time step , the goal of action model learning in general is to construct an action model , where is a description of domain dynamics in action description formalism like STRIPS, ADL or PDDL and is a probability function defined over the elements of .
{{cite journal
| doi = 10.1080/08839514.2014.927692
| last = Čertický | first = Michal
| title = Real-Time Action Model Learning with Online Algorithm 3SG
| journal = Applied Artificial Intelligence
| volume = 28
| issue = 7 | pages = 690–711
| year = 2014
| s2cid = 8210810 | doi-access = free
}}
However, many state of the art action learning methods assume determinism and do not induce . In addition to determinism, individual methods differ in how they deal with other attributes of domain (e.g. partial observability or sensoric noise).
Action learning methods
= State of the art =
Recent action learning methods take various approaches and employ a wide variety of tools from different areas of artificial intelligence and computational logic. As an example of a method based on propositional logic, we can mention SLAF (Simultaneous Learning and Filtering) algorithm, which uses agent's observations to construct a long propositional formula over time and subsequently interprets it using a satisfiability (SAT) solver. Another technique, in which learning is converted into a satisfiability problem (weighted MAX-SAT in this case) and SAT solvers are used, is implemented in ARMS (Action-Relation Modeling System).
{{cite journal
|last1=Yang
|first1=Qiang
|last2=Kangheng
|first2=Wu
|last3=Yunfei
|first3=Jiang
|title=Learning action models from plan examples using weighted MAX-SAT
|journal=Artificial Intelligence
|date=2007
|volume=171
|issue=2–3
|pages=107–143
|doi=10.1016/j.artint.2006.11.005|doi-access=free
|citeseerx=10.1.1.135.9266
}}
Two mutually similar, fully declarative approaches to action learning were based on logic programming paradigm Answer Set Programming (ASP){{cite journal|last1=Balduccini|first1=Marcelo|title=Learning Action Descriptions with A-Prolog: Action Language C|journal=AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning|date=2007|pages=13–18|url=http://www.aaai.org/Library/Symposia/Spring/2007/ss07-05-004.php}} and its extension, Reactive ASP.{{cite book|last1=Čertický|first1=Michal |chapter=Action Learning with Reactive Answer Set Programming: Preliminary Report |date=2012|pages=107–111|chapter-url=http://www.thinkmind.org/index.php?view=article&articleid=icas_2012_5_20_20056|title=ICAS 2012 : The Eighth International Conference on Autonomic and Autonomous Systems|isbn=9781612081878}} In another example, bottom-up inductive logic programming approach was employed.{{cite journal|last1=Benson|first1=Scott|title=Inductive learning of reactive action models|journal=Machine Learning: Proceedings of the Twelfth International Conference|date=1995}} Several different solutions are not directly logic-based. For example, the action model learning using a perceptron algorithm{{cite journal|last1=Mourao|first1=Kira|last2=Petrick|first2=Ronald|last3=Steedman|first3=Mark|title=Learning action effects in partially observable domains|journal= Frontiers in Artificial Intelligence and Applications|date=2010|volume=215|issue=ECAI 2010|pages=973–974|doi=10.3233/978-1-60750-606-5-973|url=http://www.ebooks.iospress.nl/volumearticle/5920|hdl=20.500.11820/810a5579-b991-441a-ad68-af0151689627|hdl-access=free}} or the multi level greedy search over the space of
possible action models.{{cite journal|last1=Zettlemoyer|first1=Luke|last2=Pasula|first2=Hanna|last3=Kaelblin|first3=Leslie Pack|title=Learning planning rules in noisy stochastic worlds|journal=AAAI|date=2005|pages=911–918|url=http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.65.3417}} In the older paper from 1992,{{cite journal|last1=Lin|first1=Long-Ji|title=Self-improving reactive agents based on reinforcement learning, planning and teaching|journal=Machine Learning|date=1992|volume=8|issue=3–4|pages=293–321|doi=10.1023/A:1022628806385|doi-access=free}} the action model learning was studied as an extension of reinforcement learning.
= Literature =
Most action learning research papers are published in journals and conferences focused on artificial intelligence in general (e.g. Journal of Artificial Intelligence Research (JAIR), Artificial Intelligence, Applied Artificial Intelligence (AAI) or AAAI conferences). Despite mutual relevance of the topics, action model learning is usually not addressed in planning conferences like the International Conference on Automated Planning and Scheduling (ICAPS).