Title |
Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture
|
---|---|
Published in |
Frontiers in Neurorobotics, January 2013
|
DOI | 10.3389/fnbot.2013.00005 |
Pubmed ID | |
Authors |
Cai Li, Robert Lowe, Tom Ziemke |
Abstract |
The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value. |
Mendeley readers
Geographical breakdown
Country | Count | As % |
---|---|---|
Germany | 4 | 6% |
United States | 2 | 3% |
Iran, Islamic Republic of | 1 | 1% |
Japan | 1 | 1% |
Unknown | 59 | 88% |
Demographic breakdown
Readers by professional status | Count | As % |
---|---|---|
Student > Ph. D. Student | 15 | 22% |
Student > Master | 13 | 19% |
Researcher | 7 | 10% |
Student > Doctoral Student | 7 | 10% |
Professor | 2 | 3% |
Other | 9 | 13% |
Unknown | 14 | 21% |
Readers by discipline | Count | As % |
---|---|---|
Engineering | 24 | 36% |
Computer Science | 11 | 16% |
Medicine and Dentistry | 5 | 7% |
Nursing and Health Professions | 2 | 3% |
Unspecified | 2 | 3% |
Other | 9 | 13% |
Unknown | 14 | 21% |