Seminar Computational Intelligence A (708.111)

SS 2006

Institut für Grundlagen der Informationsverarbeitung (708)

Lecturer: O.Univ.-Prof. Dr. Wolfgang Maass

Office hours: by appointment (via e-mail)


Location: IGI-seminar room, Inffeldgasse 16b/I, 8010 Graz
Date: starting from March 14, 2006 every Tuesday, 16.15 - 18.00 p.m.

Content of the seminar:

The goal of the Computational Intelligence Seminar A will be to present the most important
new ideas regarding methods for learning to act. More precisely, we focus on methods and
ideas for enabling a humanoid robot to learn to move. The concrete robot is the HOAP 2
(see ), on which we are
working in a joint research project with the EPFL (see
Methods from classical reinforcement learning (that are commonly covered in Machine Learning B)
have so far not been very successful in such type of task. But we will nevertheless give at the
beginning of the seminar a tutorial on reinforcement learning.

Those among the papers listed below that are not publicly available, can be found in our
pdf-archive (students who are not working at our institute can get a copy from Frau Daniela


Michael Pfeiffer: Tutorial on Reinforcement Learning
(the slides from his talk are in the pdf archive)

The book by Sutton and Barto is completely available online. It is still the most important
introductory textbook on Reinforcement Learning:

The slides of the RL tutorial by Satinder Singh at last year's NIPS are also available online:

In the "Course Material" section of our course Machine Learning B one can also find slides
for a lot of RL topics:

Finally the slides of Michael Pfeiffer's 2004 talk about "RL for Motor Control" are also available:

Gerhard Neumann and Michael Pfeiffer: Algorithms, benchmark problems and applications
of Reinforcement Learning

Additional material can be found in Gerhard Neumann's Reinforcement Learning Toolbox:

Helmut Hauser and Gerhard Neumann: The humanoid robot HOAP 2 as real-world challenge
for learning motor control: Introduction to basic results on the statics and dynamics of humanoid
robots, in particular for balance control.

Information about Hoap 2:


a) Michael Rabatscher: Multiple inverse models for motor control
Presentation: (PPT)

--Kawato M: Internal models for motor control and trajectory planning.
Current Opinion in Neurobiology, 9,718-727(1999),
(see for further work by Kawato)

--C. Miall:  Modular motor learning (review),
Trends Cogn Sci. 2002 Jan 1;6(1):1-3.

b) Martin Bachler: Work by Daniel Wolpert on motor control that is relevant for learning motor
control in robots

Publications of Wolpert:


a) Ashley Mills: ISO-learning
Presentation: (PPT)

On selected material from:
--Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and
Their Relation to Biological Mechanisms --
Wörgötter and Porr 17 (2) :   245 -- Neural Computation, 2005

b) Gerhard Neumann: ICO-learning

Porr, B., and Wörgötter, F. (2006) Strongly improved stability and faster convergence of temporal
sequence learning by utilising input correlations only.  Neural Comp. (in press).


a) Joshi Prashant: New results by d'Avella and Bizzi on biological motor primitives

d'Avella, A., Bizzi, E. (2005) Shared and specific muscle synergies in natural motor behaviors.
Proceedings of the National Academy of Sciences of the United States of America 102(8): 3076-3081
(other papers by from the Bizzi-Lab at MIT are online available from )

b) Andreas Juffinger: Lerarning and synthesis of movement primitives
for complex humanoid movements

Ilg W, Bakir G H, Mezger J, Giese M A (2004) On the representation, learning and transfer of spatio-
temporal movement characteristics. International  Journal of Humanoid Robotics, Vol 1, Number 4,
pp 613-636 from


a) Mario Hoefler: Learning of reaching movements through adaptive combination of motor primitives

Material: Thoroughman KA & Shadmehr, R. Learning of action through adaptive combination of
motor primitives. Nature, 407, 742-747 (2000)

b) Tomislav Nad:  Receptive Field Weighted Regression

Material: Atkeson and Schaal, Receptive Field Weighted Regression.

Christian Hoeflechner: Results and open problems concerning the role of basal ganglia in movement control.

The basal ganglia: learning new tricks and loving it.
Graybiel AM., Curr Opin Neurobiol, 2005)
and review talks on basal ganglia from the Okinawa Summer School 2005.

                                                                                                                                                                                                                        2006-03-23, daniela