Exercises

These pages provides documentation relevant for the individual exercises.

Exercises

• Exercise 0: Installation and self-test
  • Testing your code and handing in
• Exercise 1: The finite-horizon decision problem
  • Inventory environment
  • Agents
  • Training
  • Pacman
  • Classes and functions
  • Solutions to selected exercises
• Exercise 2: Dynamical Programming
  • The dynamical programming model
  • A primer on Pacman
  • Classes and functions
  • Solutions to selected exercises
• Exercise 3: DP reformulations and introduction to Control
  • Specifying a model
  • Simulating a policy
  • Defining bounds
  • Background: Symbolic expressions and sympy
  • Classes and functions
  • Solutions to selected exercises
• Exercise 4: Discretization and PID control
  • Example: Creating a discrete pendulum environment
  • Pendulum continued: Coordinate transformations
  • Shortcuts
  • Computing derivatives (Jacobians) of the discretized dynamics:
  • The discrete cost
  • Defining a control environment
• Exercise 5: Direct methods and control by optimization
  • Classes and functions
  • Solutions to selected exercises
• Exercise 6: Linear-quadratic problems in control
  • Example: Calling the LQR function
• Exercise 7: Linearization and iterative LQR
  • Iterative LQR
  • Classes and functions
  • Solutions to selected exercises
• Exercise 8: Exploration and Bandits
  • Bandits
  • Classes and functions
  • Solutions to selected exercises
• Exercise 9: Policy and value iteration
  • Markov decision processes
  • Classes and functions
  • Solutions to selected exercises
• Exercise 10: Monte-carlo methods and TD learning
  • Tabular methods (Q-learning, Sarsa, etc.)
  • Classes and functions
  • Solutions to selected exercises
• Exercise 11: Model-Free Control with tabular and linear methods
  • Linear function approximators
  • Classes and functions
  • Solutions to selected exercises
• Exercise 12: Eligibility traces and value-function approximations
  • Solutions to selected exercises
• Exercise 13: Q-learning and deep-Q learning
  • Deep Q-learning
  • Classes and functions
  • Solutions to selected exercises