.. Introduction to reinforcement learning and control theory documentation master file, created by
   sphinx-quickstart on Tue Aug  9 15:29:47 2022.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

.. _index:

Introduction to reinforcement learning and control theory
=====================================================================================
{#
.. warning::

   The exercises are being updated for the 2024 semester. I recommend that you use the online version to make sure you are up to date.
   I will make sure that the exercises are ready at least one week before the lecture.
#}
.. hint::
   :class: margin

   Use :kbd:`ctrl+k` to search.

This page contains material and information related to the spring 2024 version of the course  **{{course_title}}**, offered at DTU.

If you are curious about the course, you can read more :ref:`about the course here <about>`.
If you are enrolled and just starting out, you should begin with the :ref:`installation`. You can find the exercises, project descriptions in the menu to the left.

{#
- :fas:`bell` I have uploaded :ref:`previous exam sets and practical information <exam>` :fas:`bell`
#}
Practicalities
**********************************************************************************************************************
{% if not CE %}
.. note::
   :class: margin

   This page is automatically updated with typos, etc. I therefore recommend bookmarking it and using the newest version of the exercises.

{% if CE %}
:Time and Place: {{ lecture_location }}, 13:00-15:00.
:Online help: :fab:`product-hunt` `piazza.com <{{piazza}}>`_
{% else %}
:Time and place: :fas:`clock` {{ lecture_location }}, {{ lecture_time }}
{% endif %}
{% if not CE %}:DTU Learn: :fa:`house` `02465 <{{dtulearn_homepage}}>`__  {% endif %}
:Exercise code: :fab:`gitlab` `{{students_git }} <{{students_git}}>`__
{% if not CE %}:Course descriptions: :fas:`folder` `kurser.dtu.dk <https://kurser.dtu.dk/course/2023-2024/02465>`__ {% endif %}
:Lecture recordings: :fas:`film` `panopto.dtu.dk <{{videolectures}}>`__
{% if not CE %}:Discord:  :fab:`discord` `Discord channel <{{ discord }}>`__ (`invitation link <{{ discord_invite }}>`__)
:Campus-wide python support:  :fas:`circle-info` `pythonsupport.dtu.dk <https://pythonsupport.dtu.dk/>`__
:Contact: :fas:`envelope` Tue Herlau, `{{email}} <{{email}}>`__.
{% endif %}
{% else %}

:Time and Place: :fas:`clock` {{ lecture_location }}, {{lecture_time }}.
:Exercise code: :fab:`gitlab` `lab.compute.dtu.dk <{{students_git}}>`_
{% if not CE %}:Course descriptions: :fas:`folder` `02002 <https://kurser.dtu.dk/course/2023-2024/02002>`__ and `02003 <https://kurser.dtu.dk/course/2023-2024/02003>`__{% endif %}
:Online help: :fab:`product-hunt` `piazza.com <{{piazza}}>`_
{% if not CE %}:Discord:  :fab:`discord` `Discord channel <{{ discord }}>`_ (`invitation link <{{ discord_invite }}>`_)
:Python support:  :fas:`circle-info` `pythonsupport.dtu.dk <https://pythonsupport.dtu.dk/>`__
:Contact: Administrative inquiries, such as exemptions, extensions or special circumstances, should be send to  :fas:`envelope` `programmering@compute.dtu.dk <programmering@compute.dtu.dk>`_.
{% endif %}
{% endif %}


Course schedule
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The schedule and reading can be found below. Click on the titles to read the exercise and project descriptions.

.. raw:: html

    <div class="full-width small">


.. list-table::
    :header-rows: 1
    :widths: auto

    * - #
      - Date
      - Title
      - Reading
      - Homework
      - Exercise
      - Slides
    * -
      - {{ ex0['latex_abbrev_year'] }}
      - :ref:`{{ ex0['title'] }} <ex0>`
      - Chapter 1-3 :fas:`star`, :cite:`herlau`
      -
      - :asset:`[PDF] <02465ex0_Python.pdf>`
      -
    {% for l in lectures %}{% if l.number < 14 %}
    {% for r in reports.values() %}{% if r.handin.lecture == l.number and r.id > 0 %}
    * -
      - **{{r.handin.latex_abbrev_year }}**
      - :fas:`code` :ref:`project{{r.id}}`
      -
      -
      -
      -
    {% endif %}{% endfor %}
    * - {{l['number'] }}
      - {{l.latex_abbrev_year}}
      - | :ref:`{{l['title'] }} <ex{{l['number']}}>`
        {% if 1 > 2 %}| {% for r in reports.values() %}{% if r.handin.lecture == l.number %}(:ref:`project{{r.id}}`, **{{r.handin.latex_abbrev}}, 23:59**) {% endif %}{% endfor %} {% endif %}
      - {{l.reading_rst }}
      - :fas:`hand-point-right` {{ l['homework_problems'] }}
      - :asset:`[PDF] <02465ex{{ l['number'] }}_Python.pdf>`
      - {% if ( l['date'].now() - l['date'] ).days >= l['show_slides_after'] %} :asset:`[1x] <Lecture_{{l['number']}}-handout.pdf>` :asset:`[6x] <Lecture_{{l['number']}}-6up-handout.pdf>` {% else %} TBA {% endif %}
    {% endif %}{% endfor %}

.. raw:: html

    </div>

The reading material is available here:

:[Her24]: :asset:`02465_Notes.pdf`
:[SB18]: `Introduction to Reinforcement Learning (2020) <https://www2.imm.dtu.dk/courses/02465/pensum/sutton2018.pdf>`_ (Authors `homepage <http://incompleteideas.net/book/RLbook2020.pdf>`_)

You can find the exam QA slides :ref:`here <exam>`. Details about the exam QA session will be announced on DTU Learn.

.. note::

   - :fas:`star` Chapters 1--3 is background information about python and are therefore not part of the main course content (pensum). Knowledge of python is required for the exams.
   - :fas:`hand-point-right` The **Homework** column list those problems that will be discussed during class. They are also indicated by a :fas:`hand-point-right` in the margin of the PDF file. I encourage you to prepare them at home and present your solution during the exercise session.

{% if not CE %}

Exercise sessions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. hint::
   :class: margin

   I will upload solutions to some of the python problems on `gitlab <https://gitlab.compute.dtu.dk/02465material/02465students>`__.

The teaching assistants will be available Fridays 10:00--12:00 after the lecture.

..  container:: small

   {{ ".. flat-table" }}::
         :header-rows: 1
         :widths: auto

         * - Location
           - Instructor
           - Email
         {% for room in rooms %}{% set outer_loop = loop %}{% for i in room['instructors'] %}
         {% if loop.index == 1 %}{{ '*' if True else ' '}} -  {{":rspan:"}}`{{ (room['instructors'] | length - 1) }}` {{room['location'] }} {% endif %}
         {{ '*' if loop.index > 1 else ' '}} - {{i['name']}}
           - {{i['email']}}
         {% endfor %}{% endfor %}

For the exercises, you are encouraged to prepare the **homework** problems at home (see syllabus above), and present your solution during the exercise session.

{% endif %}

Additional reading material
---------------------------------------------------------------------------------------
The material below is referenced in the course but it is not part of the course syllabus.

{{ ".. flat-table" }}::
   :header-rows: 1
   :widths: auto

   * - Reference name
     - Download
   * - (:cite:t:`tassa`)
     - :asset:`tassa2012.pdf`
   * - (:cite:t:`kelly`)
     - :asset:`kelly2017.pdf`


Contents
==================================================================================

.. toctree::
   :maxdepth: 1

   information/index
   models/index
   exercises/index
   projects/index
   exam


Indices and tables
==================================================================================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`


Bibliography
==================================================================================

.. bibliography::

This page was last updated at:

.. runblock:: pycon

    >>> from datetime import datetime
    >>> print("Document updated at:", datetime.now().strftime("%d/%m/%Y %H:%M:%S"))