Pre-requisites#
Reinforcement learning and control theory are interdisciplinary fields which build on other subjects from mathematics and computer science.
The formal pre-requisites are listed in the course base. However, I personally feel that a list of courses as pre-requisites can be difficult evaluate, and perhaps more frustrating, difficult to put to use by a student who wish to refresh earlier topics.
For instance, the statistics course emphasize that you can apply appropriate statistical tests to a dataset – however, this course will apply exactly 0 statistical tests to 0 datasets, but instead we will use more “basic” concepts such as probabilities (which was part of the statistics course).
For that reason, I think it is helpful to list some examples of concrete skills from the courses that I hope that most who take this course will feel are familiar. This itself can lead to problems – what happens if you don’t know something.
It is perhaps helpful to distinguish between four different levels of knowing something:
- I what what it means and I can do it
I hope many (most) of the concrete tasks will go here, especially those that seem highschool-adjacent such as \(\frac{d}{dx} x\sin(x)\)
- I am supposed to know and do that but …
I think many will have something in this category. Everyone forgets things, especially the formulate for the normal probability density. Many things that go in here would be things I would expect you to look up anyway, and the important part is that you know this is information that you can find (and probably also where)
- I can only do that with a computer
This is a more dangerous category and sometimes it contains surprises – for instance solving a linear system like \(3x - 5 \geq 8x + 2\). If you have things in this category I would recommend that you use resources such as your previous courses, Khan Academy or AI to move them into the category above. There are two concrete reasons why this worry me: - If you can only solve something with a computer you don’t know whether it is supposed to be hard or easy, or if the result should be simple or not. I see many examples during exams where students correctly reduce a problem to a simple equation, and then obtain a really complex maple-result that refers to variables that are not in the problem due to mistyping something. - It is hard to build on top of maple-only skills.
- I have not seen that before
Obviously this is the dangerous category
asdfsadfsda
asdfsadfdf
asdfsafdfa
sadfasdf
Programming#
Many of the course exercises and about 1/3rd of the exam involves programming in python. To implement environments, and many of the algorithms, you will also need to work with concepts from object oriented programming 02002.
Linear algebra#
You should be familiar with vectors, matrices and other foundational concepts from linear algebra. for instance:
What is a linear system: \(A \mathbf{x} = \mathbf{b}\),
determinants \(\textrm{det}(\begin{bmatrix} 2 & 1 \\ -1 2 \end{bmatrix} )\)
Inverse and products of a matrix: \(A^{-1}\): or \(ABA^{-1}\)
Eigenvalues and Eigenvectors \(\lambda \mathbf{x} = A \mathbf{x}\).
Analysis#
We will use various tools from analysis during the control section. For instance:
Function For instance if \(f(x) = 3x \cos(x)\) then compute \(f(0)\)
Equations and inequalities, for instance solve these for \(x\) such as: \(3x - 5 \geq 8x + 2\) or \(2x^2 - 3 = 5x + 1\).
Derivatives \(\frac{d}{dx} x\sin(x)\)
Integrals \(\int_0^5 x^3 dx\) or \(\int \frac{1}{x}dx\).
Differential equations will play an important role in control theory. However, unlike most math courses, our focus will not be to solve the differential equations (which is the hard part!) but to use them as a tool to formulate the control problems – however, it is still a really good think if you feel you understand what an equation such as:
What is an ODE, for instance \(y'' - \cos(x) y = 3\)
Does \(y = f(x) = \cos(x)\) solve \(y'' + y' = -y\)?
What a system of linear differential equations is (see mat1)
Probability theory#
Note
Two important comments: The first is that we will not use statistical tests in the course, and secondly, that although [SB18] formulate many results using stochastic variables, I am aware this is a difficult subject for many and we will not rely on a particular in-depth understanding of this concept beyond what you know from e.g. 02450 or statistics.
You should have a good operational grasp of basic concepts from probability theory such as probabilities, densities, and mean values. For instance:
What is the normal distribution \(p(x) = \frac{1}{ \sqrt{2\pi\sigma^2} }e^{ -\frac{(x-\mu)^2}{2\sigma^2} }\)
The rules of probability. For instance \(p(x, y) = p(x) p(y | x)\)
Expectations, for instance \(\mathbb{E}[f(x)] = \int f(x) p(x) dx\) if \(x\) is a continuous variable with density \(p(x)\) or \(\mathbb{E}[f(x)] = \sum_{i=1}^n f(x_i) p(x_i) dx\) if :math:x_i is discrete and can take \(n\) values.
Understand densities. For instance, if \(x \in [0, 1]\) follows a density such as