Temporal difference learning (TD Learning)

What is temporal difference learning?

Temporal Difference Learning in reinforcement learning is an unsupervised learning technique that is very commonly used in it for the purpose of predicting the total reward expected over the future. They can, however, be used to predict other quantities as well. It is essentially a way to learn how to predict a quantity that is dependent on the future values of a given signal. It is a method that is used to compute the long-term utility of a pattern of behaviour from a series of intermediate rewards.

Essentially, Temporal Difference Learning (TD Learning) focuses on predicting a variable's future value in a sequence of states. Temporal difference learning was a major breakthrough in solving the problem of reward prediction. You could say that it employs a mathematical trick that allows it to replace complicated reasoning with a simple learning procedure that can be used to generate the very same results.

The trick is that rather than attempting to calculate the total future reward, temporal difference learning just attempts to predict the combination of immediate reward and its own reward prediction at the next moment in time. Now when the next moment comes and brings fresh information with it, the new prediction is compared with the expected prediction. If these two predictions are different from each other, the Temporal Difference Learning algorithm will calculate how different the predictions are from each other and make use of this temporal difference to adjust the old prediction toward the new prediction. 

‍

‍

The temporal difference algorithm always aims to bring the expected prediction and the new prediction together, thus matching expectations with reality and gradually increasing the accuracy of the entire chain of prediction.

Temporal Difference Learning aims to predict a combination of the immediate reward and its own reward prediction at the next moment in time. 

In TD Learning, the training signal for a prediction is a future prediction. This method is a combination of the Monte Carlo (MC) method and the Dynamic Programming (DP) method. Monte Carlo methods adjust their estimates only after the final outcome is known, but temporal difference methods tend to adjust predictions to match later, more accurate, predictions for the future, much before the final outcome is clear and know. This is essentially a type of bootstrapping.

Temporal difference learning in machine learning got its name from the way it uses changes, or differences, in predictions over successive time steps for the purpose of driving the learning process. 

The prediction at any particular time step gets updated to bring it nearer to the prediction of the same quantity at the next time step. 

What are the parameters used in temporal difference learning?

• Alpha (α): learning rate
  It shows how much our estimates should be adjusted, based on the error. This rate varies between 0 and 1.
• Gamma (γ): the discount rate
  This indicates how much future rewards are valued. A larger discount rate signifies that future rewards are valued to a greater extent. The discount rate also varies between 0 and 1.
• e: the ratio reflective of exploration vs. exploitation.
  This involves exploring new options with probability e and staying at the current max with probability 1-e. A larger e signifies that more exploration is carried out during training

How is temporal difference learning used in neuroscience?

Around the late 1980s and the early 1990s, neuroscientists were trying to understand the manner in which dopamine neurons behave. These dopamine neurons are clustered in the mid-brain, but they send projections to several areas of the brain, potentially even broadcasting some globally relevant messages. It was obvious that the firing of these neurons were related to rewards in some way, but their responses were also dependent on sensory input and they changed as the animals gained more experience in a particular task.

Luckily, some researchers had a good idea about the recent developments in neuroscience as well as artificial intelligence. They noticed that responses in some dopamine neurons represented reward prediction errors. Their firing signified the points when the animal received greater or lesser rewards than it was trained to expect.

The firing rate of the dopamine cells did not increase when the animal received the predicted reward, but the firing rate for the dopamine cells fell below the normal activation levels when the reward was less than that which was expected.

This very closely mimics the way in which the error function in temporal difference in reinforcement learning is used for.

These researchers saw that and then proposed that the brain makes use of a temporal difference algorithm - a reward prediction error gets calculated, it is then broadcast to the brain through the dopamine signal and employed to drive learning.

After that, the reward prediction error theory has been widely tested and validated in thousands of experiments, and has since turned into one of the most successful quantitative theories in neuroscience.

The relationship between the temporal difference model and potential neurological function has generated research that attempts to use temporal difference to explain several aspects of behavioural research. Temporal difference learning in machine learning has also been utilized to study and understand conditions like schizophrenia or the consequences of pharmacological manipulations of dopamine on learning.

‍

‍

What is the benefit of temporal difference learning?

The advantages of temporal difference learning in machine learning are:

• TD learning methods are able to learn in each step, online or offline.
• These methods are capable of learning from incomplete sequences, which means that they can also be used in continuous problems.
• Temporal difference learning can function in non-terminating environments.
• TD Learning has less variance than the Monte Carlo method, because it depends on one random action, transition, reward.
• It tends to be more efficient than the Monte Carlo method.
• Temporal Difference Learning exploits the Markov property, which makes it more effective in Markov environments.

What are the disadvantages of temporal difference learning?

There are two main disadvantages:

• It has greater sensitivity towards the initial value.
• It is a biased estimation.

‍

What is the temporal difference error?

TD error arises in various forms throughout reinforcement learning and δt = rt+1 + γV(st+1) − V(st) value is commonly called the TD Error. Here the TD error is the difference between the current estimate for 𝑉𝑡, the discounted value estimate of 𝑉𝑡+1, and the actual reward gained from transitioning between 𝑠𝑡 and 𝑠𝑡+1. The TD error at each time is the error in the calculation made at that time. Because the TD error at step t relies on the next state and next reward, it is not available until step t + 1. When we update the value function with the TD error, it is called a backup. The TD error is related to the Bellman equation.

‍

What is the difference between Q-learning & Temporal Difference Learning?

Temporal Difference Learning in machine learning is a method to learn how to predict a quantity that depends on future values of a given signal. It can also be used to learn both the V-function and the Q-function, whereas Q-learning is a specific TD algorithm that is used to learn the Q-function. If you have only the V-function you can still derive the Q-function by repeating all the possible next states and choosing the action that leads you to the state with the highest V-value of the signal. 

In the model-free RL concept, you don't learn the state-transition function (the model) and you can depend only on samples. However, you might be interested also in learning it because you cannot collect many samples and want to generate some virtual ones. In this case, we talk about model-based RL. Model-based RL is quite common in robotics and machine learning, where you cannot perform many real simulations or the robot will break and that is the difference in TD learning vs Q learning. 

‍

What are the different algorithms in temporal difference learning? 

There are predominantly three different categories of TD algorithms which are as follows:

1. TD(1) Algorithm

2. TD(0) Algorithm

3. TD(λ) Algorithm