Long Short-Term Memory Network

What is Long Short-Term Memory Network?

An LSTM (Long Short-Term Memory) network is a kind of recurrent neural network (RNN) that has the ability to learn order dependence in sequence prediction problems. In addition to the standard units, LSTM networks use special units. They also have memory cells that store information in memory for large durations.

Let’s say that you’re trying to classify the events occurring in different parts of a novel. As a human, you would use the previous events that occurred in the novel to understand the events that are currently happening. But a traditional neural network is not capable of doing that.

That’s where recurrent neural networks come into play. 

Recurrent neural networks have loops built into them that allow information to persist. These loops essentially let information travel from one step of the network to the next one.

But there is one issue with normal RNNs. Let’s go back to the example of a novel. If an early part of the novel it is mentioned that Jack is a powerlifter and much, much later there is a mention of Jack getting ready for a competition, a normal RNN would not be able to figure out what Jack would be competing in. 

If there is a large gap between the point where the RNN first finds information and the point when it needs that information, the RNN might not be able to connect that information. Theoretically, they should be able to do this, but it does not tend to happen in practical situations.

LSTM (Long Short-Term Memory) Networks are specifically designed to solve this problem (the long-term dependency problem). Instead of a single neural network layer, they have four neural network layers. They don’t have to struggle to remember information for long periods of time, it’s their default behavior.

A regular LSTM unit is made up of a cell, an input gate, an output gate as well as a forget gate. The cell remembers values over arbitrary time intervals, while the input, output, and forget gates manage the flow of information into and out of the cell. The gates are a way for you to optionally let information pass through. These gates are made up of a sigmoid neural net layer and a point-wise multiplication operation.

Is Long Short-Term Memory Network a deep learning model?

LSTM is an RNN architecture that is used in deep learning. In contrast with regular feedforward neural networks, long short-term memory uses feedback connections.

Why is Long Short-Term Memory Network better than standard Recurrent Neural Network?

LSTMs actually deliver on the promise made by RNNs. Standard RNNs cannot access as vast a range of contextual information as an LSTM network can. The influence of inputs on the hidden layer, and the network output, tends to either decay or blow up exponentially as it cycles through the network’s recurrent connections in an RNN.

LSTMs triumph over the challenges of vanishing gradients and exploding gradients. In addition to this, LSTMs also give you more control, which means you get higher-quality results.

In most instances where recurrent neural networks performed remarkable feats and achieved great results, the kind of recurrent neural network used was an LSTM network.

LSTMs do tend to be more effective than standard RNNs, but they also happen to be more complex and carry higher operating costs.

One of the biggest advantages of LSTMs over RNNs, hidden Markov models and other sequence learning methods is that LSTMs have relative insensitivity to the length of gaps between events in time series data.

What are the applications of Long Short-Term Memory Network?

LSTMs can handle many tasks that standard recurrent neural networks cannot. It has been proven to be extremely effective in performing speech recognition, language modeling, and machine translation tasks.

They have also been useful for protein secondary structure prediction, handwriting recognition and generation, and even for analyzing audio and video data.

LSTMs have also been used in rhythm learning, music composition, human action recognition, sign language translation, time series prediction & anomaly detection in network traffic or IDS (intrusion detection systems), object co-segmentation, and robot control.

They have even been applied in semantic parsing, traffic forecasting (in the short term), drug design, and airport passenger management.

LSTMs are very useful for classifying, processing, and making predictions based on time series data because it is possible that there will be lags of random durations between important events in a time series. They were initially created to handle the vanishing gradient problem that might be encountered when you train traditional recurrent neural networks. 

Which is better - Long Short-Term Memory Network or Gate Recurrent Unit?

Long short-term memory (LSTM) and gate recurrent unit (GRU) are both popular versions of RNNs that have long-term memory. In a study using LSTM as well as GRU on the same Yelp review dataset it was found that GRU was 29.29% faster at processing the dataset than LSTM was. 

In terms of performance, LSTM is better than GRU in all scenarios other than situations involving long text and short datasets (in which GRU will perform much better). The performance-cost ratio of GRU is higher than that of LSTM.

What are the variants on Long Short-Term Memory Network?

Classic LSTM

The architecture of a classic LSTM is characterised by a persistent linear cell state surrounded by non-linear layers that feed input and parse output from it.  It overcomes the vanishing gradient problem in an RNN unrolled in time by connecting all-time points using a persistent cell state.

Peephole LSTM

In a classic LSTM, the gating layers that determine what to add, forget, and take from the cell state does not consider the contents of the cell. LSTM peephole connections concatenate the cell connections to the gating layer inputs. This improves the ability to count and time distances between rare events.

Multiplicative LSTM

Researchers found that by training a big multiplicative LSTM (mLSTM) model on unsupervised text prediction, its capability increased and it could perform at a high level on a battery of NLP tasks with barely any fine-tuning.