Visualizing memorization in RNNs

Inspecting gradient magnitudes in context can be a powerful tool to see when recurrent units use short-term or long-term contextual understanding.

Memorization in Recurrent Neural Networks (RNNs) continues to pose a challenge
in many applications. We’d like RNNs to be able to store information over many
timesteps and retrieve it when it becomes relevant — but vanilla RNNs often struggle to do this.

Several network architectures have been proposed to tackle aspects of this problem, such
as Long-Short-Term Memory (LSTM)
units and Gated Recurrent Units (GRU).
However, the practical problem of memorization still poses a challenge.
As such, developing new recurrent units that are better at memorization
continues to be an active field of research.

To compare a recurrent unit against its alternatives, both past and recent
papers, such as the Nested LSTM paper by Monzi et al.
, heavily rely on quantitative
comparisons. These comparisons often measure accuracy or
cross entropy loss on standard problems such as Penn Treebank, Chinese
Poetry Generation, or text8, where the task is to predict the
next character given existing input.

While quantitative comparisons are useful, they only provide partial
insight into the how a recurrent unit memorizes. A model can, for example,
achieve high accuracy and cross entropy loss by just providing highly accurate
predictions in cases that only require short-term memorization, while
being inaccurate at predictions that require long-term
memorization.
For example, when autocompleting words in a sentence, a model with only short-term understanding could still exhibit high accuracy completing the ends of words once most of the characters are present.
However, without longer term contextual understanding it won’t be able to predict words when only a few characters are known.

This article presents a qualitative visualization method for comparing
recurrent units with regards to memorization and contextual understanding.
The method is applied to the three recurrent units mentioned above: Nested LSTMs, LSTMs, and GRUs.

Recurrent Units

The networks that will be analyzed all use a simple RNN structure:

Output for layer at time .

Recurrent unit of choice.

In theory, the time dependency allows it in each iteration to know
about every part of the sequence that came before. However, this time
dependency typically causes a vanishing gradient problem that results in
long-term dependencies being ignored during training
.

Vanishing Gradient: where the contribution from the
earlier steps becomes insignificant in the gradient for the vanilla RNN
unit.

Several solutions to the vanishing gradient problem have been proposed over
the years. The most popular are the aforementioned LSTM and GRU units, but this
is still an area of active research. Both LSTM and GRU are well known
and
thoroughly explained in literature. Recently, Nested LSTMs have also been proposed
 — an explanation of Nested LSTMs
can be found in the appendix.

Recurrent Unit, Nested LSTM: makes the cell update depend on another
LSTM unit, supposedly this allows more long-term memory compared to
stacking LSTM layers.

Recurrent Unit, LSTM: allows for long-term
memorization by gateing its update, thereby solving the vanishing gradient
problem.

Recurrent Unit, GRU: solves the vanishing gradient
problem without depending on an internal memory state.

It is not entirely clear why one recurrent unit performs better than another
in some applications, while in other applications it is another type of
recurrent unit that performs better. Theoretically they all solve the vanishing
gradient problem, but in practice their performance is highly application
dependent.

Understanding why these differences occur is likely an opaque and
challenging problem. The purpose of this article is to demonstrate a
visualization technique that can better highlight what these differences
are. Hopefully, such an understanding can lead to a deeper understanding.

Comparing Recurrent Units

Comparing different Recurrent Units is often more involved than simply comparing the accuracy or cross entropy
loss. Differences in these high-level quantitative measures
can have many explanations and may only be because of some small improvement
in predictions that only requires short-term contextual understanding,
while it is often the long-term contextual understanding that is of interest.

A problem for qualitative analysis

Therefore a good problem for qualitatively analyzing contextual
understanding should have a human-interpretable output and depend both on
long-term and short-term contextual understanding. The typical problems
that are often used, such as Penn Treebank, Chinese Poetry Generation, or
text8 generation do not have outputs that are easy to reason about, as they
require an extensive understanding of either grammar, Chinese poetry, or
only output a single letter.

To this end, this article studies the autocomplete problem. Each character is mapped
to a target that represents the entire word. The space leading up to the word should also map to that target.
This prediction based on the space character is in particular useful for showing contextual understanding.

The autocomplete problem is quite similar to the text8 generation
problem: the only difference is that instead of predicting the next letter,
the model predicts an entire word. This makes the output much more
interpretable. Finally, because of its close relation to text8 generation,
existing literature on text8 generation is relevant and comparable,
in the sense that models that work well on text8 generation should work
well on the autocomplete problem.

User types input sequence.
Recurrent neural network processes the sequence.
The output for the last character is used.
The most likely suggestions are extracted.

Autocomplete: An application that has a humanly
interpretable output, while depending on both short and long-term
contextual understanding. In this case, the network uses past information
and understands the next word should be a country.

The output in this figure was produced by the GRU model;
all model setups are described in the appendix.

Try removing the last letters to see
that the network continues to give meaningful suggestions.

(reset).

The autocomplete dataset is constructed from the full
text8 dataset. The
recurrent neural networks used to solve the problem have two layers, each
with 600 units. There are three models, using GRU, LSTM, and Nested LSTM.
See the appendix for more details.

Connectivity in the Autocomplete Problem

In the recently published Nested LSTM paper
, they qualitatively compared their
Nested LSTM unit to other recurrent units, to show how it memorizes in
comparison, by visualizing individual cell activations.

This visualization was inspired by Karpathy et al.
where they identify cells
that capture a specific feature. To identify a specific
feature, this visualization approach works well. However, it is not a useful
argument for memorization in general as the output is entirely dependent
on what feature the specific cell captures.

Instead, to get a better idea of how well each model memorizes and uses
memory for contextual understanding, the connectivity between the desired
output and the input is analyzed. This is calculated as:

Input time index.

Output time index.

Magnitude of the gradient, between the logits for the desired output and the input
.

Exploring the connectivity gives a surprising amount of insight into the
different models’ ability for long-term contextual understanding. Try and
interact with the figure below yourself to see what information the
different models use for their predictions.

Connectivity: the connection strength between
the target for the selected character and the input characters is highlighted in green
(reset).
Hover over or tap the text to change the selected character.

Let’s highlight three specific situations:

1

Observe how the models predict the word “learning” with only the first two
characters as input. The Nested LSTM model barely uses past
information and thus only suggests common words starting with the letter “l”.

In contrast, the LSTM and GRU models both suggests the word “learning”.
The GRU model shows stronger connectivity with the word “advanced”,
and we see in the suggestions that it predicts a higher probability for “learning” than the LSTM model.

2

Examine how the models predict the word “grammar”.
This word appears twice; when it appears for the first time the models have very little context.
Thus, no model suggests “grammar” until it has
seen at least 4 characters.

When “grammar” appears for the second time, the models have more context.
The GRU model is able to predict the word “grammar” with only
1 character from the word itself. The LSTM and Nested LSTM again
need at least 4 characters.

3

Finally, let’s look at predicting the word “schools”

given only the first character. As in the other cases,
the GRU model seems better at using past information for
contextual understanding.

What makes this case noteworthy is how the LSTM model appears to
use words from almost the entire sentence as context. However,
its suggestions are far from correct and have little to do
with the previous words it seems to use in its prediction.
This suggests that the LSTM model in this setup is capable of
long-term memorization, but not long-term contextual understanding.

1

2

3

The colored number links above change the connectivity figure’s displayed timestep and explanation.

These observations show that the connectivity visualization is a powerful tool
for comparing models in terms of which previous inputs they use for contextual understanding.
However, it is only possible to compare models on the same dataset, and
on a specific example. As such, while these observations may show that
Nested LSTM is not very capable of long-term contextual understanding in this example;
these observations may not generalize to other datasets or hyperparameters.

Future work; quantitative metric

From the above observations it appears that short-term contextual understanding
often involves the word that is being predicted itself. That is, the models switch to
using previously seen letters from the word itself, as more letters become
available. In contrast, at the beginning of predicting a word, models — especially the
GRU network — use previously seen words as context for the prediction.

This observation suggests a quantitative metric: measure the accuracy given
how many letters from the word being predicted are already known.
It is not clear that this is best quantitative metric: it is highly problem dependent,
and it also doesn’t summarize the model to a single number, which one may wish for a more direct comparison.

Accuracy Graph: shows the accuracy
given a fixed number of characters in a word that the RNN has seen.
0 characters mean that the RNN has only seen the space leading up
to the word, including all the previous text which should provide context.
The different line styles, indicates if the correct word should appear
among the top 1, 2, or 3 suggestions.

These results suggest that the GRU model is better at long-term contextual
understanding, while the LSTM model is better at short-term contextual
understanding. These observations are valuable, as it justifies why the
overall accuracy of the GRU and LSTM models
are almost identical, while the connectivity visualization shows that
the GRU model is far better at long-term contextual understanding.

While more detailed quantitative metrics like this provides new insight,
qualitative analysis like the connectivity figure presented
in this article still has great value. As the connectivity visualization gives an
intuitive understanding of how the model works, which a quantitative metric
cannot. It also shows that a wrong prediction can still be considered a
useful prediction, such as a synonym or a contextually reasonable
prediction.

Conclusion

Looking at overall accuracy and cross entropy loss in itself is not that
interesting. Different models may prioritize either long-term or
short-term contextual understanding, while both models can have similar
accuracy and cross entropy.

A qualitative analysis, where one looks at how previous input is used in
the prediction is therefore also important when judging models. In this
case, the connectivity visualization together with the autocomplete
predictions, reveals that the GRU model is much more capable of long-term
contextual understanding, compared to LSTM and Nested LSTM. In the case of
LSTM, the difference is much higher than one would guess from just looking
at the overall accuracy and cross entropy loss alone. This observation is
not that interesting in itself as it is likely very dependent on the
hyperparameters, and the specific application.

Much more valuable is that this visualization method makes it possible
to intuitively understand how the models are different, to a much higher
degree than just looking at accuracy and cross entropy. For this application,
it is clear that the GRU model uses repeating words and semantic meaning
of past words to make its prediction, to a much higher degree than the LSTM
and Nested LSTM models. This is both a valuable insight when choosing the
final model, but also essential knowledge when developing better models
in the future.

source https://nhahangcarnaval.com/visualizing-memorization-in-rnns/