2024-08-02
This is a series of quick reads on papers with the title prefix "paper:".
How I write this series: After encountering particularly interesting papers, I summarize parts of the content and explore related ideas beyond the paper. This is not a summary of the paper, nor does it guarantee that all information or reasoning is accurate.
UNSUPERVISED MACHINE TRANSLATION USING MONOLINGUAL CORPORA ONLY
Note: The authors did not provide code, but there is a related implementation on GitHub.
According to Chapter 14 of the Deep Learning Book, an auto-encoder is a neural network trained to copy inputs to outputs. It must have a hidden layer h that represents the input, thus consisting of two parts: an encoding function h = f(x) and a decoding function r = g(h).
A Denoising Auto-encoder is a common variant where the input is a data point with added noise, and it is trained to output the original data point. Research by Alain and Bengio (2013) shows that this training allows f and g to implicitly learn the structure of the data distribution $P_{data}$.
The task of machine translation is to build a translator from a source language to a target language. Typically, translation tasks are solved by collecting parallel corpora and using supervised learning.
Back-translation provides a way to utilize a target language (monolingual) corpus T: First, train a (reverse) translator from the target language to the source language, translate each sentence t in T to the source language s, and the resulting (s, t) pairs effectively form an artificial parallel corpus. Then use the (s, t) pairs as additional training data.
The effectiveness of this method is limited by the quality of the reverse translator.
Distribution matching is the attempt to find a function F such that the distribution of F(X) is similar to that of Y, given datasets X and Y. Distribution matching is a technique in unsupervised learning. It is generally considered particularly effective when X and Y are high-dimensional.
Distribution matching is the opposite of classification: Classification distinguishes data points of different categories, while distribution matching learns a mapping such that, after mapping, data points from different domains are indistinguishable.
The intuitive idea is to construct a shared hidden space for both the source language (l=src) and the target language (l=tgt). We must construct bidirectional mappings between the source (target) language and the hidden space: Mapping from the language to the hidden space is the encoder $x \rightarrow e(x, l)$; mapping from the hidden space to the language is the decoder $y \rightarrow d(y, l)$. There are two encoders and two decoders, one pair for each language involved. The resulting translator can be represented as $x \rightarrow d(e(x, src), tgt)$, i.e., encoding with the source language encoder and decoding with the target language decoder. Note that the resulting translator is naturally bidirectional. To train these encoders and decoders, the paper defines three training tasks:
How can the encoder/decoder learn the distribution structure of the language? Here, the training method of auto-encoders is adopted: The source language encoder and decoder form an auto-encoder: $ x \rightarrow d(e(x, src), src) $, training this auto-encoder (adding noise to input data points, requiring the auto-encoder's output to reconstruct the original data points). This is an unsupervised method for e and d to learn the input distribution. The same applies to the target language.
We also need e and d to truly translate. Directly training for this goal: Inspired by back-translation, we first use the translation system obtained in the previous step to translate sentences s in the source language into sentences t in the target language, (s, t) is equivalent to our artificially constructed parallel corpus. Since the translation system is very noisy during initial steps of training, this parallel corpus is also very noisy. To update parameters, we manually add noise to the target language sentence t to get c(t) (why?), then translate it according to $ c(t) \rightarrow d(e(c(t), tgt), src) $, requiring this translation to reconstruct s. This task is called cross-lingual reconstruction. This is essentially supervised learning using artificially constructed, highly noisy parallel corpora to ensure the system achieves the translation goal.
Finally, the paper uses adversarial training, requiring that the representation of the source language in the hidden space is indistinguishable from that of the target language in the hidden space, claiming that this is the condition for the encoder and decoder to work. This task is essentially distribution matching.
The second of the three training tasks mentioned above is related to the ultimate goal of translation and must be carried out, while the first and third serve as auxiliary roles (Note: Even for the second task, its training data highly depends on the translator from the previous iteration, which is a bootstrapping process. The unsupervised nature means that if the translation system is inconsistent with the real semantics at some point, the mistake will propagate to later iterations, without an explicit error correction mechanism). Experimentally, the results of removing each training task can be referenced in the Ablation Studies section of the article.
The above three training tasks correspond to three loss terms.
When using the back-translation method, a "previous step" translation system is needed, which is actually composed of the encoder and decoder from the previous step, forming $x \rightarrow d(e(x, src), tgt) $ and $ x \rightarrow d(e(x, tgt), src) $. However, at the very beginning of training, d and e are randomly initialized, and the output of these two translators is pure noise. We need a basic translation function containing information to "initialize" the entire system: this initial translation system is word-by-word translation.
For word-by-word translation, we need a bilingual correspondence of the vocabulary. According to this paper, this correspondence table can be generated without parallel corpora, based on the method that "the geometric structures of word vectors in different languages are similar," as shown in Figure 1 of this paper.
PCA_d2 PCA_d2
/|\ horse /|\ caballo
| cow dog | vaca perro
| pig | cerdo
| |
| |
| cat | gato
+------------------------>PCA_d1 +------------------------->PCA_d1
English Spanish
Regarding why the structures of word vectors in different languages are similar (and even why performing addition and subtraction on word vectors can correspond to semantics: the word vector closest to v(Rome) + v(France) - v(Paris) is likely to be v(Italy)), some papers explain it as "different languages are based on the real world." Although I am not satisfied with this explanation, it is probably difficult to provide a better reason to explain whether this phenomenon inevitably occurs.