How Multilingual is Multilingual BERT?

Abstract

In this paper, we show that Multilingual BERT (M-BERT), released by To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Identifiers

DOI

10.18653/v1/p19-1493

Citations and references