Improving Natural Language Understanding by Reverse Mapping Bytepair Encoding

Improving Natural Language Understanding by Reverse Mapping Bytepair Encoding

We propose a method called reverse mapping bytepair encoding, which maps named-entity information and other word-level linguistic features back to subwords during the encoding procedure of bytepair encoding (BPE). We employ this method to the Generative Pre-trained Transformer (OpenAI GPT) by adding a weighted linear layer after the embedding layer.We also propose a new model architecture named as the multi-channel separate transformer to employ a training process without parameter-sharing. Evaluation on Stories Cloze, RTE, SciTail and SST-2 datasets demonstrates the effectiveness of our approach.

通过反向映射字节对编码提高对自然语言的理解

我们提出了一种称为反向映射字节对编码的方法，该方法在字节对编码（BPE）的编码过程中将命名实体信息和其他词级语言特征映射回子词。通过在嵌入层之后添加加权线性层，我们将这种方法应用于生成型预训练变压器（OpenAI GPT）。.. 我们还建议命名为多路独立的变压器采用训练过程中没有参数共享的新模式架构。对故事完形填空，RTE，SciTail和SST-2数据集的评估证明了我们方法的有效性。 （阅读更多）