Publications

2024

1. NAACL

   When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale

   Baziotis, Christos, Zhang, Biao, Birch, Alexandra, and Haddow, Barry

   In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2024

   Abstract PDF Blog

   Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.

2023

1. EACL

   Automatic Evaluation and Analysis of Idioms in Neural Machine Translation

   Baziotis, Christos, Mathur, Prashant, and Hasler, Eva

   In Proceedings of the European Chapter of the Association for Computational Linguistics 2023

   Abstract PDF Blog

   A major open problem in neural machine translation (NMT) is the translation of idiomatic expressions, such as “under the weather”. The meaning of these expressions is not composed by the meaning of their constituent words, and NMT models tend to translate them literally (i.e., word-by-word), which leads to confusing and nonsensical translations. Research on idioms in NMT is limited and obstructed by the absence of automatic methods for quantifying these errors. In this work, first, we propose a novel metric for automatically measuring the frequency of literal translation errors without human involvement. Equipped with this metric, we present controlled translation experiments with models trained in different conditions (with/without the test-set idioms) and across a wide range of (global and targeted) metrics and test sets. We explore the role of monolingual pretraining and find that it yields substantial targeted improvements, even without observing any translation examples of the test-set idioms. In our analysis, we probe the role of idiom context. We find that the randomly initialized models are more local or “myopic” as they are relatively unaffected by variations of the idiom context, unlike the pretrained ones.

2022

1. EMNLP

   Multilingual Machine Translation with Hyper-Adapters

   Baziotis, Christos, Artetxe, Mikel, Cross, James, and Bhosale, Shruti

   In Proceedings of the Conference on Empirical Methods in Natural Language Processing 2022

   Abstract PDF Blog

   Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters. However, adapters of related languages are unable to transfer information, and their total number of parameters becomes prohibitively expensive as the number of languages grows. In this work, we overcome these drawbacks using hyper-adapters – hyper-networks that generate adapters from language and layer embeddings. While past work had poor results when scaling hyper-networks, we propose a rescaling fix that significantly improves convergence and enables training larger hyper-networks. We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters. When using the same number of parameters and FLOPS, our approach consistently outperforms regular adapters. Also, hyper-adapters converge faster than alternative approaches and scale better than regular dense networks. Our analysis shows that hyper-adapters learn to encode language relatedness, enabling positive transfer across languages.

2021

1. Findings of ACL

   Exploration of Unsupervised Pretraining Objectives for Machine Translation

   Baziotis, Christos, Titov, Ivan, Birch, Alexandra, and Haddow, Barry

   In Findings of the Association for Computational Linguistics 2021

   Abstract PDF Blog Code

   Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English↔German, English↔Nepali and English↔Sinhala monolingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretraining objective leads to surprisingly small differences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models using a series of probes and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities.

2020

1. EMNLP

   Language Model Prior for Low-Resource Neural Machine Translation

   Baziotis, Christos, Haddow, Barry, and Birch, Alexandra

   In Proceedings of the Conference on Empirical Methods in Natural Language Processing 2020

   Abstract PDF Code Slides Website

   The scarcity of large parallel corpora is an important obstacle for neural machine translation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM “disagrees” with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the target language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. We present an analysis of the effects that different methods have on the distributions of the TM. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.

2019

1. NAACL

   SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

   Baziotis, Christos, Androutsopoulos, Ion, Konstas, Ioannis, and Potamianos, Alexandros

   In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2019

   Abstract HTML PDF Code Slides

   Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder (SEQ\^3), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets.

2. ACL

   Attention-based Conditioning Methods for External Knowledge Integration

   Margatina, Katerina, Baziotis, Christos, and Potamianos, Alexandros

   In Proceedings of the Annual Meeting of the Association for Computational Linguistics 2019

   Abstract HTML PDF Code

   In this paper, we present a novel approach for incorporating external knowledge in Recurrent Neural Networks (RNNs). We propose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution, enforces the contribution of the most salient words for the task at hand. We introduce three methods, namely attentional concatenation, feature-based gating and affine transformation. Experiments on six benchmark datasets show the effectiveness of our methods. Attentional feature-based gating yields consistent performance improvement across tasks. Our approach is implemented as a simple add-on module for RNN-based models with minimal computational overhead and can be adapted to any deep neural architecture.

3. NAACL

   An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

   Chronopoulou, Alexandra, Baziotis, Christos, and Potamianos, Alexandros

   In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2019

   Abstract HTML PDF Code

   A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

2018

1. Interspeech

   Integrating Recurrence Dynamics for Speech Emotion Recognition

   Tzinis, Efthymios, Paraskevopoulos, Georgios, Baziotis, Christos, and Potamianos, Alexandros

   In Proceedings of Interspeech 2018

   HTML PDF Code
2. SemEval

   1st in Task1E

   

   NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning

   Baziotis, Christos, Nikolaos, Athanasiou, Chronopoulou, Alexandra, Kolovou, Athanasia, Paraskevopoulos, Georgios, Ellinas, Nikolaos, Narayanan, Shrikanth, and Potamianos, Alexandros

   In Proceedings of The International Workshop on Semantic Evaluation 2018

   Abstract HTML PDF Code

   In this paper we present deep-learning models that submitted to the SemEval-2018 Task 1 competition: “Affect in Tweets”. We participated in all subtasks for English tweets. We propose a Bi-LSTM architecture equipped with a multi-layer self attention mechanism. The attention mechanism improves the model performance and allows us to identify salient words in tweets, as well as gain insight into the models making them more interpretable. Our model utilizes a set of word2vec word embeddings trained on a large collection of 550 million Twitter messages, augmented by a set of word affective features. Due to the limited amount of task-specific training data, we opted for a transfer learning approach by pretraining the Bi-LSTMs on the dataset of Semeval 2017, Task 4A. The proposed approach ranked 1st in Subtask E “Multi-Label Emotion Classification”, 2nd in Subtask A “Emotion Intensity Regression” and achieved competitive results in other subtasks.

3. SemEval

   2nd place

   

   NTUA-SLP at SemEval-2018 Task 2: Predicting Emojis using RNNs with Context-aware Attention

   Baziotis, Christos, Nikolaos, Athanasiou, Kolovou, Athanasia, Paraskevopoulos, Georgios, Ellinas, Nikolaos, and Potamianos, Alexandros

   In Proceedings of The International Workshop on Semantic Evaluation 2018

   Abstract HTML PDF Code

   In this paper we present a deep-learning model that competed at SemEval-2018 Task 2 “Multilingual Emoji Prediction”. We participated in subtask A, in which we are called to predict the most likely associated emoji in English tweets. The proposed architecture relies on a Long Short-Term Memory network, augmented with an attention mechanism, that conditions the weight of each word, on a “context vector” which is taken as the aggregation of a tweet’s meaning. Moreover, we initialize the embedding layer of our model, with word2vec word embeddings, pretrained on a dataset of 550 million English tweets. Finally, our model does not rely on hand-crafted features or lexicons and is trained end-to-end with back-propagation. We ranked 2nd out of 48 teams.

4. SemEval

   2nd place

   

   NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs

   Baziotis, Christos, Nikolaos, Athanasiou, Papalampidi, Pinelopi, Kolovou, Athanasia, Paraskevopoulos, Georgios, Ellinas, Nikolaos, and Potamianos, Alexandros

   In Proceedings of The International Workshop on Semantic Evaluation 2018

   Abstract HTML PDF Code

   In this paper we present two deep-learning systems that competed at SemEval-2018 Task 3 “Irony detection in English tweets”. We design and ensemble two independent models, based on recurrent neural networks (Bi-LSTM), which operate at the word and character level, in order to capture both the semantic and syntactic information in tweets. Our models are augmented with a self-attention mechanism, in order to identify the most informative words. The embedding layer of our word-level model is initialized with word2vec word embeddings, pretrained on a collection of 550 million English tweets. We did not utilize any handcrafted features, lexicons or external datasets as prior information and our models are trained end-to-end using back propagation on constrained data. Furthermore, we provide visualizations of tweets with annotations for the salient tokens of the attention layer that can help to interpret the inner workings of the proposed models. We ranked 2nd out of 42 teams in Subtask A and 2nd out of 31 teams in Subtask B. However, post-task-completion enhancements of our models achieve state-of-the-art results ranking 1st for both subtasks.

5. WASSA-ACL

   NTUA-SLP at IEST 2018: Ensemble of Neural Transfer Methods for Implicit Emotion Classification

   Chronopoulou, Alexandra, Margatina, Aikaterini, Baziotis, Christos, and Potamianos, Alexandros

   In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis 2018

   Abstract HTML PDF Code

   In this paper we present our approach to tackle the Implicit Emotion Shared Task (IEST) organized as part of WASSA 2018 at EMNLP 2018. Given a tweet, from which a certain word has been removed, we are asked to predict the emotion of the missing word. In this work, we experiment with neural Transfer Learning (TL) methods. Our models are based on LSTM networks, augmented with a self-attention mechanism. We use the weights of various pretrained models, for initializing specific layers of our networks. We leverage a big collection of unlabeled Twitter messages, for pretraining word2vec word embeddings and a set of diverse language models. Moreover, we utilize a sentiment analysis dataset for pretraining a model, which encodes emotion related information. The submitted model consists of an ensemble of the aforementioned TL models. Our team ranked 3rd out of 30 participants, achieving an F1 score of 0.703.

2017

1. SemEval

   1st in Task4A, 2nd in Task4B,C,D

   

   DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis

   Baziotis, Christos, Pelekis, Nikos, and Doulkeridis, Christos

   In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) 2017

   Abstract HTML PDF Code

   In this paper we present two deep-learning systems that competed at SemEval-2017 Task 4 “Sentiment Analysis in Twitter”. We participated in all subtasks for English tweets, involving message-level and topic-based sentiment polarity classification and quantification. We use Long Short-Term Memory (LSTM) networks augmented with two kinds of attention mechanisms, on top of word embeddings pre-trained on a big collection of Twitter messages. Also, we present a text processing tool suitable for social network messages, which performs tokenization, word normalization, segmentation and spell correction. Moreover, our approach uses no hand-crafted features or sentiment lexicons. We ranked 1st (tie) in Subtask A, and achieved very competitive results in the rest of the Subtasks. Both the word embeddings and our text processing tool are available to the research community.

2. SemEval

   2nd in Task6A

   

   DataStories at SemEval-2017 Task 6: Siamese LSTM with Attention for Humorous Text Comparison

   Baziotis, Christos, Pelekis, Nikos, and Doulkeridis, Christos

   In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) 2017

   Abstract HTML PDF Code

   In this paper we present a deep-learning system that competed at SemEval-2017 Task 6 ”#HashtagWars: Learning a Sense of Humor”. We participated in Subtask A, in which the goal was, given two Twitter messages, to identify which one is funnier. We propose a Siamese architecture with bidirectional Long Short-Term Memory (LSTM) networks, augmented with an attention mechanism. Our system works on the token-level, leveraging word embeddings trained on a big collection of unlabeled Twitter messages. We ranked 2nd in 7 teams. A post-completion improvement of our model, achieves state-of-the-art results on #HashtagWars dataset.