Mark Stevenson


2023

pdf bib
Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review
Reem Bin-Hezam | Mark Stevenson
Findings of the Association for Computational Linguistics: EMNLP 2023

Technology Assisted Review (TAR) stopping rules aim to reduce the cost of manually assessing documents for relevance by minimising the number of documents that need to be examined to ensure a desired level of recall. This paper extends an effective stopping rule using information derived from a text classifier that can be trained without the need for any additional annotation. Experiments on multiple data sets (CLEF e-Health, TREC Total Recall, TREC Legal and RCV1) showed that the proposed approach consistently improves performance and outperforms several alternative methods.

2021

pdf bib
Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis
Xutan Peng | Guanyi Chen | Chenghua Lin | Mark Stevenson
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Knowledge Graph Embeddings (KGEs) have been intensively explored in recent years due to their promise for a wide range of applications. However, existing studies focus on improving the final model performance without acknowledging the computational cost of the proposed approaches, in terms of execution time and environmental impact. This paper proposes a simple yet effective KGE framework which can reduce the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches, while producing competitive performance. We highlight three technical innovations: full batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis for KGEs, and non-negative-sampling training. In addition, as the first KGE method whose entity embeddings also store full relation information, our trained models encode rich semantics and are highly interpretable. Comprehensive experiments and ablation studies involving 13 strong baselines and two standard datasets verify the effectiveness and efficiency of our algorithm.

pdf bib
Cross-Lingual Word Embedding Refinement by 1 Norm Optimisation
Xutan Peng | Chenghua Lin | Mark Stevenson
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Cross-Lingual Word Embeddings (CLWEs) encode words from two or more languages in a shared high-dimensional space in which vectors representing words with similar meaning (regardless of language) are closely located. Existing methods for building high-quality CLWEs learn mappings that minimise the ℓ2 norm loss function. However, this optimisation objective has been demonstrated to be sensitive to outliers. Based on the more robust Manhattan norm (aka. ℓ1 norm) goodness-of-fit criterion, this paper proposes a simple post-processing step to improve CLWEs. An advantage of this approach is that it is fully agnostic to the training process of the original CLWEs and can therefore be applied widely. Extensive experiments are performed involving ten diverse languages and embeddings trained on different corpora. Evaluation results based on bilingual lexicon induction and cross-lingual transfer for natural language inference tasks show that the ℓ1 refinement substantially outperforms four state-of-the-art baselines in both supervised and unsupervised settings. It is therefore recommended that this strategy be adopted as a standard for CLWE methods.

pdf bib
Identifying Automatically Generated Headlines using Transformers
Antonis Maronikolakis | Hinrich Schütze | Mark Stevenson
Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda

False information spread via the internet and social media influences public opinion and user activity, while generative models enable fake content to be generated faster and more cheaply than had previously been possible. In the not so distant future, identifying fake content generated by deep learning models will play a key role in protecting users from misinformation. To this end, a dataset containing human and computer-generated headlines was created and a user study indicated that humans were only able to identify the fake headlines in 47.8% of the cases. However, the most accurate automatic approach, transformers, achieved an overall accuracy of 85.7%, indicating that content generated from language models can be filtered out accurately.

2020

pdf bib
Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs
Haiyang Zhang | Alison Sneyd | Mark Stevenson
Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing

It has been shown that word embeddings can exhibit gender bias, and various methods have been proposed to quantify this. However, the extent to which the methods are capturing social stereotypes inherited from the data has been debated. Bias is a complex concept and there exist multiple ways to define it. Previous work has leveraged gender word pairs to measure bias and extract biased analogies. We show that the reliance on these gendered pairs has strong limitations: bias measures based off of them are not robust and cannot identify common types of real-world bias, whilst analogies utilising them are unsuitable indicators of bias. In particular, the well-known analogy “man is to computer-programmer as woman is to homemaker” is due to word similarity rather than bias. This has important implications for work on measuring bias in embeddings and related work debiasing embeddings.

pdf bib
ParaPat: The Multi-Million Sentences Parallel Corpus of Patents Abstracts
Felipe Soares | Mark Stevenson | Diego Bartolome | Anna Zaretskaya
Proceedings of the Twelfth Language Resources and Evaluation Conference

The Google Patents is one of the main important sources of patents information. A striking characteristic is that many of its abstracts are presented in more than one language, thus making it a potential source of parallel corpora. This article presents the development of a parallel corpus from the open access Google Patents dataset in 74 language pairs, comprising more than 68 million sentences and 800 million tokens. Sentences were automatically aligned using the Hunalign algorithm for the largest 22 language pairs, while the others were abstract (i.e. paragraph) aligned. We demonstrate the capabilities of our corpus by training Neural Machine Translation (NMT) models for the main 9 language pairs, with a total of 18 models. Our parallel corpus is freely available in TSV format and with a SQLite database, with complementary information regarding patent metadata.

2019

pdf bib
Modelling Stopping Criteria for Search Results using Poisson Processes
Alison Sneyd | Mark Stevenson
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

Text retrieval systems often return large sets of documents, particularly when applied to large collections. Stopping criteria can reduce the number of these documents that need to be manually evaluated for relevance by predicting when a suitable level of recall has been achieved. In this work, a novel method for determining a stopping criterion is proposed that models the rate at which relevant documents occur using a Poisson process. This method allows a user to specify both a minimum desired level of recall to achieve and a desired probability of having achieved it. We evaluate our method on a public dataset and compare it with previous techniques for determining stopping criteria.

pdf bib
Re-Ranking Words to Improve Interpretability of Automatically Generated Topics
Areej Alokaili | Nikolaos Aletras | Mark Stevenson
Proceedings of the 13th International Conference on Computational Semantics - Long Papers

Topics models, such as LDA, are widely used in Natural Language Processing. Making their output interpretable is an important area of research with applications to areas such as the enhancement of exploratory search interfaces and the development of interpretable machine learning models. Conventionally, topics are represented by their n most probable words, however, these representations are often difficult for humans to interpret. This paper explores the re-ranking of topic words to generate more interpretable topic representations. A range of approaches are compared and evaluated in two experiments. The first uses crowdworkers to associate topics represented by different word rankings with related documents. The second experiment is an automatic approach based on a document retrieval task applied on multiple domains. Results in both experiments demonstrate that re-ranking words improves topic interpretability and that the most effective re-ranking schemes were those which combine information about the importance of words both within topics and their relative frequency in the entire corpus. In addition, close correlation between the results of the two evaluation approaches suggests that the automatic method proposed here could be used to evaluate re-ranking methods without the need for human judgements.

2018

pdf bib
Topic or Style? Exploring the Most Useful Features for Authorship Attribution
Yunita Sari | Mark Stevenson | Andreas Vlachos
Proceedings of the 27th International Conference on Computational Linguistics

Approaches to authorship attribution, the task of identifying the author of a document, are based on analysis of individuals’ writing style and/or preferred topics. Although the problem has been widely explored, no previous studies have analysed the relationship between dataset characteristics and effectiveness of different types of features. This study carries out an analysis of four widely used datasets to explore how different types of features affect authorship attribution accuracy under varying conditions. The results of the analysis are applied to authorship attribution models based on both discrete and continuous representations. We apply the conclusions from our analysis to an extension of an existing approach to authorship attribution and outperform the prior state-of-the-art on two out of the four datasets used.

pdf bib
HiDE: a Tool for Unrestricted Literature Based Discovery
Judita Preiss | Mark Stevenson
Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations

As the quantity of publications increases daily, researchers are forced to narrow their attention to their own specialism and are therefore less likely to make new connections with other areas. Literature based discovery (LBD) supports the identification of such connections. A number of LBD tools are available, however, they often suffer from limitations such as constraining possible searches or not producing results in real-time. We introduce HiDE (Hidden Discovery Explorer), an online knowledge browsing tool which allows fast access to hidden knowledge generated from all abstracts in Medline. HiDE is fast enough to allow users to explore the full range of hidden connections generated by an LBD system. The tool employs a novel combination of two approaches to LBD: a graph-based approach which allows hidden knowledge to be generated on a large scale and an inference algorithm to identify the most promising (most likely to be non trivial) information. Available at https://skye.shef.ac.uk/kdisc

2017

pdf bib
Continuous N-gram Representations for Authorship Attribution
Yunita Sari | Andreas Vlachos | Mark Stevenson
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers

This paper presents work on using continuous representations for authorship attribution. In contrast to previous work, which uses discrete feature representations, our model learns continuous representations for n-gram features via a neural network jointly with the classification layer. Experimental results demonstrate that the proposed model outperforms the state-of-the-art on two datasets, while producing comparable results on the remaining two.

2016

pdf bib
User profiling with geo-located posts and demographic data
Adam Poulston | Mark Stevenson | Kalina Bontcheva
Proceedings of the First Workshop on NLP and Computational Social Science

2015

pdf bib
A Hybrid Distributional and Knowledge-based Model of Lexical Semantics
Nikolaos Aletras | Mark Stevenson
Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics

pdf bib
Improving distant supervision using inference learning
Roland Roller | Eneko Agirre | Aitor Soroa | Mark Stevenson
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)

pdf bib
Held-out versus Gold Standard: Comparison of Evaluation Strategies for Distantly Supervised Relation Extraction from Medline abstracts
Roland Roller | Mark Stevenson
Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis

pdf bib
Making the most of limited training data using distant supervision
Roland Roller | Mark Stevenson
Proceedings of BioNLP 15

pdf bib
Automatic Detection of Answers to Research Questions from Medline Abstracts
Abdulaziz Alamri | Mark Stevenson
Proceedings of BioNLP 15

2014

pdf bib
Measuring the Similarity between Automatically Generated Topics
Nikolaos Aletras | Mark Stevenson
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers

pdf bib
Applying UMLS for Distantly Supervised Relation Detection
Roland Roller | Mark Stevenson
Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi)

pdf bib
Labelling Topics using Unsupervised Graph-based Methods
Nikolaos Aletras | Mark Stevenson
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

2013

pdf bib
Representing Topics Using Images
Nikolaos Aletras | Mark Stevenson
Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

pdf bib
Unsupervised Domain Tuning to Improve Word Sense Disambiguation
Judita Preiss | Mark Stevenson
Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

pdf bib
DALE: A Word Sense Disambiguation System for Biomedical Documents Trained using Automatically Labeled Examples
Judita Preiss | Mark Stevenson
Proceedings of the 2013 NAACL HLT Demonstration Session

pdf bib
Evaluating Topic Coherence Using Distributional Semantics
Nikolaos Aletras | Mark Stevenson
Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers

pdf bib
Identification of Genia Events using Multiple Classifiers
Roland Roller | Mark Stevenson
Proceedings of the BioNLP Shared Task 2013 Workshop

pdf bib
Generating Paths through Cultural Heritage Collections
Samuel Fernando | Paula Goodale | Paul Clough | Mark Stevenson | Mark Hall | Eneko Agirre
Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities

pdf bib
Distinguishing Common and Proper Nouns
Judita Preiss | Mark Stevenson
Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity

pdf bib
UBC_UOS-TYPED: Regression for typed-similarity
Eneko Agirre | Nikolaos Aletras | Aitor Gonzalez-Agirre | German Rigau | Mark Stevenson
Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity

pdf bib
PATHS: A System for Accessing Cultural Heritage Collections
Eneko Agirre | Nikolaos Aletras | Paul Clough | Samuel Fernando | Paula Goodale | Mark Hall | Aitor Soroa | Mark Stevenson
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations

2012

pdf bib
Comparing Taxonomies for Organising Collections of Documents
Samuel Fernando | Mark Hall | Eneko Agirre | Aitor Soroa | Paul Clough | Mark Stevenson
Proceedings of COLING 2012

pdf bib
Computing Similarity between Cultural Heritage Items using Multimodal Features
Nikolaos Aletras | Mark Stevenson
Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities

pdf bib
Adapting Wikification to Cultural Heritage
Samuel Fernando | Mark Stevenson
Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities

pdf bib
Scaling up WSD with Automatically Generated Examples
Weiwei Cheng | Judita Preiss | Mark Stevenson
BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing

pdf bib
Mapping WordNet synsets to Wikipedia articles
Samuel Fernando | Mark Stevenson
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

Lexical knowledge bases (LKBs), such as WordNet, have been shown to be useful for a range of language processing tasks. Extending these resources is an expensive and time-consuming process. This paper describes an approach to address this problem by automatically generating a mapping from WordNet synsets to Wikipedia articles. A sample of synsets has been manually annotated with article matches for evaluation purposes. The automatic methods are shown to create mappings with precision of 87.8% and recall of 46.9%. These mappings can then be used as a basis for enriching WordNet with new relations based on Wikipedia links. The manual and automatically created data is available online.

pdf bib
Matching Cultural Heritage items to Wikipedia
Eneko Agirre | Ander Barrena | Oier Lopez de Lacalle | Aitor Soroa | Samuel Fernando | Mark Stevenson
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

Digitised Cultural Heritage (CH) items usually have short descriptions and lack rich contextual information. Wikipedia articles, on the contrary, include in-depth descriptions and links to related articles, which motivate the enrichment of CH items with information from Wikipedia. In this paper we explore the feasibility of finding matching articles in Wikipedia for a given Cultural Heritage item. We manually annotated a random sample of items from Europeana, and performed a qualitative and quantitative study of the issues and problems that arise, showing that each kind of CH item is different and needs a nuanced definition of what ``matching article'' means. In addition, we test a well-known wikification (aka entity linking) algorithm on the task. Our results indicate that a substantial number of items can be effectively linked to their corresponding Wikipedia article.

pdf bib
Detecting Text Reuse with Modified and Weighted N-grams
Rao Muhammad Adeel Nawab | Mark Stevenson | Paul Clough
*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)

pdf bib
University_Of_Sheffield: Two Approaches to Semantic Text Similarity
Sam Biggins | Shaabi Mohammed | Sam Oakley | Luke Stringer | Mark Stevenson | Judita Preiss
*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)

2011

pdf bib
Extracting Relations Within and Across Sentences
Kumutha Swampillai | Mark Stevenson
Proceedings of the International Conference Recent Advances in Natural Language Processing 2011

2010

pdf bib
Inter-sentential Relations in Information Extraction Corpora
Kumutha Swampillai | Mark Stevenson
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

In natural language relationships between entities can asserted within a single sentence or over many sentences in a document. Many information extraction systems are constrained to extracting binary relations that are asserted within a single sentence (single-sentence relations) and this limits the proportion of relations they can extract since those expressed across multiple sentences (inter-sentential relations) are not considered. The analysis in this paper focuses on finding the distribution of inter-sentential and single-sentence relations in two corpora used for the evaluation of Information Extraction systems: the MUC6 corpus and the ACE corpus from 2003. In order to carry out this analysis we had to manually mark up all the management succession relations described in the MUC6 corpus. It was found that inter-sentential relations constitute 28.5% and 9.4% of the total number of relations in MUC6 and ACE03 respectively. This places upper bounds on the recall of information extraction systems that do not consider relations that are asserted across multiple sentences (71.5% and 90.6% respectively).

pdf bib
Improving Summarization of Biomedical Documents Using Word Sense Disambiguation
Laura Plaza | Mark Stevenson | Alberto Díaz
Proceedings of the 2010 Workshop on Biomedical Natural Language Processing

pdf bib
The Effect of Ambiguity on the Automated Acquisition of WSD Examples
Mark Stevenson | Yikun Guo
Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics

pdf bib
IIITH: Domain Specific Word Sense Disambiguation
Siva Reddy | Abhilash Inumella | Diana McCarthy | Mark Stevenson
Proceedings of the 5th International Workshop on Semantic Evaluation

2009

pdf bib
Disambiguation of Biomedical Abbreviations
Mark Stevenson | Yikun Guo | Abdulaziz Alamri | Robert Gaizauskas
Proceedings of the BioNLP 2009 Workshop

2008

pdf bib
Knowledge Sources for Word Sense Disambiguation of Biomedical Text
Mark Stevenson | Yinkun Guo | Robert Gaizauskas | David Martinez
Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing

pdf bib
Acquiring Sense Tagged Examples using Relevance Feedback
Mark Stevenson | Yikun Guo | Robert Gaizauskas
Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)

2007

pdf bib
Learning Expressive Models for Word Sense Disambiguation
Lucia Specia | Mark Stevenson | Maria das Graças Volpe Nunes
Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics

pdf bib
A Task-based Comparison of Information Extraction Pattern Models
Mark Greenwood | Mark Stevenson
ACL 2007 Workshop on Deep Linguistic Processing

2006

pdf bib
Proceedings of the Workshop on Information Extraction Beyond The Document
Mary Elaine Califf | Mark A. Greenwood | Mark Stevenson | Roman Yangarber
Proceedings of the Workshop on Information Extraction Beyond The Document

pdf bib
Comparing Information Extraction Pattern Models
Mark Stevenson | Mark A. Greenwood
Proceedings of the Workshop on Information Extraction Beyond The Document

pdf bib
Improving Semi-supervised Acquisition of Relation Extraction Patterns
Mark A. Greenwood | Mark Stevenson
Proceedings of the Workshop on Information Extraction Beyond The Document

pdf bib
Multilingual versus Monolingual WSD
Lucia Specia | Maria das Graças Volpe Nunes | Mark Stevenson | Gabriela Castelo Branco Ribeiro
Proceedings of the Workshop on Making Sense of Sense: Bringing Psycholinguistics and Computational Linguistics Together

pdf bib
Translation Context Sensitive WSD
Lucia Specia | Maria das Graças Volpe Nunes | Mark Stevenson
Proceedings of the 11th Annual Conference of the European Association for Machine Translation

2005

pdf bib
A Semantic Approach to IE Pattern Induction
Mark Stevenson | Mark Greenwood
Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)

2004

pdf bib
EuroWordNet as a Resource for Cross-language Information Retrieval
Mark Stevenson | Paul Clough
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

pdf bib
Information Extraction from Single and Multiple Sentences
Mark Stevenson
COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics

2002

pdf bib
The Reuters Corpus Volume 1 -from Yesterday’s News to Tomorrow’s Language Resources
Tony Rose | Mark Stevenson | Miles Whitehead
Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02)

pdf bib
Augmenting Noun Taxonomies by Combining Lexical Similarity Metrics
Mark Stevenson
COLING 2002: The 19th International Conference on Computational Linguistics

2001

pdf bib
The Interaction of Knowledge Sources in Word Sense Disambiguation
Mark Stevenson | Yorick Wilks
Computational Linguistics, Volume 27, Number 3, September 2001

2000

pdf bib
Experiments on Sentence Boundary Detection
Mark Stevenson | Robert Gaizauskas
Sixth Applied Natural Language Processing Conference

pdf bib
Using Corpus-derived Name Lists for Named Entity Recognition
Mark Stevenson | Robert Gaizauskas
Sixth Applied Natural Language Processing Conference

1999

pdf bib
A Corpus-Based Approach to Deriving Lexical Mappings
Mark Stevenson
Ninth Conference of the European Chapter of the Association for Computational Linguistics

1998

pdf bib
Word Sense Disambiguation using Optimised Combinations of Knowledge Sources
Yorick Wilks | Mark Stevenson
36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2

pdf bib
Word Sense Disambiguation using Optimised Combinations of Knowledge Sources
Yorick Wilks | Mark Stevenson
COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics

pdf bib
Implementing a Sense Tagger in a General Architecture for Text Engineering
Hamish Cunningham | Mark Stevenson | Yorick Wilks
New Methods in Language Processing and Computational Natural Language Learning

1997

pdf bib
Sense Tagging: Semantic Tagging with a Lexicon
Yorick Wilks | Mark Stevenson
Tagging Text with Lexical Semantics: Why, What, and How?