Johan Boye


2023

pdf bib
Quasi: a synthetic Question-Answering dataset in Swedish using GPT-3 and zero-shot learning
Dmytro Kalpakchi | Johan Boye
Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)

This paper describes the creation and evaluation of a synthetic dataset of Swedish multiple-choice questions (MCQs) for reading comprehension using GPT-3. Although GPT-3 is trained mostly on English data, with only 0.11% of Swedish texts in its training material, the model still managed to generate MCQs in Swedish. About 44% of the generated MCQs turned out to be of sufficient quality, i.e. they were grammatically correct and relevant, with exactly one answer alternative being correct and the others being plausible but wrong. We provide a detailed analysis of the errors and shortcomings of the rejected MCQs, as well an analysis of the level of difficulty of the accepted MCQs. In addition to giving insights into GPT-3, the synthetic dataset could be used for training and evaluation of special-purpose MCQ-generating models.

2022

pdf bib
Textinator: an Internationalized Tool for Annotation and Human Evaluation in Natural Language Processing and Generation
Dmytro Kalpakchi | Johan Boye
Proceedings of the Thirteenth Language Resources and Evaluation Conference

We release an internationalized annotation and human evaluation bundle, called Textinator, along with documentation and video tutorials. Textinator allows annotating data for a wide variety of NLP tasks, and its user interface is offered in multiple languages, lowering the entry threshold for domain experts. The latter is, in fact, quite a rare feature among the annotation tools, that allows controlling for possible unintended biases introduced due to hiring only English-speaking annotators. We illustrate the rarity of this feature by presenting a thorough systematic comparison of Textinator to previously published annotation tools along 9 different axes (with internationalization being one of them). To encourage researchers to design their human evaluation before starting to annotate data, Textinator offers an easy-to-use tool for human evaluations allowing importing surveys with potentially hundreds of evaluation items in one click. We finish by presenting several use cases of annotation and evaluation projects conducted using pre-release versions of Textinator. The presented use cases do not represent Textinator’s full annotation or evaluation capabilities, and interested readers are referred to the online documentation for more information.

2021

pdf bib
Minor changes make a difference: a case study on the consistency of UD-based dependency parsers
Dmytro Kalpakchi | Johan Boye
Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)

pdf bib
BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset
Dmytro Kalpakchi | Johan Boye
Proceedings of the 14th International Conference on Natural Language Generation

An important part when constructing multiple-choice questions (MCQs) for reading comprehension assessment are the distractors, the incorrect but preferably plausible answer options. In this paper, we present a new BERT-based method for automatically generating distractors using only a small-scale dataset. We also release a new such dataset of Swedish MCQs (used for training the model), and propose a methodology for assessing the generated distractors. Evaluation shows that from a student’s perspective, our method generated one or more plausible distractors for more than 50% of the MCQs in our test set. From a teacher’s perspective, about 50% of the generated distractors were deemed appropriate. We also do a thorough analysis of the results.

2020

pdf bib
UDon2: a library for manipulating Universal Dependencies trees
Dmytro Kalpakchi | Johan Boye
Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020)

UDon2 is an open-source library for manipulating dependency trees represented in the CoNLL-U format. The library is compatible with the Universal Dependencies. UDon2 is aimed at developers of downstream Natural Language Processing applications that require manipulating dependency trees on the sentence level (in addition to other available tools geared towards working with treebanks).

2019

pdf bib
SpaceRefNet: a neural approach to spatial reference resolution in a real city environment
Dmytro Kalpakchi | Johan Boye
Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue

Adding interactive capabilities to pedestrian wayfinding systems in the form of spoken dialogue will make them more natural to humans. Such an interactive wayfinding system needs to continuously understand and interpret pedestrian’s utterances referring to the spatial context. Achieving this requires the system to identify exophoric referring expressions in the utterances, and link these expressions to the geographic entities in the vicinity. This exophoric spatial reference resolution problem is difficult, as there are often several dozens of candidate referents. We present a neural network-based approach for identifying pedestrian’s references (using a network called RefNet) and resolving them to appropriate geographic objects (using a network called SpaceRefNet). Both methods show promising results beating the respective baselines and earlier reported results in the literature.

2016

pdf bib
SpaceRef: A corpus of street-level geographic descriptions
Jana Götze | Johan Boye
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)

This article describes SPACEREF, a corpus of street-level geographic descriptions. Pedestrians are walking a route in a (real) urban environment, describing their actions. Their position is automatically logged, their speech is manually transcribed, and their references to objects are manually annotated with respect to a crowdsourced geographic database. We describe how the data was collected and annotated, and how it has been used in the context of creating resources for an automatic pedestrian navigation system.

2015

pdf bib
Resolving Spatial References using Crowdsourced Geographical Data
Jana Götze | Johan Boye
Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015)

2014

pdf bib
Crowdsourcing Street-level Geographic Information Using a Spoken Dialogue System
Raveesh Meena | Johan Boye | Gabriel Skantze | Joakim Gustafson
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

2013

pdf bib
Deriving Salience Models from Human Route Directions
Jana Götze | Johan Boye
Proceedings of the IWCS 2013 Workshop on Computational Models of Spatial Language Interpretation and Generation (CoSLI-3)

2010

pdf bib
How Was Your Day?
Stephen Pulman | Johan Boye | Marc Cavazza | Cameron Smith | Raúl Santos de la Cámara
Proceedings of the 2010 Workshop on Companionable Dialogue Systems

2007

pdf bib
Dialogue Management for Automatic Troubleshooting and other Problem-solving Applications
Johan Boye
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
Multi-slot semantics for natural-language call routing systems
Johan Boye | Mats Wirén
Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies

2005

pdf bib
How to do Dialogue in a Fairy-tale World
Johan Boye | Joakim Gustafson
Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue

2004

pdf bib
The NICE Fairy-tale Game System
Joakim Gustafson | Linda Bell | Johan Boye | Anders Lindström | Mats Wirén
Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004

2001

pdf bib
Do CFG-Based Language Models Need Agreement Constraints?
Manny Rayner | Genevieve Gorrell | Beth Ann Hockey | John Dowding | Johan Boye
Second Meeting of the North American Chapter of the Association for Computational Linguistics

pdf bib
Plug and Play Speech Understanding
Manny Rayner | Ian Lewin | Genevieve Gorrell | Johan Boye
Proceedings of the Second SIGdial Workshop on Discourse and Dialogue