Thomas L. Griffiths

Also published as: Thomas Griffiths


2022

pdf bib
Probing BERT’s priors with serial reproduction chains
Takateru Yamakoshi | Thomas Griffiths | Robert Hawkins
Findings of the Association for Computational Linguistics: ACL 2022

Sampling is a promising bottom-up method for exposing what generative models have learned about language, but it remains unclear how to generate representative samples from popular masked language models (MLMs) like BERT. The MLM objective yields a dependency network with no guarantee of consistent conditional distributions, posing a problem for naive approaches. Drawing from theories of iterated learning in cognitive science, we explore the use of serial reproduction chains to sample from BERT’s priors. In particular, we observe that a unique and consistent estimator of the ground-truth joint distribution is given by a Generative Stochastic Network (GSN) sampler, which randomly selects which token to mask and reconstruct on each step. We show that the lexical and syntactic statistics of sentences from GSN chains closely match the ground-truth corpus distribution and perform better than other methods in a large corpus of naturalness judgments. Our findings establish a firmer theoretical foundation for bottom-up probing and highlight richer deviations from human priors.

2020

pdf bib
Investigating representations of verb bias in neural language models
Robert Hawkins | Takateru Yamakoshi | Thomas Griffiths | Adele Goldberg
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Languages typically provide more than one grammatical construction to express certain types of messages. A speaker’s choice of construction is known to depend on multiple factors, including the choice of main verb – a phenomenon known as verb bias. Here we introduce DAIS, a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and systematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under comparable parameter and training settings. Additional analyses of internal feature representations suggest that transformers may better integrate specific lexical information with grammatical constructions.

2011

pdf bib
Exploring the Relationship Between Learnability and Linguistic Universals
Anna N. Rafferty | Thomas L. Griffiths | Marc Ettlinger
Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics

2009

pdf bib
Improved Reconstruction of Protolanguage Word Forms
Alexandre Bouchard-Côté | Thomas L. Griffiths | Dan Klein
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics

2007

pdf bib
A Probabilistic Approach to Diachronic Phonology
Alexandre Bouchard | Percy Liang | Thomas Griffiths | Dan Klein
Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)

pdf bib
Bayesian Inference for PCFGs via Markov Chain Monte Carlo
Mark Johnson | Thomas Griffiths | Sharon Goldwater
Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference

2006

pdf bib
Unsupervised Topic Modelling for Multi-Party Spoken Discourse
Matthew Purver | Konrad P. Körding | Thomas L. Griffiths | Joshua B. Tenenbaum
Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics

pdf bib
Contextual Dependencies in Unsupervised Word Segmentation
Sharon Goldwater | Thomas L. Griffiths | Mark Johnson
Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics