I am a postdoc at the NYU Center for Data Science working with Sam Bowman. I earned my PhD from the University of Washington where my advisor was Luke Zettlemoyer. I work on a variety of things, mostly focused on AI alignment or formal semantics of natural language.
In alignment, I focus on scalable oversight and agent alignment, from the lens of task formulation, data collection, and evaluation methodology. I’m especially interested in using debate as a training and evaluation paradigm, and recently tested this with humans. I’m also interested in pushing the boundaries of difficult evaluations for scalable oversight, as in our recent release of GPQA.
In language, I work on ways to use data and machine learning to help us do a better science of language, particularly when it comes to syntax and semantics. I lay out this scientific paradigm in my thesis, described best in my talk at the Big Picture Workshop. In constructing the building blocks for this, I have developed approaches to crowdsourcing annotation for syntactic parsing, semantic role labeling, and predicate-argument structure.
More broadly, I am interested in the Science of AI and NLP, using empirical methods to improve our understanding of intelligent behavior and language use. Along these lines, I have worked on broad-coverage and fine-grained evaluation of models, unsupervised discovery of linguistic structure, and explicitly incorporating ambiguity into task design. See my publications for a full list.
Selected publications (see all)
-
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
COLM 2024 (Spotlight)
pdf arxiv data twitter reviews bib
Media: AI Index (2024; Ch. 2), Nature
Selected talks (see all)
Other writings
- An in-depth review of a mid-2021 version of the OpenPhil Biological Anchors report on transformative AI timelines.
- To Dissect an Octopus, a blog post taking a deep dive into the form/meaning debate around language models.
- A long comment thread on the Alignment Forum discussing limits on the extrapolations we can make about automation potential based on ML benchmarks.
- The GLUE diagnostic set guide, which doubles as a quick tour of fun phenomena in semantics.
- Fulfilling Imperatives, an essay investigating of the semantics of imperative sentences.
- Modern Cosmology: Explaining the Universe, an essay investigating whether inflation theory qualifies as science.