I am a postdoc at the NYU Center for Data Science working with Sam Bowman. I earned my PhD from the University of Washington where my advisor was Luke Zettlemoyer. I work on a variety of things, mostly focused on AI alignment or formal semantics of natural language.
In alignment, I focus on scalable oversight and agent alignment, from the lens of task formulation, data collection, and evaluation methodology. I’m especially interested in using debate as a training and evaluation paradigm, and recently tested this with humans. I’m also interested in pushing the boundaries of difficult evaluations for scalable oversight, as in our recent release of GPQA.
In language, I work on ways to use data and machine learning to help us do a better science of language, particularly when it comes to syntax and semantics. I lay out this scientific paradigm in my thesis, described best in my talk at the Big Picture Workshop. In constructing the building blocks for this, I have developed approaches to crowdsourcing annotation for syntactic parsing, semantic role labeling, and predicate-argument structure.
More broadly, I am interested in the Science of AI and NLP, using empirical methods to improve our understanding of intelligent behavior and language use. Along these lines, I have worked on broad-coverage and fine-grained evaluation of models, unsupervised discovery of linguistic structure, and explicitly incorporating ambiguity into task design. See my publications for a full list.
Selected Publications (see all)
Selected Talks (see all)
This is the best entry point to my thesis work, laying out my proposal for how to use machine learning to do better science, specifically in the case of syntax and semantics.
video slides paper
A summary of the contemporary state of NLP and a (novel at the time, as far as I'm aware) proposal for how to use language models for scientific data analysis. Includes spicy takes on the relationship between deep learning and psychiatric drugs and more.
video slides colab
An introduction to foundational issues in AI ethics from a philosophical perspective. I try to connect contemporary views and disagreements to their philosophical roots in consequentialism, deontology, social contract theory, and critical theory.
video slides
A 10-minute, accessible colloquium talk summarizing some of my early PhD work on crowdsourcing representations of language structure and meaning. My experience with the projects described in this talk led me to invest further in QA-SRL.
video slides
Random Thoughts
- An in-depth review of a mid-2021 version of the OpenPhil Biological Anchors report on transformative AI timelines.
- To Dissect an Octopus, a blog post taking a deep dive into the form/meaning debate around language models.
- A long comment thread on the Alignment Forum discussing limits on the extrapolations we can make about automation potential based on ML benchmarks.
- The GLUE diagnostic set guide, which doubles as a quick tour of fun phenomena in semantics.
- Fulfilling Imperatives, an essay investigating of the semantics of imperative sentences.
- Modern Cosmology: Explaining the Universe, an essay investigating whether inflation theory qualifies as science.