Hi, I’m Julian. I currently head the Safety, Evaluations, and Alignment Lab (SEAL) at Scale AI. We do research related to safeguarding AI system behavior and ensuring it preserves and amplifies human agency. Prior to SEAL, I did research on a variety of topics, mostly focused on AI alignment or formal semantics of natural language.
In alignment, I focus on scalable oversight and agent alignment, from the lens of task formulation, data collection, and evaluation methodology. I’m especially interested in using debate as a training and evaluation paradigm, which I helped test with humans. I’m also interested in pushing the boundaries of difficult evaluations for scalable oversight, as in our release of GPQA.
In language, I work on ways to use data and machine learning to help us do a better science of language, particularly when it comes to syntax and semantics. I lay out this scientific paradigm in my PhD thesis, described best in my 2023 talk at the Big Picture Workshop. In constructing the building blocks for this, I have developed approaches to crowdsourcing annotation for syntactic parsing, semantic role labeling, and predicate-argument structure.
More broadly, I am interested in the Science of AI and NLP, using empirical methods to improve our understanding of intelligent behavior and language use. Along these lines, I have worked on broad-coverage and fine-grained evaluation of models, unsupervised discovery of linguistic structure, and explicitly incorporating ambiguity into task design. See my publications for a full list.
Selected publications (see all)
Selected talks (see all)
Other writings (see all)
- An in-depth review of a mid-2021 version of the OpenPhil Biological Anchors report on transformative AI timelines.
- To Dissect an Octopus, a blog post taking a deep dive into the form/meaning debate around language models.
- A long comment thread on the Alignment Forum discussing limits on the extrapolations we can make about automation potential based on ML benchmarks.
- The GLUE diagnostic set guide, which doubles as a quick tour of fun phenomena in semantics.
- Fulfilling Imperatives, an essay investigating of the semantics of imperative sentences.
- Modern Cosmology: Explaining the Universe, an essay investigating whether inflation theory qualifies as science.