Publications

View on Semantic Scholar / Google Scholar / ACL Anthology / DBLP

Showing (All/None): Conferences / Findings / Journals / Workshops / Technical reports / Theses / Preprints

2026

1. LLM Novice Uplift on Dual-Use, In Silico Biology Tasks
   Chen Bo Calvin Zhang, Christina Q. Knight, Nicholas Kruus, Jason Hausenloy, Pedro Medeiros, Nathaniel Li, Aiden Kim, Yury Orlovskiy, Coleman Breen, Bryce Cai, Jasper Götting, Andrew Bo Liu, Samira Nedungadi, Paula Rodriguez, Yannis Yiming He, Mohamed Shaaban, Zifan Wang, Seth Donoughe and Julian Michael
   preprint
   pdf arxiv bib

2025

1. Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models
   Boyi Wei, Zora Che, Nathaniel Li, Udari Madhushani Sehwag, Jasper Götting, Samira Nedungadi, Julian Michael, Summer Yue, Dan Hendrycks, Peter Henderson, Zifan Wang, Seth Donoughe and Mantas Mazeika
   preprint
   pdf arxiv bib
2. Remote Labor Index: Measuring AI Automation of Remote Work
   Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, Oliver Ingebretsen, Aditya Mehta, Jean Mottola, John Teichmann, Kevin Yu, Zaina Shaik, Adam Khoja, Richard Ren, Jason Hausenloy, Long Phan, Ye Htet, Ankit Aich, Tahseen Rabbani, Vivswan Shah, Andriy Novykov, Felix Binder, Kirill Chugunov, Luis Ramirez, Matias Geralnik, Hernán Mesura, Dean Lee, Ed-Yeremai Hernandez Cardona, Annette Diamond, Summer Yue, Alexandr Wang, Bing Liu, Ernesto Hernandez and Dan Hendrycks
   preprint
   website pdf arxiv bib
3. Quantifying Elicitation of Latent Capabilities in Language Models
   Elizabeth Donoway, Hailey Joren, Arushi Somani, Henry Sleight, Julian Michael, Michael R. DeWeese, John Schulman, Ethan Perez, Fabien Roger and Jan Leike
   NeurIPS 2025
   pdf openreview code bib
4. Search-Time Data Contamination
   Ziwen Han, Meher Mankikar, Julian Michael and Zifan Wang
   preprint
   pdf arxiv bib
5. Inverse Scaling in Test-Time Compute
   Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton and Ethan Perez
   TMLR (12/2025; Featured Certification)
   pdf arxiv openreview bib
6. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
   Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah and Vlad Mikulik
   preprint
   pdf arxiv bib
7. Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning
   Miles Turpin, Andy Arditi, Marvin Li, Joe Benton and Julian Michael
   ICML 2025 R2-FM Workshop
   pdf arxiv bib
8. The Singapore Consensus on Global AI Safety Research Priorities
   Yoshua Bengio, Tegan Maharaj, Luke Ong, Stuart Russell, Dawn Song, Max Tegmark, Lan Xue, Ya-Qin Zhang, Stephen Casper, Wan Sie Lee, Sören Mindermann, Vanessa Wilfred, Vidhisha Balachandran, Fazl Barez, Michael Belinsky, Imane Bello, Malo Bourgon, Mark Brakel, Siméon Campos, Duncan Cass-Beggs, Jiahao Chen, Rumman Chowdhury, Kuan Chua Seah, Jeff Clune, Juntao Dai, Agnes Delaborde, Nouha Dziri, Francisco Eiras, Joshua Engels, Jinyu Fan, Adam Gleave, Noah Goodman, Fynn Heide, Johannes Heidecke, Dan Hendrycks, Cyrus Hodes, Bryan Low Kian Hsiang, Minlie Huang, Sami Jawhar, Wang Jingyu, Adam Tauman Kalai, Meindert Kamphuis, Mohan Kankanhalli, Subhash Kantamneni, Mathias Bonde Kirk, Thomas Kwa, Jeffrey Ladish, Kwok-Yan Lam, Wan Lee Sie, Taewhi Lee, Xiaojian Li, Jiajun Liu, Chaochao Lu, Yifan Mai, Richard Mallah, Julian Michael, Nick Moës, Simon Möller, Kihyuk Nam, Kwan Yee Ng, Mark Nitzberg, Besmira Nushi, Seán O hÉigeartaigh, Alejandro Ortega, Pierre Peigné, James Petrie, Benjamin Prud’Homme, Reihaneh Rabbany, Nayat Sanchez-Pi, Sarah Schwettmann, Buck Shlegeris, Saad Siddiqui, Aradhana Sinha, Martín Soto, Cheston Tan, Dong Ting, William Tjhi, Robert Trager, Brian Tse, Anthony Tung K. H., John Willes, Denise Wong, Wei Xu, Rongwu Xu, Yi Zeng, HongJiang Zhang and Djordje Žikelić
   SCAI 2025
   pdf arxiv bib
9. Why Do Some Language Models Fake Alignment While Others Don’t?
   Abhay Sheshadri, John Hughes, Julian Michael, Alex Mallen, Arun Jose, Janus and Fabien Roger
   NeurIPS 2025 (Spotlight)
   pdf arxiv bib
10. FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
    Christina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh, Meher Mankikar, Scale Red Team, SEAL Research Team and Julian Michael
    preprint
    s2 pdf arxiv bib
11. AI Debate Aids Assessment of Controversial Claims
    Salman Rahman, Sheriff Issaka, Ashima Suvarna, Genglin Liu, James Shiffer, Jaeyoung Lee, Md Rizwan Parvez, Hamid Palangi, Shi Feng, Nanyun Peng, Yejin Choi, Julian Michael, Liwei Jiang and Saadia Gabriel
    MAS 2025
    s2 pdf arxiv bib
12. A Red Teaming Roadmap Towards System-Level Safety
    Zifan Wang, Christina Q. Knight, Jeremy Kritz, Willow E. Primack and Julian Michael
    preprint
    s2 pdf arxiv bib
13. International AI Safety Report
    Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, Hoda Heidari, Anson Ho, Sayash Kapoor, Leila Khalatbari, Shayne Longpre, Sam Manning, Vasilios Mavroudis, Mantas Mazeika, Julian Michael, Jessica Newman, Kwan Yee Ng, Chinasa T. Okolo, Deborah Raji, Girish Sastry, Elizabeth Seger, Theodora Skeadas, Tobin South, Emma Strubell, Florian Tramèr, Lucia Velasco, Nicole Wheeler, Daron Acemoglu, Olubayo Adekanmbi, David Dalrymple, Thomas G. Dietterich, Edward W. Felten, Pascale Fung, Pierre-Olivier Gourinchas, Fredrik Heintz, Geoffrey Hinton, Nick Jennings, Andreas Krause, Susan Leavy, Percy Liang, Teresa Ludermir, Vidushi Marda, Helen Margetts, John McDermid, Jane Munga, Arvind Narayanan, Alondra Nelson, Clara Neppel, Alice Oh, Gopal Ramchurn, Stuart Russell, Marietje Schaake, Bernhard Schölkopf, Dawn Song, Alvaro Soto, Lee Tiedrich, Gaël Varoquaux, Andrew Yao, Ya-Qin Zhang, Olubunmi Ajala, Fahad Albalawi, Marwan Alserkal, Guillaume Avrin, Christian Busch, André Carlos Ponce de Leon Ferreira de Carvalho, Bronwyn Fox, Amandeep Singh Gill, Ahmet Halit Hatip, Juha Heikkilä, Chris Johnson, Gill Jolly, Ziv Katzir, Saif M. Khan, Hiroaki Kitano, Antonio Krüger, Kyoung Mu Lee, Dominic Vincent Ligot, José Ramón López Portillo, Oleksii Molchanovskyi, Andrea Monti, Nusu Mwamanzi, Mona Nemer, Nuria Oliver, Raquel Pezoa Rivera, Balaraman Ravindran, Hammam Riza, Crystal Rugege, Ciarán Seoighe, Jerry Sheehan, Haroon Sheikh, Denise Wong and Yi Zeng
    UK DSIT/AISI Report
    website s2 pdf arxiv bib

2024

1. Alignment faking in large language models
   Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman and Evan Hubinger
   preprint
   website s2 pdf arxiv code video memo bib
2. Rapid Response: Mitigating LLM Jailbreaks with a Few Examples
   Alwin Peng, Julian Michael, Henry Sleight, Ethan Perez and Mrinank Sharma
   ICLR 2025
   s2 pdf arxiv code openreview bib
3. Training Language Models to Win Debates with Self-Play Improves Judge Accuracy
   Samuel Arnesen, David Rein and Julian Michael
   preprint
   s2 pdf arxiv code bib
   Media: Quanta Magazine
4. GPQA: A Graduate-Level Google-Proof Q&A Benchmark
   David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael† and Samuel R. Bowman†
   COLM 2024 (Spotlight)
   pdf arxiv data talk twitter reviews bib
   Media: AI Index (2024; Ch. 2), Nature
5. Research Agenda for Sociotechnical Approaches to AI Safety
   Samuel Curtis, Ravi Iyer, Cameron Domenico Kirk-Giannini, Victoria Krakovna, David Krueger, Nathan Lambert, Bruno Marnette, Colleen McKenzie, Julian Michael, Evan Miyazono, Noyuri Mima, Aviv Ovadya, Luke Thorburn and Deger Turan
   preprint
   pdf bib
6. Analyzing the Role of Semantic Representations in the Era of Large Language Models
   Zhijing Jin,* Yuen Chen,* Fernando Gonzalez,* Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf and Mona Diab
   NAACL 2024
   pdf arxiv code bib
7. Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
   James Chua,* Edward Rees,* Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez and Miles Turpin
   arXiv preprint
   pdf arxiv code twitter bib
   Media: NYU CDS Blog

2023

1. Debate Helps Supervise Unreliable Experts
   Julian Michael,* Salsabila Mahdi,* David Rein,* Jackson Petty, Julien Dirani, Vishakh Padmakumar and Samuel R. Bowman
   arXiv preprint
   website pdf arxiv code data twitter bib
2. Eliciting Language Model Behaviors using Reverse Language Models
   Jacob Pfau,* Alex Infanger,* Abhay Sheshadri,* Ayush Panda, Curtis Huebner and Julian Michael
   SoLaR 2023 (Spotlight)
   reviews pdf bib
3. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
   Miles Turpin, Julian Michael, Ethan Perez and Samuel R. Bowman
   NeurIPS 2023
   s2 pdf arxiv code lw twitter bib
   Media: Nature, Neue Zürcher Zeitung
4. The Case for Scalable, Data-Driven Theory: A Paradigm for Scientific Progress in NLP
   Julian Michael
   The Big Picture Workshop (Best Paper)
   pdf arxiv slides twitter bib
5. We’re Afraid Language Models Aren’t Modeling Ambiguity
   Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith and Yejin Choi
   EMNLP 2023
   s2 pdf arxiv data code twitter bib
6. Building Blocks for Data-Driven Theories of Language Understanding
   Julian Michael
   PhD Thesis, University of Washington, 2023
   pdf bib
7. What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
   Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang and Samuel R. Bowman
   ACL 2023
   website s2 pdf arxiv acl poster talk twitter bib
   Media: AI Index (2022; Ch. 8), Data Skeptic Podcast, IFLScience, Tages-Anzeiger, New Scientist, The Times, Yahoo, NLP Deep Dive, NYU CDS Blog

2022

1. Overconfidence in the Face of Ambiguity with Adversarial Data
   Margaret Li* and Julian Michael*
   DADC 2022 (Best Paper)
   s2 pdf code reviews bib
2. Nearest Neighbor Zero-Shot Inference
   Weijia Shi, Julian Michael, Suchin Gururangan and Luke Zettlemoyer
   EMNLP 2022
   s2 pdf arxiv twitter bib

2021

1. Asking It All: Generating Contextualized Questions for any Semantic Role
   Valentina Pyatkin, Paul Roit, Julian Michael, Yoav Goldberg, Reut Tsarfaty and Ido Dagan
   EMNLP 2021
   s2 pdf arxiv code twitter bib
2. Inducing Semantic Roles Without Syntax
   Julian Michael and Luke Zettlemoyer
   Findings of ACL 2021
   website s2 pdf code bib
3. Prompting Contrastive Explanations for Commonsense Reasoning Tasks
   Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi and Luke Zettlemoyer
   Findings of ACL 2021
   s2 pdf arxiv code bib

2020

1. Asking without Telling: Exploring Latent Ontologies in Contextual Representations
   Julian Michael, Jan A. Botha and Ian Tenney
   EMNLP 2020
   s2 pdf arxiv twitter bib
2. Controlled Crowdsourcing for High-Quality QA-SRL Annotation
   Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer and Ido Dagan
   ACL 2020
   s2 pdf arxiv code data bib
3. AmbigQA: Answering Ambiguous Open-domain Questions
   Sewon Min, Julian Michael, Hannaneh Hajishirzi and Luke Zettlemoyer
   EMNLP 2020
   website s2 pdf arxiv code data twitter bib

2019

1. SuperGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
   Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman
   NeurIPS 2019
   website s2 pdf arxiv code bib
2. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
   Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman
   ICLR 2019
   website s2 reviews pdf arxiv code bib

2018

1. Large-Scale QA-SRL Parsing
   Nicholas FitzGerald, Julian Michael, Luheng He and Luke Zettlemoyer
   ACL 2018 (Honorable Mention)
   website s2 pdf arxiv code data twitter bib
2. Crowdsourcing Question-Answer Meaning Representations
   Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan and Luke Zettlemoyer
   NAACL 2018
   s2 pdf arxiv poster code data bib
3. Supervised Open Information Extraction
   Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer and Ido Dagan
   NAACL 2018
   s2 pdf poster code bib

2016

1. Human-in-the-Loop Parsing
   Luheng He, Julian Michael, Mike Lewis and Luke Zettlemoyer
   EMNLP 2016
   s2 pdf code slides bib
2. Proving Infinitary Formulas
   Amelia Harrison, Vladimir Lifschitz and Julian Michael
   TPLP Vol. 16, 5–6; Presented at ICLP 2016
   s2 pdf bib

2015

1. The Theory of Correlation Formulas and Their Application to Discourse Coherence
   Julian Michael
   Undergraduate Honors Thesis, UT Austin, 2015
   s2 pdf bib
2. The Winograd Schema Challenge and Reasoning about Correlation
   Daniel Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz and Julian Michael
   Commonsense 2015
   s2 pdf bib