Technical seminar by Prof. Flavio P. Calmon, Havard University: Inference-Time Information Theory

#machine-learning #information-theory #large-language-models #coding-theory #watermarking
Share

In this talk, we argue that “inference-time” large language model (LLM) operation, where we interact with models without modifying their weights, is fertile ground for information-theoretic methods. As LLMs increasingly serve as black-box components of more complex systems, information and coding theory offer a principled toolkit for shaping, verifying, and controlling their outputs. We focus on one challenge in particular: watermarking LLM-generated text. Watermarks enable authentication of text provenance and help curb misuse of machine-generated content. 

We present recent results demonstrating a close connection between LLM watermarking and coding theory, showing that classical tools such as the Plotkin bound yield fundamental limits on watermark performance. This perspective also informs the design of two practical watermarks: SimplexWater and HeavyWater. We also briefly survey other inference-time challenges that can be addressed with information theory, such as inference-time alignment.


  Date and Time

  Location

  Hosts

  Registration


  • Add_To_Calendar_icon Add Event to Calendar

Loading virtual attendance info...

  • KTH
  • Osquars Backe 5
  • Stockholm, Stockholms lan
  • Sweden
  • Room Number: Digital Futures hub, floor 2
  • Contact Event Host

  • Tobias Oechtering, KTH, oech@kth.se

  • Co-sponsored by KTH Royal Institute of Technology

  Speakers

Prof. Flavio du Pin Calmon of Havard University

Topic:

Inference-Time Information Theory

In this talk, we argue that “inference-time” large language model (LLM) operation, where we interact with models without modifying their weights, is fertile ground for information-theoretic methods. As LLMs increasingly serve as black-box components of more complex systems, information and coding theory offer a principled toolkit for shaping, verifying, and controlling their outputs. We focus on one challenge in particular: watermarking LLM-generated text. Watermarks enable authentication of text provenance and help curb misuse of machine-generated content. 

We present recent results demonstrating a close connection between LLM watermarking and coding theory, showing that classical tools such as the Plotkin bound yield fundamental limits on watermark performance. This perspective also informs the design of two practical watermarks: SimplexWater and HeavyWater. We also briefly survey other inference-time challenges that can be addressed with information theory, such as inference-time alignment.

Biography:

Flavio P. Calmon is the Thomas D. Cabot Associate Professor of Electrical Engineering at the Harvard John A. Paulson School of Engineering and Applied Sciences and a Visiting Faculty Researcher at Google Research. Before joining Harvard, he was the inaugural Data Science for Social Good Post-Doctoral Fellow at IBM Research in Yorktown Heights, New York. He received his Ph.D. in Electrical Engineering and Computer Science from MIT. His research develops the information-theoretic foundations of trustworthy and reliable machine learning and artificial intelligence. 

Prof. Calmon received the 2024 James L. Massey Award from the IEEE InformationTheory Society, the NSF CAREER award, faculty awards from Google, IBM, JPMorganChase, and Amazon, the Harvard Dean of Undergraduate Studies Commendation for “Extraordinary Teaching during Extraordinary Times,” and the Capers and Marion McDonald Award for Excellence in Mentoring and Advising at Harvard SEAS. He also received the inaugural “Título de Honra ao Mérito” (Honor to the Merit Title) given to alumni from the Universidade de Brasília (Brazil).

Email:

Address:Havard University, Harvard John A. Paulson School of Engineering and Applied Sciences, Allston, Massachusetts, United States, MA 20134