BEGIN:VCALENDAR VERSION:2.0 PRODID:IEEE vTools.Events//EN CALSCALE:GREGORIAN BEGIN:VTIMEZONE TZID:America/New_York BEGIN:DAYLIGHT DTSTART:20250309T030000 TZOFFSETFROM:-0500 TZOFFSETTO:-0400 RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3 TZNAME:EDT END:DAYLIGHT BEGIN:STANDARD DTSTART:20241103T010000 TZOFFSETFROM:-0400 TZOFFSETTO:-0500 RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11 TZNAME:EST END:STANDARD END:VTIMEZONE BEGIN:VEVENT DTSTAMP:20241209T162412Z UID:252900C1-421D-4161-B574-24EE9FEB76AF DTSTART;TZID=America/New_York:20241206T110000 DTEND;TZID=America/New_York:20241206T120000 DESCRIPTION:TALK: "From Filtering to Fingerprints: Constructing Pretraining Datasets for LLMs and Measuring Biases in the Data"\n\nVISITING SPEAKER: [Reinhard Heckel](https://n64lfj4ab.cc.rs6.net/tn.jsp?f=001J3H0PVaGqMWV08i sURJ3iOT_XJ8STTrSgdX8UUsy2-vFgfli0aZQg3Pi1LL9g2PWaBzJGMsEhy1OApDf2eKLjLS8c aqMEJiR1CRS9gyAgrxdGXWjW0mW2uXFlykK-5bRimHpINmMnuXwBDIDEa2PmmK9cnYrJsrF&c= oPKUxcWmtgc-Id8lnSU74HxZvh1vnZUBjSIvOambXJBoJClnrTi5gQ==&ch=InJBzzwmlc0zBx UTWJDASwbYGTZ6tUEUtBhMTOiqHonzieo8d1tkBQ==)\, an associate professor of ma chine learning at the Technical University of Munich\n\nWHEN: Friday\, Dec ember 6\, 2024 at 11 a.m.\n\nLOCATION: 5105 Iribe Center\, University of M aryland\n\nABSTRACT: In this talk\, we first discuss how pre-trained datas ets for LLMs are sourced from the web through heuristic and machine learni ng based filtering techniques. We then investigate biases in pretraining d atasets for large language models (LLMs) through dataset classification ex periments. Building on prior work demonstrating the existence of biases in popular computer vision datasets\, we analyze popular open-source pretrai ning text datasets derived from CommonCrawl including C4\, RefinedWeb\, Do lmaCC\, RedPajama-V2\, FineWeb and others. Despite those datasets being ob tained with similar filtering and deduplication steps\, LLMs can classify surprisingly well which dataset a single text sequence belongs to\, signif icantly better than a human can. This indicates that popular pretraining d atasets have their own unique biases or fingerprints.\n\nBIO: [Reinhard He ckel](https://n64lfj4ab.cc.rs6.net/tn.jsp?f=001J3H0PVaGqMWV08isURJ3iOT_XJ8 STTrSgdX8UUsy2-vFgfli0aZQg3Pi1LL9g2PWaBzJGMsEhy1OApDf2eKLjLS8caqMEJiR1CRS9 gyAgrxdGXWjW0mW2uXFlykK-5bRimHpINmMnuXwBDIDEa2PmmK9cnYrJsrF&c=oPKUxcWmtgc- Id8lnSU74HxZvh1vnZUBjSIvOambXJBoJClnrTi5gQ==&ch=InJBzzwmlc0zBxUTWJDASwbYGT Z6tUEUtBhMTOiqHonzieo8d1tkBQ==) is an associate professor of machine learn ing in the Department of Computer Engineering at the Technical University of Munich\, and an adjunct faculty member at Rice University. From 2017– 2019\, he was an assistant professor of electrical and computer engineerin g at Rice University. Before that\, Heckel was a postdoctoral researcher i n the Berkeley Artificial Intelligence Research Lab at UC Berkeley and a r esearcher at IBM Research Zurich. He completed his Ph.D. in 2014 at ETH Zu rich and was a visiting Ph.D. student at Stanford University's Statistics Department. Heckel's work focuses on machine learning\, artificial intelli gence\, and information processing. He specializes in developing algorithm s and foundations for deep learning\, particularly for medical imaging\, e stablishing mathematical and empirical underpinnings for machine learning\ , and utilizing DNA as a digital information technology.\n\nCo-sponsored b y: UMD Center for Machine Learning\n\nSpeaker(s): \, Reinhard Heckel\n\nRo om: 5105\, Bldg: Brendan Iribe Center for Computer Science and Engineering \, 8125 Paint Branch Dr\, College Park\, Maryland\, United States\, 20740 LOCATION:Room: 5105\, Bldg: Brendan Iribe Center for Computer Science and E ngineering\, 8125 Paint Branch Dr\, College Park\, Maryland\, United State s\, 20740 ORGANIZER:furongh@umd.edu SEQUENCE:11 SUMMARY:UMD Center for Machine Learning Visiting Talk URL;VALUE=URI:https://events.vtools.ieee.org/m/450312 X-ALT-DESC:Description: <br /><p>TALK: "From Filtering to Fingerprints: Con structing Pretraining Datasets for LLMs and Measuring Biases in the Data"< /p>\n<p>VISITING SPEAKER: <a href="https://n64lfj4ab.cc.rs6.net/tn.jsp?f=0 01J3H0PVaGqMWV08isURJ3iOT_XJ8STTrSgdX8UUsy2-vFgfli0aZQg3Pi1LL9g2PWaBzJGMsE hy1OApDf2eKLjLS8caqMEJiR1CRS9gyAgrxdGXWjW0mW2uXFlykK-5bRimHpINmMnuXwBDIDEa 2PmmK9cnYrJsrF&amp\;c=oPKUxcWmtgc-Id8lnSU74HxZvh1vnZUBjSIvOambXJBoJClnrTi5 gQ==&amp\;ch=InJBzzwmlc0zBxUTWJDASwbYGTZ6tUEUtBhMTOiqHonzieo8d1tkBQ==" tar get="_blank" rel="noopener noreferrer" data-saferedirecturl="https://www.g oogle.com/url?q=https://n64lfj4ab.cc.rs6.net/tn.jsp?f%3D001J3H0PVaGqMWV08i sURJ3iOT_XJ8STTrSgdX8UUsy2-vFgfli0aZQg3Pi1LL9g2PWaBzJGMsEhy1OApDf2eKLjLS8c aqMEJiR1CRS9gyAgrxdGXWjW0mW2uXFlykK-5bRimHpINmMnuXwBDIDEa2PmmK9cnYrJsrF%26 c%3DoPKUxcWmtgc-Id8lnSU74HxZvh1vnZUBjSIvOambXJBoJClnrTi5gQ%3D%3D%26ch%3DIn JBzzwmlc0zBxUTWJDASwbYGTZ6tUEUtBhMTOiqHonzieo8d1tkBQ%3D%3D&amp\;source=gma il&amp\;ust=1733322013636000&amp\;usg=AOvVaw2hu7liFaFA6G1J58mlpgcJ">Reinha rd Heckel</a>\, an associate professor of machine learning at the Technica l University of Munich</p>\n<p>WHEN:&nbsp\;Friday\, December 6\, 2024 at 1 1 a.m.</p>\n<p>LOCATION: 5105 Iribe Center\, University of Maryland</p>\n< p>ABSTRACT: In this talk\, we first discuss how pre-trained datasets for L LMs are sourced from the web through heuristic and machine learning based filtering techniques. We then investigate biases in pretraining datasets f or large language models (LLMs) through dataset classification experiments . Building on prior work demonstrating the existence of biases in popular computer vision datasets\, we analyze popular open-source pretraining text datasets derived from CommonCrawl including C4\, RefinedWeb\, DolmaCC\, R edPajama-V2\, FineWeb and others. Despite those datasets being obtained wi th similar filtering and deduplication steps\, LLMs can classify surprisin gly well which dataset a single text sequence belongs to\, significantly b etter than a human can. This indicates that popular pretraining datasets h ave their own unique biases or fingerprints.&nbsp\;</p>\n<p>BIO:&nbsp\;<a href="https://n64lfj4ab.cc.rs6.net/tn.jsp?f=001J3H0PVaGqMWV08isURJ3iOT_XJ8 STTrSgdX8UUsy2-vFgfli0aZQg3Pi1LL9g2PWaBzJGMsEhy1OApDf2eKLjLS8caqMEJiR1CRS9 gyAgrxdGXWjW0mW2uXFlykK-5bRimHpINmMnuXwBDIDEa2PmmK9cnYrJsrF&amp\;c=oPKUxcW mtgc-Id8lnSU74HxZvh1vnZUBjSIvOambXJBoJClnrTi5gQ==&amp\;ch=InJBzzwmlc0zBxUT WJDASwbYGTZ6tUEUtBhMTOiqHonzieo8d1tkBQ==" target="_blank" rel="noopener no referrer" data-saferedirecturl="https://www.google.com/url?q=https://n64lf j4ab.cc.rs6.net/tn.jsp?f%3D001J3H0PVaGqMWV08isURJ3iOT_XJ8STTrSgdX8UUsy2-vF gfli0aZQg3Pi1LL9g2PWaBzJGMsEhy1OApDf2eKLjLS8caqMEJiR1CRS9gyAgrxdGXWjW0mW2u XFlykK-5bRimHpINmMnuXwBDIDEa2PmmK9cnYrJsrF%26c%3DoPKUxcWmtgc-Id8lnSU74HxZv h1vnZUBjSIvOambXJBoJClnrTi5gQ%3D%3D%26ch%3DInJBzzwmlc0zBxUTWJDASwbYGTZ6tUE UtBhMTOiqHonzieo8d1tkBQ%3D%3D&amp\;source=gmail&amp\;ust=1733322013636000& amp\;usg=AOvVaw2hu7liFaFA6G1J58mlpgcJ">Reinhard Heckel</a> is an associate professor of machine learning in the Department of Computer Engineering a t the Technical University of Munich\, and an adjunct faculty member at Ri ce University. From 2017&ndash\;2019\, he was an assistant professor of el ectrical and computer engineering at Rice University. Before that\,&nbsp\; Heckel&nbsp\;was a postdoctoral researcher in the Berkeley Artificial Inte lligence Research Lab at UC Berkeley and a researcher at IBM Research Zuri ch. He completed his Ph.D. in 2014 at ETH Zurich and was a visiting Ph.D. student at Stanford University's Statistics Department.&nbsp\;Heckel's&nbs p\;work focuses on machine learning\, artificial intelligence\, and inform ation processing. He specializes in developing algorithms and foundations for deep learning\, particularly for medical imaging\, establishing mathem atical and empirical underpinnings for machine learning\, and utilizing DN A as a digital information technology.</p> END:VEVENT END:VCALENDAR