BEGIN:VCALENDAR VERSION:2.0 PRODID:IEEE vTools.Events//EN CALSCALE:GREGORIAN BEGIN:VTIMEZONE TZID:America/New_York BEGIN:DAYLIGHT DTSTART:20240310T030000 TZOFFSETFROM:-0500 TZOFFSETTO:-0400 RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3 TZNAME:EDT END:DAYLIGHT BEGIN:STANDARD DTSTART:20241103T010000 TZOFFSETFROM:-0400 TZOFFSETTO:-0500 RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11 TZNAME:EST END:STANDARD END:VTIMEZONE BEGIN:VEVENT DTSTAMP:20241003T144111Z UID:93598DDF-4AF2-4822-872C-DDC4EF94A85A DTSTART;TZID=America/New_York:20240502T110000 DTEND;TZID=America/New_York:20240502T120000 DESCRIPTION:IEEE ComSoc Norther Virginia chapter and GMU Department of Comp uter Science invites you to attend the following Distinguished Lecture:\n\ nTitle: Why do small language models underperform?\n\nSpeaker: Benoît Sag ot\, Director of Research at INRIA\n\nDate: May 2\, 2024\n\nTime: 11:00am – 12:00pm\n\nIn person Location: GMU Fairfax campus\, Nguyen Engineering Bldg.\, Conference Room 4201\nVirtual: Microsoft Teams: [Join the meeting now](https://nam11.safelinks.protection.outlook.com/ap/t-59584e83/?url=ht tps%3A%2F%2Fteams.microsoft.com%2Fl%2Fmeetup-join%2F19%253ameeting_OGUwZGI 0OTktNTdhZS00NmNlLWEzZGEtZTJhNGI2Yjg5YmJi%2540thread.v2%2F0%3Fcontext%3D%2 57b%2522Tid%2522%253a%25229e857255-df57-4c47-a0c0-0546460380cb%2522%252c%2 522Oid%2522%253a%2522f9586db0-74ee-4635-a01b-2383b74f8a0c%2522%257d&data=0 5%7C02%7Ckhassan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e857255d f574c47a0c00546460380cb%7C0%7C0%7C638501649141618849%7CUnknown%7CTWFpbGZsb 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0% 7C%7C%7C&sdata=MUjIg2xib2wJ22N21llS60nE%2FSTEPpqg%2FSVOdLXJFHA%3D&reserved =0) Meeting ID: 292 789 339 112 Passcode: jM8w7c\n------------------------ ---------------------------------------\n\nDial-in by phone\n[+1 571-397-2 084\,\,218888141#](tel:+15713972084\,\,218888141#) United States\, Arlingt on\n[Find a local number](https://nam11.safelinks.protection.outlook.com/? url=https%3A%2F%2Fdialin.teams.microsoft.com%2F9424c9fe-3b57-41d6-9131-1d3 b9b7cf4a9%3Fid%3D218888141&data=05%7C02%7Ckhassan1%40gmu.edu%7Cc082474ca08 14691061508dc69ddedfd%7C9e857255df574c47a0c00546460380cb%7C0%7C0%7C6385016 49141626316%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=r6YY83JURqPLJf9J2QcdQ5H6w3 QBZTlOs4HAUUBvvSQ%3D&reserved=0)\nPhone conference ID: 218 888 141#\nFor o rganizers: [Meeting options](https://nam11.safelinks.protection.outlook.co m/?url=https%3A%2F%2Fteams.microsoft.com%2FmeetingOptions%2F%3ForganizerId %3Df9586db0-74ee-4635-a01b-2383b74f8a0c%26tenantId%3D9e857255-df57-4c47-a0 c0-0546460380cb%26threadId%3D19_meeting_OGUwZGI0OTktNTdhZS00NmNlLWEzZGEtZT JhNGI2Yjg5YmJi%40thread.v2%26messageId%3D0%26language%3Den-US&data=05%7C02 %7Ckhassan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e857255df574c4 7a0c00546460380cb%7C0%7C0%7C638501649141633701%7CUnknown%7CTWFpbGZsb3d8eyJ WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C% 7C&sdata=KSJGsLYD3jKk7GBpn2XuSPetsCGuvQn9I6sJYXPZfs0%3D&reserved=0) | [Res et dial-in PIN](https://nam11.safelinks.protection.outlook.com/?url=https% 3A%2F%2Fdialin.teams.microsoft.com%2Fusp%2Fpstnconferencing&data=05%7C02%7 Ckhassan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e857255df574c47a 0c00546460380cb%7C0%7C0%7C638501649141641487%7CUnknown%7CTWFpbGZsb3d8eyJWI joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C &sdata=2ONDG7yjY3RckniAVPkRkliPHOBWdNqIhM2RgAA8O10%3D&reserved=0)\n\nAbstr act:\n\nLanguage models\, and in particular generative and conversational language models\, are at the heart of recent advances in natural language processing (NLP). Understanding how these models represent textual content and how they learn these representations still raises multiple research q uestions. In this talk\, I will start from an observation that small model s are less efficient than expected. I will show that language models relyi ng on the Transformer architecture tend to produce vector representations that are not isotropically distributed in space. This anisotropy is linked to the way in which these models are learned\, which leads to the frequen cy of the tokens taking a preponderant place in their representation. I wi ll show that this effect has negative consequences on the ability of small models to train satisfactorily (“performance saturation”) but does no t seem to affect larger models. I will then describe a new approach for tr aining language models intended to avoid the undesirable effects of this p revalence of frequency information. The resulting “headless” models di splay a number of advantages over standard models\, including on downstrea m performance.\n\nBio:\n\nBenoît Sagot is a computer scientist specialize d in natural language processing (NLP). He is a Senior Researcher (Directe ur de Recherches) at INRIA\, where is heads the INRIA research project ALM AnaCH in Paris\, France. He also holds a chair in the PRAIRIE institute de dicated to artificial intelligence\, and currently holds the annual chair for computer science in the Collège de France. His research focuses on la nguage modelling\, machine translation\, language resource development and computational linguistics\, with a focus on French in all its form and on less-resourced languages.\n\n____________________________________________ ____________________________________\n\nCo-sponsored by: GMU Department of Computer Science\n\nBldg: Nguyen Engineering Bldg.\, Conference Room 4201 \, 4400 University Drive \, Fairfax\, Virginia\, United States\, 22030\, V irtual: https://events.vtools.ieee.org/m/419402 LOCATION:Bldg: Nguyen Engineering Bldg.\, Conference Room 4201\, 4400 Unive rsity Drive \, Fairfax\, Virginia\, United States\, 22030\, Virtual: https ://events.vtools.ieee.org/m/419402 ORGANIZER: SEQUENCE:3 SUMMARY:Why do small language models underperform? URL;VALUE=URI:https://events.vtools.ieee.org/m/419402 X-ALT-DESC:Description: <br /><div>\n<div class="x_WordSection1">\n<p class ="x_MsoNormal">IEEE ComSoc Norther Virginia chapter and GMU Department of Computer Science invites you to attend the following Distinguished Lecture :</p>\n<p class="x_MsoNormal">Title:<strong>&nbsp\;</strong><strong>Why do small language models underperform?</strong></p>\n<p class="x_MsoNormal"> Speaker:<strong>&nbsp\;Beno&icirc\;t Sagot\, Director of Research at INRIA </strong></p>\n<p class="x_MsoNormal" style="text-align: left\;" align="ce nter">Date: May 2\, 2024</p>\n<p class="x_MsoNormal" style="text-align: le ft\;" align="center">Time: 11:00am &ndash\; 12:00pm</p>\n<p class="x_MsoNo rmal" style="text-align: left\;" align="center">In person Location: GMU Fa irfax campus\, Nguyen Engineering Bldg.\, Conference Room 4201</p>\n<div>< span class="x_me-email-text">Virtual: Microsoft Teams: &nbsp\;</span><a id ="x_meet_invite_block.action.join_link" class="x_me-email-headline" title= "Original URL: https://teams.microsoft.com/l/meetup-join/19%3ameeting_OGUw ZGI0OTktNTdhZS00NmNlLWEzZGEtZTJhNGI2Yjg5YmJi%40thread.v2/0?context=%7b%22T id%22%3a%229e857255-df57-4c47-a0c0-0546460380cb%22%2c%22Oid%22%3a%22f9586d b0-74ee-4635-a01b-2383b74f8a0c%22%7d. Click or tap if you trust this link. " href="https://nam11.safelinks.protection.outlook.com/ap/t-59584e83/?url= https%3A%2F%2Fteams.microsoft.com%2Fl%2Fmeetup-join%2F19%253ameeting_OGUwZ GI0OTktNTdhZS00NmNlLWEzZGEtZTJhNGI2Yjg5YmJi%2540thread.v2%2F0%3Fcontext%3D %257b%2522Tid%2522%253a%25229e857255-df57-4c47-a0c0-0546460380cb%2522%252c %2522Oid%2522%253a%2522f9586db0-74ee-4635-a01b-2383b74f8a0c%2522%257d&amp\ ;data=05%7C02%7Ckhassan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e 857255df574c47a0c00546460380cb%7C0%7C0%7C638501649141618849%7CUnknown%7CTW FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%7C0%7C%7C%7C&amp\;sdata=MUjIg2xib2wJ22N21llS60nE%2FSTEPpqg%2FSVOdLXJFHA %3D&amp\;reserved=0" target="_blank" rel="noreferrer noopener" data-auth=" Verified" data-linkindex="1">Join the meeting now</a><span class="x_me-ema il-text-secondary">&nbsp\;Meeting ID: </span><span class="x_me-email-text" >292 789 339 112</span><span class="x_me-email-text-secondary">&nbsp\;Pass code: </span><span class="x_me-email-text">jM8w7c</span></div>\n<div><hr>< /div>\n<div>\n<div><span class="x_me-email-text">Dial-in by phone</span></ div>\n<div><a id="x_meet_invite_block.action.join_phone_number1" class="x_ me-email-link" href="tel:+15713972084\,\,218888141#" target="_blank" rel=" noopener noreferrer" data-auth="NotApplicable" data-linkindex="2">+1 571-3 97-2084\,\,218888141#</a>&nbsp\;<span class="x_me-email-text">United State s\, Arlington</span></div>\n<div><a id="x_meet_invite_block.action.join_ph one_find_local_number" class="x_me-email-link" title="Original URL: https: //dialin.teams.microsoft.com/9424c9fe-3b57-41d6-9131-1d3b9b7cf4a9?id=21888 8141. Click or tap if you trust this link." href="https://nam11.safelinks. protection.outlook.com/?url=https%3A%2F%2Fdialin.teams.microsoft.com%2F942 4c9fe-3b57-41d6-9131-1d3b9b7cf4a9%3Fid%3D218888141&amp\;data=05%7C02%7Ckha ssan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e857255df574c47a0c00 546460380cb%7C0%7C0%7C638501649141626316%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM C4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp \;sdata=r6YY83JURqPLJf9J2QcdQ5H6w3QBZTlOs4HAUUBvvSQ%3D&amp\;reserved=0" ta rget="_blank" rel="noopener noreferrer" data-auth="Verified" data-linkinde x="3">Find a local number</a></div>\n</div>\n<div><span class="x_me-email- text-secondary">Phone conference ID:&nbsp\;</span><span class="x_me-email- text">218 888 141#</span></div>\n<div><span class="x_me-email-text-seconda ry">For organizers:&nbsp\;</span><a id="x_meet_invite_block.action.organiz er_meet_options" class="x_me-email-link" title="Original URL: https://team s.microsoft.com/meetingOptions/?organizerId=f9586db0-74ee-4635-a01b-2383b7 4f8a0c&amp\;tenantId=9e857255-df57-4c47-a0c0-0546460380cb&amp\;threadId=19 _meeting_OGUwZGI0OTktNTdhZS00NmNlLWEzZGEtZTJhNGI2Yjg5YmJi@thread.v2&amp\;m essageId=0&amp\;language=en-US. Click or tap if you trust this link." href ="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fteams. microsoft.com%2FmeetingOptions%2F%3ForganizerId%3Df9586db0-74ee-4635-a01b- 2383b74f8a0c%26tenantId%3D9e857255-df57-4c47-a0c0-0546460380cb%26threadId% 3D19_meeting_OGUwZGI0OTktNTdhZS00NmNlLWEzZGEtZTJhNGI2Yjg5YmJi%40thread.v2% 26messageId%3D0%26language%3Den-US&amp\;data=05%7C02%7Ckhassan1%40gmu.edu% 7Cc082474ca0814691061508dc69ddedfd%7C9e857255df574c47a0c00546460380cb%7C0% 7C0%7C638501649141633701%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp\;sdata=KSJGsLYD 3jKk7GBpn2XuSPetsCGuvQn9I6sJYXPZfs0%3D&amp\;reserved=0" target="_blank" re l="noreferrer noopener" data-auth="Verified" data-linkindex="4">Meeting op tions</a>&nbsp\;|&nbsp\;<a id="x_meet_invite_block.action.organizer_reset_ dialin_pin" class="x_me-email-link" title="Original URL: https://dialin.te ams.microsoft.com/usp/pstnconferencing. Click or tap if you trust this lin k." href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F% 2Fdialin.teams.microsoft.com%2Fusp%2Fpstnconferencing&amp\;data=05%7C02%7C khassan1%40gmu.edu%7Cc082474ca0814691061508dc69ddedfd%7C9e857255df574c47a0 c00546460380cb%7C0%7C0%7C638501649141641487%7CUnknown%7CTWFpbGZsb3d8eyJWIj oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C& amp\;sdata=2ONDG7yjY3RckniAVPkRkliPHOBWdNqIhM2RgAA8O10%3D&amp\;reserved=0" target="_blank" rel="noreferrer noopener" data-auth="Verified" data-linki ndex="5">Reset dial-in PIN</a></div>\n<p class="x_MsoNormal"><strong>Abstr act:</strong></p>\n<p class="x_MsoNormal">Language models\, and in particu lar generative and conversational language models\, are at the heart of re cent advances in natural language processing (NLP). Understanding how thes e models represent textual content and how they learn these representation s still raises multiple research questions. In this talk\, I will start fr om an observation that small models are less efficient than expected. I wi ll show that language models relying on the Transformer architecture tend to produce vector representations that are not isotropically distributed i n space. This anisotropy is linked to the way in which these models are le arned\, which leads to the frequency of the tokens taking a preponderant p lace in their representation. I will show that this effect has negative co nsequences on the ability of small models to train satisfactorily (&ldquo\ ;performance saturation&rdquo\;) but does not seem to affect larger models . I will then describe a new approach for training language models intende d to avoid the undesirable effects of this prevalence of frequency informa tion. The resulting &ldquo\;headless&rdquo\; models display a number of ad vantages over standard models\, including on downstream performance.</p>\n <p class="x_MsoNormal"><strong>Bio:</strong>&nbsp\;</p>\n<p class="x_MsoNo rmal">Beno&icirc\;t Sagot is a computer scientist specialized in natural l anguage processing (NLP). He is a Senior Researcher (Directeur de Recherch es) at INRIA\, where is heads the INRIA research project ALMAnaCH in Paris \, France. He also holds a chair in the PRAIRIE institute dedicated to art ificial intelligence\, and currently holds the annual chair for computer s cience in the Coll&egrave\;ge de France. His research focuses on language modelling\, machine translation\, language resource development and comput ational linguistics\, with a focus on French in all its form and on less-r esourced languages.</p>\n</div>\n</div>\n<p>&nbsp\;</p>\n<div class="x_me- email-text">\n<div>_______________________________________________________ _________________________</div>\n</div> END:VEVENT END:VCALENDAR