Expert Tech Talks - Crawling the web for PDF files: The challenge of scaling from thousands to millions.
Special Presentation by Charles Mutigwe
Hosted by the Expert Tech Talks (IEEE Philadelphia Section)
Date/Time: Monday, 2 November 2026 @ 7 PM Eastern Time
Topic:
Crawling the web for PDF files: The challenge of scaling from thousands to millions.
Abstract:
In this talk I will discuss the technical challenges of designing a distributed system that crawls the web for open-source PDF research articles. Beginning with a list of over 60 million URLs from OpenAlex, we proceed to spawn 454 QEMU virtual machines across 3 servers on a Lambda cluster with a SAN from VAST Data. Each virtual machine runs a pdf crawler. The verification of the URLs, physical storage of the downloaded files, orchestration of the VMs, throttling of each crawler, network resources, logging and tracking. This is an on-going project where these crawlers run for months and so far, we’ve visited over 24 million URLs and collected over 10 million PDF research articles taking up almost 50TB of storage.
Speaker:
|
Dr. Charles Mutigwe is an Associate Professor of Business Analytics & Information Management at Western New England University. He holds a D.Tech in Electrical Engineering from the Central University of Technology (South Africa), an MBA from Norwich University, a M.S. in Electrical Engineering from Western New England University, and a B.S. in Electrical Engineering from the University of Zimbabwe. He also holds an Applied AI Professional certificate from IBM and a FinTech certificate from the University of Hong Kong. Dr. Mutigwe has more than 25 years of industry experience in the IT field. He has had roles as a network engineer, systems administrator, systems engineer, systems developer, business intelligence consultant, and IT manager. Part of his work experience has been in the financial services industry. From 2001 to 2004 he worked as systems engineer for Pitney Bowes in Danbury, CT, where he was responsible for implementing online bill payment platforms at client sites. His clients included the Royal Mail (UK), DTE Energy, Auto Owners Insurance, ESL Federal Credit Union, South Central Connecticut Regional Water Authority, and Aquarion Water Company. From 2007 to 2010 he worked as a systems developer for Berkshire Bank in Pittsfield, MA where he was responsible for developing reporting, data analytics, and new customer in-house referrals tools. |
Date and Time
Location
Hosts
Registration
-
Add Event to Calendar
Loading virtual attendance info...
- Contact Event Host
- Co-sponsored by IEEE Future Networks AI/ML Working Group