Expert Tech Talks - Crawling the web for PDF files: The challenge of scaling from thousands to millions.

#DistributedSystems #WebCrawling #OpenAccessResearch #PDFMining #Virtualization #QEMU #HighPerformanceComputing #CloudInfrastructure #DataEngineering #ScalableArchitecture
Share

Special Presentation by Charles Mutigwe

Hosted by the Expert Tech Talks (IEEE Philadelphia Section)

Date/Time: Monday, 2 November 2026 @ 7 PM Eastern Time

Topic:

Crawling the web for PDF files: The challenge of scaling from thousands to millions.

Abstract:

In this talk I will discuss the technical challenges of designing a distributed system that crawls the web for open-source PDF research articles. Beginning with a list of over 60 million URLs from OpenAlex, we proceed to spawn 454 QEMU virtual machines across 3 servers on a Lambda cluster with a SAN from VAST Data. Each virtual machine runs a pdf crawler. The verification of the URLs, physical storage of the downloaded files, orchestration of the VMs, throttling of each crawler, network resources, logging and tracking. This is an on-going project where these crawlers run for months and so far, we’ve visited over 24 million URLs and collected over 10 million PDF research articles taking up almost 50TB of storage.

 

Speaker:

 

Dr. Charles Mutigwe is an Associate Professor of Business Analytics & Information Management at Western New England University. He holds a D.Tech in Electrical Engineering from the Central University of Technology (South Africa), an MBA from Norwich University, a M.S. in Electrical Engineering from Western New England University, and a B.S. in Electrical Engineering from the University of Zimbabwe. He also holds an Applied AI Professional certificate from IBM and a FinTech certificate from the University of Hong Kong. Dr. Mutigwe has more than 25 years of industry experience in the IT field. He has had roles as a network engineer, systems administrator, systems engineer, systems developer, business intelligence consultant, and IT manager. Part of his work experience has been in the financial services industry. From 2001 to 2004 he worked as systems engineer for Pitney Bowes in Danbury, CT, where he was responsible for implementing online bill payment platforms at client sites. His clients included the Royal Mail (UK), DTE Energy, Auto Owners Insurance, ESL Federal Credit Union, South Central Connecticut Regional Water Authority, and Aquarion Water Company. From 2007 to 2010 he worked as a systems developer for Berkshire Bank in Pittsfield, MA where he was responsible for developing reporting, data analytics, and new customer in-house referrals tools.
In his current role as an educator and researcher, Dr. Mutigwe teaches courses in business analytics, enterprise systems & business processes, strategic information management, database management systems, social media analytics, and artificial intelligence for managers. He is actively engaged in three areas of FinTech research. The first is using deep learning natural language models on financial markets-related data from social media platforms, such as Twitter and Reddit. The second area is applying different anomaly detection techniques for fraud detection. The last area of research is studying the potential benefits and challenges of open finance from both the technical and social perspectives with a focus on open banking, open wallets, and central bank digital currencies.
Dr. Mutigwe leads the WNE FinTech & AI Incubator at Western New England University, as well as the associated FinTech Lab. In this role he helps launch fintech and AI startups and oversees research.

 



  Date and Time

  Location

  Hosts

  Registration



  • Add_To_Calendar_icon Add Event to Calendar

Loading virtual attendance info...

  • Contact Event Host
  • Co-sponsored by IEEE Future Networks AI/ML Working Group