ViTCap-R: Unifying Image Captioning and Cross-Modal Retrieval with Vision Transformers and Patch-Level Attention

#transformers #machine-learning #attention #captioning
Share

Abstract: Generating accurate image captions and retrieving semantically matched image-text pairs are typically treated as independent problems, limiting the depth of visual-semantic understanding a model can develop. This talk presents ViTCap-R, a unified framework that integrates a Vision Transformer encoder, a patch-level attention LSTM decoder, and a dual-encoder contrastive retrieval module into a single cohesive pipeline. By attending over 196 spatial patch embeddings during decoding, the model produces interpretable captions grounded in specific image regions. A shared 256-dimensional embedding space, trained with symmetric InfoNCE loss and hard negative mining, enables effective cross-modal retrieval and caption reranking at inference time.

We evaluate ViTCap-R on Flickr8k and MS COCO 2014, reporting BLEU, METEOR, and Recall@K metrics alongside qualitative attention heatmaps, PCA projections, and t-SNE visualizations of the learned embedding space. Results demonstrate that unified training improves semantic alignment between modalities, while patch-level attention offers meaningful interpretability gains over standard global-feature baselines. We also discuss the challenges of scaling to high-diversity datasets and the trade-offs introduced by hard negative mining, pointing toward future directions involving transformer-based decoders and larger-scale pretraining.

 


  Date and Time

  Location

  Hosts

  Registration


  • Add_To_Calendar_icon Add Event to Calendar

Loading virtual attendance info...

  • 2 Civic Center Drive
  • East Brunswick, New Jersey
  • United States 08816
  • Room Number: Meeting Rooms 2,3
  • Contact Event Host
  • Starts 08 May 2026 04:00 AM UTC
  • Ends 18 July 2026 04:00 AM UTC
  • No Admission Charge

  Speakers

Prayash Das

Biography:

Prayash Das is a Machine Learning Engineer specializing in large language models, retrieval-augmented generation (RAG), and end-to-end AI system design. With a Master’s in Applied Artificial Intelligence from Stevens Institute of Technology, he has built scalable, production-ready AI solutions spanning intelligent search, career matching systems, and conversational agents. His work combines deep learning, vector databases, and modern LLM frameworks to deliver high-impact, real-world applications. Passionate about bridging research and product, Prayash focuses on developing intelligent, user-centric systems that drive personalization, automation, and decision-making at scale.




Agenda

Hybrid event, in-person or online:
2:30 Introduction
Technical Talks
Question and Answer
Networking
4:30 Conclusion