Prof. Saman Halgamuwa

Large Language Models (LLMs) have transformed the field of artificial intelligence. Their extension to computer vision has led to the emergence of Vision-Language Models (VLMs), paving the way for the next generation of Physical AI through Language-Action Models (LAMs) and Vision-Language-Action (VLA) models. These models integrate visual and linguistic information to generate concrete actions for autonomous agents and machines including robotic systems.

VLA models are fundamentally changing robotics. Rather than executing pre-programmed sequences of operations in structured environments, VLA-enabled robots are expected to interpret instructions, reason using external knowledge, adapt to unfamiliar situations, and perform tasks in previously unseen environments. Concurrent advances in world models, agentic AI, reinforcement learning, and foundation models are accelerating the development of intelligent robots capable of autonomous action.

The assumption that VLAs can fully represent the physical world using vision, language, and sensor inputs encoded as tokens is, at present, optimistic. Although these multimodal inputs can be fused through cross-attention mechanisms and transformer architectures can generate action sequences, there remain open research challenges.

This talk will examine the current state of Physical AI, the scientific and engineering challenges that remain, and the opportunities for deploying intelligent machines in complex, dynamic, and unstructured real-world environments.

One promising avenue we pursue in our research group is the development of richer world models that integrate scientific knowledge with data-driven learning. Physics-Informed Neural Networks (PINNs) incorporate physical laws, expressed as differential equations, into machine learning models. However, a comprehensive world model should extend beyond physics and training data and the corpora of general text it was trained on (if the backbone used is an LLM). Knowledge is also encoded in subject-specific textual descriptions, knowledge graphs and other structured representations. To build more robust and generalisable world models, we should exploit all available forms of prior knowledge. We have developed a new framework that enriches parameter discovery of model equations by integrating auxiliary knowledge sources alongside observational data [1,2]..

A second challenge we pursue is the integration of continuous time-series data into VLA models. Industrial robots operate in environments rich in continuous sensor signals, including force, vibration, current, temperature, and motion measurements. However, most LLMs and VLMs are pretrained on discrete modalities such as text and images, making their adaptation to continuous temporal signals non-trivial. To address this gap, we propose a temporal modulation framework that fine-tunes LLMs and VLMs to effectively process time-series inputs alongside images and text, enabling VLA models to capture complex temporal dynamics and make more informed decisions in real-world robotic applications [3].

1.     Knowledge Inclusive Machine Learning for Disease Gene Prioritisation, CJ Gamage, Y Xia, R Rupasinghe, S Senevirathne, D Senanayake, ...S. Halgamuge, bioRxiv, 2026.04. 29.721522, 2026

2.     Knowledge Inclusive Adaptive Physics-Informed Neural Network for Microbial Interaction Modelling

R Rupasinghe, R Vidanaarachchi, A Hevapathige, S Seneviratne, S. L. Tang, S. Halgamuge, arXiv preprint arXiv:2606.07686, 2026

3.     NeST: Neighborhood-aware semantic alignment and temporal modulation for LLM based time series forecasting, J Bogahawatte, S Seneviratne, M Perera, S Halgamuge, arXiv preprint arXiv:2412.04806, 2026

Biography:

Prof Saman Halgamuge, Fellow of IEEE, IET, AAIA and NASSL, is a Professor at The University of Melbourne. He previously served as Head of the School of Engineering at the Australian National University, Associate Dean of the Faculty of Engineering and Information Technology at The University of Melbourne, and a member of the Australian Research Council (ARC) grant assessment panel.

Prof Halgamuge received his Dipl.-Ing. and Ph.D. degrees in Data Engineering from the Technical University of Darmstadt, Germany. He is recognised among the top 2% of the world’s most-cited researchers in Artificial Intelligence and Image Processing in the Stanford/Elsevier database. In 2026, he was recognised as a Distinguished Contributor of the IEEE Computer Society. He is appointed by IEEE Computer Society Distinguished Visitor (2024–2026) and served as an IEEE Computational Intelligence Society Distinguished Lecturer (2018–2021).

His research has been supported by major national and international funding agencies, including the Australian Research Council, National Health and Medical Research Council, the US Department of Defence Biomedical Research Program, and leading industry partners such as Bosch Germany and Google US. His interdisciplinary research contributions span artificial intelligence, machine learning, optimisation, and biomedical applications.

Prof Halgamuge has supervised more than 50 PhD graduates and delivered over 60 keynote and invited presentations at international conferences worldwide. His research publications are available through his Google Scholar profile:

https://scholar.google.com.au/citations?hl=en&user=9cafqywAAAAJ&pagesize=80&view_op=list_works&sortby=pubdate

Address:Australia

Prof. Todor Ganchev

Address:Bulgaria