🛠️ To address this gap, we:
1. Identify pitfalls and define core requirements for robust VLM evaluation.
2. Introduce SURE-VQA — an open-source framework for systematic robustness evaluation.
3. Validate it empirically and benchmark the robustness of various fine-tuning methods.
1
1
57
📊 With our large-scale robustness study we find that:
1. No fine-tuning method consistently outperforms others in robustness
2. Robustness varies more by shift than by method
3. Simple image-free baselines perform surprisingly well and LoRA performs best on in-distribution data
1
45
Special thanks to my co-first author Kim-Celine Kahl (@kc_kahl) for the valuable contribution to this project to push research in this field!
Also thanks to the collaborators in this project - @jeremiastraub , @CarTLueth, @lena_maierhein, @maierhein and @pfjaeger.
2
46
📣 I'm excited to share that our paper "SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks" has been accepted to #TMLR !
📃 Read our paper: openreview.net/pdf?id=qjNdGp…
💻 Check out the code: github.com/IML-DKFZ/sure-vqa
More ⬇️
🔍 TLDR: We reveal gaps in current robustness evaluations of medical vision-language models and introduce SURE-VQA, a framework addressing them via real-world shifts, improved metrics, and sanity baselines—validated with a large study across 3 datasets and 4 distribution shifts.
1
40
Selen Erkan retweeted
We’re thrilled to welcome Eric Brachmann (@eric_brachmann), Senior Staff Scientist at Niantic Spatial, Inc. and a leading expert in visual relocalisation & pose estimation, to our heidelberg.ai / NCT Data Science Seminar on July 16th at 4 PM.
👇Details Below 1/3
Selen Erkan retweeted
In case you missed the last heidelberg.ai talky by Prof. Yuki Asano (@y_m_asano) on "Post-Pretraining in Vision, and Language Foundation Models", it is just released on the heidelberg.ai Youtube Channel: youtube.com/watch?v=5UTCjQKB…




