Pinned Tweet
Didn't expect to see LEGATO in the GPT-6 report 🤯🎼
GPT-6 Astra cites our work — and even uses the LEGATO camera subset for its OMR eval.
We’ve since pushed it further with LEGATO 2:
arxiv.org/abs/2607.05769
Can AI turn a picture of sheet music into editable, searchable, playable music? 🎼
Excited to share that our paper LEGATO was accepted to ICLR 2026!
LEGATO is an OMR model that converts sheet music images into machine-readable music notation.
Demo: huggingface.co/spaces/guangy…
LEGATO is designed for realistic typeset scores:
• full-page scores
• multi-page documents
• diverse notation styles
• end-to-end recognition
• ABC notation output
Music notation is not ordinary visual text.
It has tiny symbols, dense layouts, multiple voices, rhythm, pitch, measures, ties, beams, and long-range structure.
That’s why OMR needs models built specifically for music notation.
OMR matters because so much music still lives as PDFs, scans, and images.
If we can convert those pages into symbolic music, scores become easier to search, edit, analyze, transpose, and play back.
General-purpose AI models like GPT/Gemini still struggle with OMR. LEGATO makes a big leap:
• 68% TEDn error reduction
• 47.6% OMR-NED error reduction
Paper: arxiv.org/abs/2506.19065
Audio & music evals in multimodal generation are tough—noisy metrics, vague correctness. 🎧😵💫
Our new work MMMG improves this with clear tasks, reliable metrics, and human-aligned judgments. 💯 If you’re working on audio/music in multimodal models, this benchmark is a must-try!
We introduce MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask Multimodal Generation
✅ Reliable: 94.3% agreement with human judgment
✅ Comprehensive: 4 modality combination × 49 tasks × 937 instructions
🔍Results and Takeaways:
> GPT-Image-1 from @OpenAI leads image generation at 78.3% accuracy—13.7% ahead of the next-best model. The top open-source model, BAGEL from #ByteDance , achieves 45.5% accuracy.
> Audio generation is still challenging: Top open-sourced models achieve only 48.7% accuracy in sound (Make-An-Audio 2 from #ByteDance) and 41.9% in music (MusicGen from @AIatMeta).
📜 Paper: arxiv.org/abs/2505.17613v1
🛠️ Code and Evaluation Suite: github.com/yaojh18/MMMG
🥇Leaderboard: yaojh18.github.io/mmmg-leade…
🧵1/N
