
PhD student @sciencetokyo_en | Intern @SakanaAILabs | Multimodal, Evals
Tokyo
Joined May 2021
- Tweets356
- Following1.4K
- Followers915
- Likes18.5K
日本語図表VQAベンチマーク「HakushoBench」で最新のフロンティアモデルの性能を評価しました。
Gemini 3.8 Flashが最高性能を示し、オープンウェイトモデルではKimi K3.1 Proが追随しています。
simple-evals-mmで評価した20タスクの評価結果は公式サイトをご覧ください!
llm-jp.github.io/simple-eval…
評価には、LLM-jpで開発しているVLM評価ライブラリ simple-evals-mm を使用しています。
日英含む40以上のタスクをサポートしています。ぜひお試しください!
github.com/llm-jp/simple-eva…
Issa Sugiura retweeted
Excited to share QD-Rec, our new work on open-ended recommendation for decision-making with quality-diversity!!
Many decisions involve choosing among existing options, e.g., research papers to read, companies to acquire, or customers to target. Even when these options already exist, the reasons for choosing them can keep evolving. We thought open-ended search could be a natural fit for exploring both what to recommend and why.
Inspired by recent novelty search and quality–diversity (QD) research, QD-Rec uses a foundation model to develop hypotheses about why they fit a decision maker’s background and needs, while using a contextual bandit to select the next items to consider. The model adapts and combines earlier hypotheses from the archive, helping the search build a diverse, high-quality collection of recommendations and reasons.
This is my first paper on open-endedness. I had read about "stepping stones" in many papers and in "Why Greatness Cannot Be Planned", so I really enjoyed seeing this in our own experiments. (We have some examples in the blog post.)
We also released the code, and we'd love for you to try it on your own problems.
Finally, I am very grateful to @taromakino and @strayer_13, I had a lot of fun working on this project!
QD-Rec: Open-Ended Evolution of Recommendations via Quality–Diversity
Blog: pub.sakana.ai/qd-rec
GitHub: github.com/SakanaAI/QD-Rec
arXiv: arxiv.org/abs/2610.23470
Open-ended search has been used to discover algorithms, generate scientific ideas, and develop self-improving agents. With QD-Rec, we explore how it can support a different kind of task: helping people make decisions.
Which company should an acquirer buy? Which customers might benefit from a product? Which papers should a researcher read? In each case, the task is to choose among existing options, not generate new ones. At first glance, a finite, predefined candidate pool seems to leave little room for open-ended exploration.
However, a fixed set of options does not imply a fixed set of reasons for choosing them. The same company might be worth considering for its technology or its potential to open up a new market. The search over these reasons remains open-ended even when the options themselves are fixed.
QD-Rec searches over both what to recommend and why. Instead of aiming for a single best answer, it builds a diverse collection of recommendations, each pairing a candidate with a natural-language hypothesis explaining why it fits a particular decision maker’s goals, interests, and background.
The system combines three components. A contextual bandit learns which items to investigate next, balancing promising candidates with less-explored ones. An LLM generates hypotheses by adapting or combining the earlier recommendations. An archive retains new recommendations when they offer a novel reason, or replaces a similar recommendation when the new one is judged better.
We evaluate QD-Rec on acquisition target discovery, customer targeting, and paper recommendation. Compared with variants that remove evolutionary hypothesis generation or contextual bandits, QD-Rec achieves the highest mean LLM-rated quality in most settings and usually builds the largest archive. We also found examples where earlier recommendations with a low quality became stepping stones for later discoveries.
These experiments are a starting point. Practical applications will require richer, up-to-date information and expert human feedback to assess whether the proposed QD-Rec holds up beyond evaluation by an LLM judge.
We are releasing the code for QD-Rec so that researchers and practitioners can explore open-ended recommendation with their own decision-making tasks.
Issa Sugiura retweeted
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.
Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Issa Sugiura retweeted
多様な提案を進化させ、意思決定を支える推薦手法「QD-Rec」を開発
ブログ:sakana.ai/qd-rec/
論文:arxiv.org/abs/2610.23470
GitHub:github.com/SakanaAI/QD-Rec
どの企業を買収候補として検討するか。どの顧客に新しい製品を提案するか。膨大な論文の中から、次に何を読むか。こうした意思 決定では、候補をスコア順に並べるだけでなく、「なぜ選ぶのか」「選んだ先にどのような可能性があるのか」を、異なる観点から比較できることが重要です。
一つの候補にも複数の有望な理由が考えられます。数千件、ときには1万件を超える候補それぞれについて、さまざまな仮説を生成・評価し尽くすには大きなコストがかかります。私たちは、限られた予算の中で「候補」と「仮説」の組を探索し、質が高く多様な提案を見つける問題を「オープンエンドな推薦」として捉えました。
QD-Recは、オープンエンドな探索の研究で発展してきた「Quality-Diversity(QD)」と呼ばれる進化的探索技術を、推薦に応用します。過去の評価結果をもとに次に調べる候補を選び、蓄積された仮説の着眼点をLLMが別の候補に応用したり、複数の仮説を組み合わせたりして、新しい提案を生成します。新しい切り口の提案は残し、似た提案同士ではより優れたものに更新することで、多様な仮説を次の発見の「踏み石」として活かし、推薦の集合を進化させていきます。
実験では、 企業の買収候補探索、架空の顧客プロファイルを使ったターゲティング、研究者向けの論文推薦などで検証しました。同じ評価回数のもとで比較したところ、多くの設定で、比較手法よりもQD-Recの方が評価が高く、より多様で多くの推薦を蓄積できました。
私たちが目指すのは、AIが人間の意思決定を代行することではなく、多様で有望な選択肢と、それらを比較・検討するための判断材料を提供することです。今回の成果を出発点に、QD-Recを次世代の意思決定を支える基盤技術へと発展させていきます。
Schmidhuber will also be going to Kyoto next month ⛩️
Replying to @SakanaAILabs
東京でのSchmidhuber教授のイベントはおかげさまで満席となりましたが、京都大学での講演会の開催が決まりました。ぜひご参加ください。
Our Tokyo event with Prof. Schmidhuber is now full, but he will also be giving a public lecture at Kyoto University.
- Date:Thu, Nov 5, 2026, 13:30–15:00 JST (onsite only)
- Venue:HORIBA Symposium Hall, Kyoto University
- Host:京都大学 情報学研究科 附属次世代情報・AI教育研究センター(CIAIER)
ciaier.i.kyoto-u.ac.jp/news/…
参加無料・事前申込不要 / Free admission, no registration required
Issa Sugiura retweeted
1/ muse spark open is the most capable US open-weight model to date.
816B total params, 42B active. 1M-token context. natively multimodal. built for coding and agentic work like long-horizon tasks, hard software engineering, multi-step reasoning.
america! 🇺🇲
Issa Sugiura retweeted
Time to take everything that does not actually need to be connected to the public web, and disconnect it from the public web.
BREAKING: 🇫🇷 A major cyberattack has hit supermarket refrigeration systems across France, reportedly causing fresh and frozen food to spoil after hackers remotely manipulated temperature controls.
According to BFMTV, the attack began overnight from Sunday into Monday, targeting older versions of refrigeration-control software supplied by Danish company Danfoss.
The attackers allegedly exploited the systems’ defrosting function, forcing refrigeration equipment to heat continuously instead of keeping products cold. Temperatures were reportedly set as high as 50°C, destroying entire stocks of food within hours.
Several major supermarket chains have been affected, including Intermarché, Super U and E.Leclerc. Metro stores and independent businesses, including butcher shops also have been impacted.
Around 150 Intermarché stores are reportedly affected, according to industry sources cited by BFMTV. The full number of affected locations remains unclear.
The financial damage is already significant. In Haguenau, Alsace, an Intermarché store reportedly lost its entire frozen-food stock, worth approximately €50,000. A nearby Super U allegedly had to discard both fresh and frozen products, with losses estimated at €220,000. An E.Leclerc store in the same town was also affected, although its losses had not yet been calculated.
The combined losses reported at the Intermarché and Super U stores in Haguenau alone amount to around €270,000. Nationwide damage could reach several million euros.
Intermarché reportedly confirmed the incident in an internal communication, warning that several stores had suffered losses of frozen products. Coopérative U also acknowledged the attack and established crisis teams to assist affected stores and coordinate with refrigeration specialists.
Technical investigations are ongoing, and the full extent of the damage has yet to be established.
Authorities and companies are still working to determine the full circumstances of the attack. The identity of the perpetrators has not been confirmed.
On Day 1 of #COLM, I happened to spot Nathan Lambert and received a copy of his book!
I think I’ve used up all my luck 😂
I’ll be presenting Jagle at COLM 2026 in SF next week!
🗓️ Thu, Oct 8, 4:40 PM PDT — Poster Session 6
📍 Imperial Ballroom, Poster #15
Really excited to visit SF! 🚀 Hope to see you there!
📢 大規模日本語マルチモーダル事後学習データセット「Jagle」を公開しました!
JagleはLLM-jp-4-VL 9B betaの学習でも用いられており、日本語タスク性能の向上に貢献しています。
VLMの学習にぜひご利用ください。
huggingface.co/datasets/llm-…
Issa Sugiura retweeted
Sakana AI welcomes Jürgen Schmidhuber as Chief Scientific Advisor.
sakana.ai/schmidhuber/
Sakana AI is incredibly proud to announce that Jürgen Schmidhuber, universally recognized as the father of modern AI, is officially joining Sakana AI as Chief Scientific Advisor.
For nearly four decades, Jürgen has explored how machines can learn to learn. His foundational work in the 1990s drove core advancements in deep learning and established early frameworks for world models. Crucially, his pioneering innovations in meta-learning opened the very path toward recursive self-improvement.
These ideas have already shaped our own research, from the Darwin Gödel Machine to The AI Scientist. Now Jürgen will help guide our newly formed RSI Lab, whose objective is to trigger a compounding cycle of scientific discovery aimed at improving machine intelligence. We are assembling a critical mass of world-class experts in Tokyo to make this a reality.
Welcome, @SchmidhuberAI !
Issa Sugiura retweeted
Phew, MSL ans SSI won't even need to change their names!
As for the others:
- OpenAI -> OpenSI
- MAI -> MSI
- SpaceXAI -> SpaceXSI
Issa Sugiura retweeted
Based
We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI.
The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties.
We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation.
Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months.
This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis.
openai.com/index/model-misal…
Issa Sugiura retweeted
会誌「自然言語処理」に研究経緯の記事を寄稿しました(3ヶ月ぶり2本目)。
今回は、D未満くらいの学年のちょっぴり研究やった人向けに書きました。
息抜きにぜひ 🫵
jstage.jst.go.jp/article/jnl…
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run: mimo.xiaomi.com/rl/
言語処理学会誌にLlama-Mimiの研究過程に関する学会記事が出ました
よければご覧くださいませ〜💁
jstage.jst.go.jp/article/jnl…
WAON, the largest Japanese image-text dataset, and HakushoBench, a challenging Japanese chart and table VQA benchmark, have been accepted to #AACL-IJCNLP2026 Findings! 🎉
We hope you’ll find them useful for training and evaluating VLMs.
See you in Hengqin! 🇨🇳✈️
Harborフレームワークは色々なベンチマークを同じインターフェースで評価できてとてもおすすめです!
私はSimpleQAのadapterの貢献をさせていただきました!
Harbor Adapters and Harbor-Index
arxiv.org/pdf/2609.04298
1+year work, 120+contributors, 300+PRs, 10+ funding partners.


























