If it weren’t for DeepSeek, they would still be feeding us incremental improvements on benchmarks by increasing by 1 percentage point one after another for 5 more years. Now they had to reveal all they had and give it for free. DeepSeek saved us 5 years. Thank you, @deepseek_ai.

🚀 #InductionBench Super difficult benchmark alert! Even o3-mini achieve 5% accuracy! LLM reasoning benchmarks have long been focused on math, code, and knowledge domains. But we’ve actually missed a huge area -- inductive reasoning: ✨ from limited data, can you generate the simplest underlying function? ✨ a.k.a. the minimum-length description, which is also a white-box explanation. 🤔🤔It’s also what most data analysts and scientific researchers care about: given limited data, how do we discover the underlying principle in limited time? ✨📊🔑 Truth is always simple. 📐 Previously, there were indeed some inductive reasoning benchmarks (like concept learning, etc.), but the entire field of inductive reasoning actually has very strict mathematical definitions: 1⃣ what class of function are we dealing with 2⃣ is there a unique optimal solution 3⃣ how big is the search space 4⃣ how much data is needed 5⃣ and in how much time can we find the shortest description? This is the central idea of my article: In a strictly defined inductive function class, how capable are LLMs? We chose the simplest function class for inductive reasoning: regular functions/transducers, and within the subregular hierarchy of regular transducers, we picked the subregular classes with the lowest complexity: ISL, L-OSL, and R-OSL. According to existing research and proofs, these three non-comparable classes can yield a unique shortest description with polynomial data and polynomial computation time. 😱 I selected several very simple settings as target functions in these classes, and o3-mini’s performance was only 5% success rate. 🚧❌In other words, current LLMs have a significant flaw in inductive summarization, and this flaw makes me feel that their possibilities in #AI4Sci are greatly reduced. Moreover, few-shot learning doesn’t help much with this type of reasoning, and the more data there is, the harder it becomes (after ruling out context length issues). 🤖🔥🏆 Of course, given that these three function classes have already been thoroughly researched, to avoid people using existing algorithms to “cheat,” I created an exploration version of the benchmark. From the definitions, this function class only looks slightly more complicated than those three function classes. But as of now, there’s no mature research, no proof, no algorithm for it. 🏆🏆So if an LLM can succeed on the exploration benchmark, that would truly qualify as superhuman! arxiv：arxiv.org/pdf/2502.15823 GitHub：github.com/Wenyueh/inductive… #LLM #InductiveReasoning #MachineLearning #DataAnalysis #Benchmark #RegularTransducer #ISL #LOSL #ROSL #Ai4Sci #HuggingFace #SuperhumanPotential

Google presents PlanGEN for complex planning and reasoning. PlanGEN is a multi-agent framework designed to enhance planning and reasoning in LLMs through constraint-guided iterative verification and adaptive algorithm selection. Key insights include: Constraint-Guided Verification for Planning – PlanGEN integrates three agents: (1) a constraint agent that extracts problem-specific constraints, (2) a verification agent that evaluates plan quality and assigns scores, and (3) a selection agent that dynamically chooses the best inference algorithm based on instance complexity. Improving Inference-Time Algorithms – PlanGEN enhances existing reasoning frameworks like Best of N, Tree-of-Thought (ToT), and REBASE by iteratively refining outputs through constraint validation. Adaptive Algorithm Selection – Using a modified Upper Confidence Bound (UCB) policy, the selection agent optimally assigns problem instances to inference algorithms based on performance history and complexity. State-of-the-Art Performance – PlanGEN achieves +8% improvement on NATURAL PLAN, +4% on OlympiadBench, +7% on DocFinQA, and +1% on GPQA, surpassing standard multi-agent baselines.

Want to build R1-like multimodal models but unable to find suitable reasoning data for training/tuning? Meet our preliminarily released 🔥VL-Thinking🤔: An R1-Derived Vision-Language Dataset for Thinkable LVLMs 🔥 Specifically, we use a 4-step process to infuse R1-style reasoning in building high-quality multimodal reasoning dataset at scale: 1️⃣ GPT-4o crafts detailed image captions 🖼️ 2️⃣ R1 processes these captions + questions to generate answers 🤖 3️⃣ GPT-4o refines the reasoning for natural fluency ✍️ 4️⃣ GPT-3.5-Turbo verifies answers against ground truth ✅ The result? 76,469 high-quality vision-language instances ready for training! Access the dataset now: ucsc-vlaa.github.io/VL-Think…

Are expensive labeled data and rejection sampling truly necessary for developing self-improving reasoning models? Introducing Unsupervised Prefix Fine-Tuning (UPFT) -- an efficient method that trains models on only the first 8-32 tokens of single self-generated solutions, exploiting Prefix Self-Consistency across different reasoning trajectories. 🪡 We identify Prefix Self-Consistency as a critical phenomenon, showing that early reasoning steps are highly consistent across trajectories, enabling efficient self-improvement learning without supervision. 🪡 We propose an unsupervised fine-tuning method UPFT that leverages only prefix substrings (i.e., minimal initial tokens of model-generated responses), eliminating the need for labeled data or rejection sampling. 🪡 We conduct comprehensive empirical validation to demonstrate that UPFT exhibits exceptional data efficiency and versatility, outperforming vanilla full-token fine-tuning in unsupervised settings and achieving competitive performance with rejection sampling fine-tuning while drastically reducing sampling and tuning overhead. Paper: dx.doi.org/10.13140/RG.2.2.3… 🧵