Pyuyi

Hi, there! I’m Pyuyi, mmm as well (she/her). This account is for academic sharing. What I mainly post here: 1. Paper notes and thoughts on AI interpretability, causal inference, math and learning theory. 2. Summaries and discussions of recent work from venues — including causal model reduction, RL interpretability, topology & replicability, latent reasoning as dynamical systems, etc. 3.Occasional updates on my own related work and open-source experiments. If you’re interested in math, AI interpretability, or causal inference, follow@Pyuyi2333--would love to chat and exchange iddeas!

Lately, I've been reading quite a few papers and theories on AI Alignment. I usually feed the papers and my notes into an AI, then try to make sense of the insights it gives me. And as we all know, AI has this notoriously weird and sometimes almost incomprehensible way of writing. But if I want to read papers more efficiently, I guess I just have to put up with it and try my best to understand... Had a bit of Spritz tonight, and maybe it was the alcohol talking, but a thought suddenly crossed my mind: **isn't this, in some sense, also a form of Alignment? Except that this time, it's humans being forced to align with the way AI thinks...** It reminded me of a system-based web novel where the protagonist was given a mission involving an "intelligence crisis": Prevent the destruction of civilization. However, unclear about which civilization whether make human civilization align with an intelligent civilization to save the intelligent one or other. Would that also count as a way of avoiding disaster...? (MOSS, just how far can your eyes see...? (Wait, what am I even talking about... At this point, humanity's obsession with efficiency seems almost unstoppable. Just like how we probably can't imagine going back to writing all our code manually, our pursuit of technological "prosthetics" seems pretty much inevitable. Whether it's giving ourselves "three heads and six arms," or implanting chips into our brains to help us learn, process information, or control things remotely, it all feels like a natural direction for us to take. And when that happens, it'll be hard to say whether it's the human brain controlling the chips and cybernetic limbs, or the chips and limbs controlling the human brain. (Just think of Doctor Octopus...) Ethics committees might eventually become completely meaningless. People with cybernetic enhancements would gradually outcompete and replace those without them, both at work and in education, much like how people who know how to use AI are already starting to replace those who don't. That would inevitably give rise to underground markets and illegal clinics, and just like that, the cyberpunk era begins. Meanwhile, these "prosthetics" would only widen the gap between the rich and the poor, pushing society closer and closer to the kinds of worlds we've seen in cyberpunk fiction. And here's another thought: while humans are gradually becoming more like machines, humanoid robots are becoming more and more like humans. So who's to say that a human body with most of its parts replaced by cybernetics isn't essentially a humanoid robot? With AI evolving so rapidly and accelerating technological progress along the way, I feel like all of this is becoming increasingly foreseeable. Maybe some of it will even happen within our lifetimes... Ah, wouldn't that be something. I really, really want to go out there, explore the universe, and see the stars...

Language models learn to predict sequences, while accurate prediction alone does not guarantee good decisions against adaptive adversaries. The authors from ICML26 establish a striking theoretical connection: with access to the full interaction history, every sequence distribution is exponentially close in total variation distance to one that induces low-regret decision-making. In principle, prediction quality and adversarial robustness need not fundamentally conflict. Yet finite context windows introduce an impossibility result: certain distributions remain a constant distance from every low-regret distribution. The paper also proves that modest context expansion can restore near-optimal regret guarantees, and that the full-history construction can be implemented by adding a constant number of Transformer layers, supported by preliminary experiments with small models. The deeper insight is that the difference between predicting well and deciding well may lie in the structure of the prediction algorithm rather than its training objective alone. This connects language modeling with online learning, game theory and computational complexity. However, the guarantees concern specific adversarial decision settings, not general-purpose LLM agents. The following challenge might be building finite-context agents that retain predictive performance while achieving provable low regret during real tool-mediated interactions. Perhaps better decision-making requires not only more accurate world predictions, but algorithms that remain effective when those predictions are strategically challenged. arxiv.org/abs/2603.28499

Standard RLVR methods such as GRPO learn primarily from outcome rewards, leaving the internal causes of failure largely unaddressed. Self-Distillation Policy Optimization (SDPO) raised in ICML26 takes a different approach: the model revisits its own failed attempt with environmental feedback, then distills its feedback-informed next-token distribution back into the original policy. No external teacher is needed. On chemistry reasoning with Olmo3-7B, SDPO reaches GRPO's accuracy approximately 6× faster in wall-clock training time. On difficult LiveCodeBench problems, test-time self-distillation achieves comparable solution-discovery probabilities with up to 3× fewer attempts. The deeper insight is that a model's ability to recognize mistakes after receiving feedback can become a dense learning signal rather than remaining a temporary in-context capability. This connects reinforcement learning with KL-based distillation, credit assignment, and amortized inference. However, the method depends on reliable feedback and sufficiently capable self-teachers, while additional computation and out-of-distribution generalization remain open questions. The next challenge is integrating self-distillation with persistent agent memory and tool feedback, allowing failures to produce transferable improvements rather than merely triggering another attempt. arxiv.org/abs/2601.20802

Conventional in-context learning faces an architectural limitation: simulating larger neural networks or more optimization steps generally requires greater model depth or width. The authors from ICML26 establish a theoretical alternative: a constant-depth Transformer can simulate arbitrarily many gradient descent steps on an N-layer neural network by unfolding computation into its generated sequence. Their key innovation combines intermediate computational states with dynamic attention masking, avoiding unnecessary attention operations and improving theoretical computational complexity by an O(N) factor over the previous best construction. Numerical experiments support the analysis. The deeper connection is to algorithm simulation, computational complexity, and iterative optimization: a model's effective computational capacity depends not only on its parameters, but also on how computation is organized over time. This suggests a possible route toward more powerful test-time learning without proportionally expanding network size. However, the guarantees concern specific mathematical constructions, not arbitrary pretrained LLMs, and theoretical complexity improvements do not establish equivalent hardware speedups. The next question is whether Transformers can learn these optimization procedures autonomously and transfer them to real neural network training or online agent adaptation. Perhaps the future of reasoning models lies as much in learning better computational processes as in scaling their parameters. openreview.net/forum?id=uZ8J…

I'm going to London in November for participating in Intensive Course about AI Alignment in @Iliad_research. Coffeechats are welcomed and we have already some booked chats with friends in UCL and IC.

Most representation learning methods rely on prediction objectives, external rewards or fixed datasets. This paper from ICML26 explores a different principle: an agent should represent the world according to its capacity to influence it. Building on information-theoretic empowerment, the authors maximize mutual information between an agent's skills and their future outcomes. Their analysis reveals two complementary representations: a forward representation describing which outcomes can be influenced and a backward representation describing how states can be reached. Under specified assumptions, both become invariant to control-irrelevant information without requiring task rewards or expert demonstrations. In experiments involving Point-Maze, Ant, and visual manipulation, learned representations show at least 1000× greater sensitivity to control-relevant state variables than irrelevant noise. Linear probes also recover the true coordinates of a noisy maze with R² approaching 1, while downstream agents achieve substantially better control performance than baselines using noisy observations directly. The deeper implication is that intelligent perception may emerge from intervention rather than prediction alone. This connects reinforcement learning with mutual information, causal representation learning, and control theory, suggesting a route toward task-independent world models. However, the theoretical guarantees rely on specific assumptions and fully observable environments; learning controllable features does not establish recovery of the complete causal structure. The next question is whether the same principle can help partially observable or language-model-based agents develop transferable causal representations through tool use and environmental interaction. Perhaps building more capable agents requires teaching them not only to predict what happens next, but to discover which aspects of the future they can actually change. arxiv.org/abs/2605.30656

Standard RLVR maximizes expected correctness, but this objective can provide weak learning signals for problems the model rarely solves. The authors from ICML26 oral show that expected-reward RL captures only the first-order term of a more fundamental objective: the log-likelihood of generating correct solutions. Their proposed MaxRL bridges that gap by expressing maximum-likelihood gradients through a weighted combination of pass@k objectives. Surprisingly, the practical modification is almost trivial: normalize successful trajectory gradients by the number of successes rather than the total number of samples. The resulting estimator is unbiased for a truncated likelihood objective and approaches maximum-likelihood optimization as sampling compute increases. In Qwen3 mathematical reasoning experiments, MaxRL improves solution coverage while maintaining competitive pass@1 performance, requiring up to approximately 19.2× fewer test-time samples than GRPO to reach comparable success rates in selected settings. The deeper insight is that an optimization objective determines not just what a model learns, but which problems receive enough gradient signal to become learnable. This connects reinforcement learning with maximum likelihood, logarithmic scoring rules, and gradient geometry. However, the method assumes reliable binary verification, cannot directly learn from prompts producing zero successful trajectories, and incurs additional training costs. The next challenge is combining adaptive sampling, difficulty estimation, and likelihood-based optimization to help models learn from increasingly rare successes. Perhaps better reasoning RL begins with deciding how learning signals should be distributed—not simply how rewards should be maximized. arxiv.org/abs/2602.02710

Most self-improving agents accumulate trajectories, retrieve past experiences and refine their behavior within a manually designed memory system. MemEvolve from ICML26 challenges that assumption by introducing two coupled processes: evolving what an agent remembers and evolving how its memory works. The authors organize 12 representative memory systems into four components—encoding, storage, retrieval, and management—then use trajectory analysis, candidate generation, and iterative evaluation to search for better architectures. Across four agent benchmarks, the method reports performance improvements of up to 17.06%, with evidence that some evolved memory architectures transfer across tasks and backbone models. The deeper insight is that an agent's learning mechanism can itself become an optimization target. This connects agent memory with meta-learning, bilevel optimization and evolutionary search: one process accumulates useful experience, while another improves the machinery responsible for learning from it. Unlike ordinary retrieval-augmented memory, the improvement operates at the level of system design. However, benchmark-specific architectural gains do not establish open-ended recursive self-improvement and evaluation overfitting, search costs, and long-term stability remain unresolved. The next challenge is whether agents can jointly evolve memory, tool-use policies, and experience-transfer mechanisms while demonstrating measurable improvements in future learning efficiency. Perhaps scalable self-improvement requires not just learning more from experience, but learning better ways to learn. arxiv.org/abs/2512.18746

Conventional zeroth-order optimizers such as MeZO estimate gradients through random parameter perturbations, wasting evaluations on directions unrelated to meaningful updates. AGZO from ICML26 exploits a simple but powerful geometric property: the weight gradient of a linear layer lies in the subspace spanned by its input activations. Rather than sampling perturbations across the entire parameter space, it extracts low-rank activation subspaces during forward computation and restricts gradient estimation to those directions. Under spectral concentration assumptions, the authors prove better expected alignment with the true gradient than isotropic perturbations. On Qwen3-0.6B, AGZO achieves 89.2% accuracy on CB versus 80.3% for MeZO, while enabling Qwen3-4B fine-tuning in a hardware setting where full first-order training runs out of memory. The deeper insight is that avoiding backpropagation does not require treating neural networks as structureless black boxes. Forward activations already reveal useful information about optimization geometry, connecting zeroth-order learning with randomized linear algebra and low-dimensional representation structure. However, the method remains substantially more iteration-intensive than first-order training, and its advantages have not been established at frontier pretraining scale. The next question is whether adaptive activation subspaces can reduce both gradient-estimation noise and the number of forward evaluations required for effective learning. Perhaps the next generation of optimizers will use forward computation not merely to evaluate loss, but to determine where parameter updates should occur. arxiv.org/abs/2601.17261

Most reinforcement learning with verifiable rewards (RLVR) reduces an entire reasoning attempt to a single success or failure signal, discarding information contained in compiler errors, failed tests and other environment feedback. The authors introduce Self-Distillation Policy Optimization (SDPO) in ICML26, which lets a model reconsider its predictions after observing feedback, then distills those feedback-conditioned predictions into its original policy. Instead of learning exclusively from sparse outcome rewards, the model receives dense token-level supervision from its own informed predictions. Across scientific reasoning, tool use and competitive programming, SDPO improves learning efficiency over strong RLVR baselines. On difficult binary-reward problems, its test-time variant achieves comparable solution-discovery probabilities with 3× fewer attempts. The deeper insight is that a model's ability to recognize mistakes can become a mechanism for improving its future behavior. This connects self-distillation with policy improvement, amortized inference, and continual learning, suggesting a promising direction for agents that learn from their interactions. But self-generated supervision is only useful when the underlying feedback is trustworthy and the model can interpret it correctly. Otherwise, distillation may reinforce existing errors. The next question is whether tool feedback, environmental observations, and persistent memory can be converted into a unified learning signal that enables agents to accumulate genuinely transferable capabilities over time. arxiv.org/abs/2601.20802

The paper from ICML26 asks a fundamental question: can an AI reason causally about a world whose objects and relationships have changed? Classical structural causal models typically assume a fixed set of variables, but real environments rarely preserve that structure. The authors prove that observations from familiar configurations may be insufficient even to identify the observational distribution of an unseen configuration. Their solution, relational structural causal models (RSCMs), introduces graphical identification conditions that enable causal reasoning across changing object configurations under explicit assumptions, including certain forms of hidden confounding. In simulated traffic environments, their neural implementation reduces causal-effect estimation error by roughly 100× in some identifiable settings, successfully transferring knowledge between different arrangements of cars, pedestrians and traffic signals. The deeper contribution is a mathematical bridge between causal inference and compositional generalization, connecting Pearl’s causal framework with relational learning, graph neural networks and object-centric world models. It suggests that generalizable intelligence may require learning not just representations of objects, but the causal mechanisms governing their interactions. The limitations matter: the theoretical framework relies on assumptions including finite discrete variables, bounded relational neighborhoods and restrictions on cross-object hidden confounding, while experiments remain largely synthetic. The next challenge is learning such structures from high-dimensional observations and using them for planning and reinforcement learning. Perhaps the most important thing a world model should generalize is not the world it has seen, but the causal rules that generate possible worlds. arxiv.org/abs/2606.14892

The paper from ICML26 challenges a fundamental assumption in RL: why optimize the agent while treating its environment and reward function as fixed? RLAnything replaces that separation with a joint optimization loop. The policy learns from both process-level and outcome feedback, the reward model improves through consistency signals and the environment adapts its training tasks using critic feedback. Rather than simply adding more training examples, the system continuously changes what the agent learns, how its behavior is evaluated and which experiences it encounters next. The authors report improvements of 8.5% on OSWorld, 21.2% on AlfWorld, and 12.1% on LiveBench in their evaluated model settings, with training code publicly available. The deeper contribution is a shift from policy optimization to learning-system optimization, connecting agentic RL with bilevel optimization, active learning, automated curriculum design and adaptive control. This offers a concrete research direction for recursive self-improvement: an agent might improve not only its behavior but also the training experiences and feedback mechanisms responsible for future improvements. However, benchmark gains do not establish sustained self-improvement. Jointly evolving rewards and environments can create dangerous feedback loops in which biased evaluations reinforce increasingly misleading training tasks. The next question is whether independent verification, principled improvement criteria and compute-aware environment generation can make this process stable and genuinely transferable. If so, the next frontier in agent training may be optimizing the entire policy–reward–environment ecosystem, rather than the neural network alone. arxiv.org/abs/2602.02488

The paper from ACL26 attacks a more important problem than simply extending context length: how should an agent learn what to remember, revise, forget and actually read later? Memory-R1 turns external memory from a hand-engineered database into a learned control policy. A Memory Manager is RL-trained to choose among ADD, UPDATE, DELETE and NOOP, while a separate Answer Agent selects and distills relevant entries before reasoning and both are optimized with outcome-driven PPO/GRPO rather than fixed summarization or retention heuristics. The striking part is its data efficiency: with only 152 training QA pairs, the system generalizes across LoCoMo, MSC, LongMemEval, and model scales from 3B to 14B. In the authors’ released implementation, the GRPO version on Llama-3.1-8B reaches 45.0 F1, 37.5 BLEU-1, and 62.7 LLM-as-a-Judge on LoCoMo, versus 30.4 / 22.2 / 45.7 for Mem0. The deeper idea is that memory management can be treated as policy learning rather than infrastructure plumbing. That links agent memory to belief-state compression in POMDPs, long-term value estimation in cache replacement and the continual-learning question of which pieces of history remain sufficient statistics for future tasks. For production agents—research assistants, coding agents, customer-service systems, or personal assistants—the implication is that persistent memory may eventually be optimized for future utility instead of growing as an ever-larger transcript. The current evidence is still narrow: controlled long-memory QA benchmarks do not establish robustness under open-ended tool trajectories, corrupted memories, or months of non-stationary interaction. The next step worth watching is joint memory-and-tool policy learning, where an agent learns when to query the world, when to write persistent state and when to forget while explicitly pricing tokens, latency, and error propagation. If that scales, agent memory stops looking like an add-on RAG store and starts looking like the agent’s learned state-transition mechanism. arxiv.org/abs/2508.19828