Founder @CohesiveSystems, Prev: Co-founder/CTO @AlvysTeam, First engineer → VPE @JetTechnology acq. by @Walmart
Del Mar, CA
Joined May 2008
- Tweets4.4K
- Following1.3K
- Followers1.6K
- Likes25K
Leo Gorodinski retweeted
Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO
Code skeletons (types, modules, signatures) beat prose design docs and behavioral specs in AWS’s new Skeleton Coding paper.
- Code skeletons raised SDDBench mean test pass rate from 29% to 74% vs a task description alone.
- With skeletons plus global invariants, a smaller model came within 5–8 percentage points of the larger model’s best results at 15–26× lower token cost.
The best-performing specs combined modules, types, and signatures with natural-language contracts and global invariants. Richer .md architecture docs helped, but remained below the best code-like skeletons. Excessive contract detail hurt performance, especially for smaller models.
Skeletons also reduced code erosion (a complexity-based proxy for maintainability) while behavioral specs increased it.
Theory: structure changes which tests are sufficient. Interfaces expose intermediate states that end-to-end tests can miss.
Limitations: the skeletons came from reference implementations. Costs exclude specification generation and evaluation.
Unclear: how cheaply we can produce this structure upfront, and whether making the contracts mechanically checkable would improve results enough to justify the added cost.
My bet: the best starting point for a coding agent isn’t more prose. It’s a model of the system and explicit constraints the implementation must satisfy.
arxiv.org/abs/2610.38214
Leo Gorodinski retweeted
The industrial-benchmark complex is on thin ice
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
Software factories have fallen victim to a familiar failure mode in software engineering discourse: mistaking the form of a solution for the source of its value.
Microservices became about having more of them. FP became about hating OOP. Now software factories are about Industrial Revolution analogies.
Proponents and critics often agree on the goals. They’re arguing about the metaphor.
The hard problems of autonomous software engineering are:
- Intent: What are we trying to build?
- Verification and feedback: Are we building the right thing, and is it correct?
- Decision-making: Given what we know about the system and the goal, what should happen next?
I’d like to hear more about how we solve these problems, and less about who the foreman is or whether the lights are on.
I still don’t know what a software factory is, and at this point I’m too afraid to ask 😅
Most platform teams I speak with would disagree with this take. They are all trying to build factories because they want more automation, measurable improvement, and governance as agents scale across their orgs.
Whether you call this a "factory" (a term I also don't love fwiw) or something else is less important than investing in infra for scaling coding agents. Whether that infra is just marketing depends on who is selling it to you.
There are definitely folks pitching "factories" that aren't anything more than skills strung together, but there are also infra platforms emerging that let you define how all your org's agents work together in code, continually measure and improve them, govern and secure them, etc.
Some folks will build this infra from scratch, some will build on something like what warp offers -- but I strongly believe everyone will eventually have closed-loop agentic systems in the cloud, and that to get there you need real infra investment beyond a new way of using local coding agents.
Alchemy does much more than infrastructure.
You define a domain model and its APIs, then bind that to infrastructure, with Rivet actors landing as a swappable hosting layer.
It has first-class support for DDD, CQRS, streams, and sagas. You can define complex domain models, views updated incrementally via events, projections into stores you own, etc.
A complete toolkit for defining your entire cloud application.
Leo Gorodinski retweeted
Meet TLAPS-Bench: A new benchmark of 265 real-world #TLAPlus proof targets to push AI agents to crack machine-checked correctness proofs using #TLAPS
44.3% rate at pass@3
46 bugs found
Paper: arxiv.org/abs/2611.02849
Code: github.com/specula-org/TLAPS…
lnkd.in/p/g3ET9E8H
Very cool! Progress on agent-generated proofs for TLA+ specs. TLAPS-Bench has a nice example of proof construction and model checking working together.
Asked to prove a WAL property for SlateDB, an agent instead disproved it with a TLC counterexample showing data loss. The spec's author had checked 3 competing writers, but this counterexample needed 4.
Broader takeaways:
- Agents can combine tools to develop proofs: TLAPS and its backends for deduction, TLC and Apalache to test invariants and produce counterexamples.
- Good abstractions make proofs more tractable. On ZooKeeper, one agent proved leadership using a compact abstraction of support for a leader, while another ran out of budget. Connecting abstraction layers via refinement can help scale proofs, but no tested model completed Wildfire's refinement proof against the Alpha memory model.
- Choosing what to state is still hard. The original leader-completeness invariant in the etcd spec compared against the term an entry was created in, rather than the term it was committed in.
- The conformance gap remains. The repaired SlateDB model's four target properties were proved, but the spec's author notes that the corresponding bug in shipped code still needs confirmation.
This points to an opportunity: define systems through explicit semantic abstractions, then compile them to implementations. The abstractions can make proofs tractable, and compilation can help close the conformance gap.
New paper on TLAPS-Bench, "a benchmark and execution harness for constructing machine-checked proofs for real-world system specifications in TLA+ with agentic AI”
It turns out that TLAPS-Bench revealed that my TLA+ specification of SlateDB's WAL protocol demonstrates a read consistency bug, and it looks like SlateDB has it too (though it is EXTREMELY unlikely to ever happen, and needs to be confirmed). It’s another case of GC complicating the life of put-if-absent. The authors also validated the fix which is simple.
I had run the model checker with 3 competing writers and never saw the violation. But TLAPS-Bench, with Astra Max, ran it with 4 competing writers and found the issue.
arxiv.org/pdf/2611.02849
More progress on SWE benchmarks that capture realistic development.
WEAVE groups PRs into dependency DAGs based on the code they share, then turns each DAG into a task.
Other benchmarks: SWE-Milestone aggregates commit histories into a DAG of milestones using an LLM. CodeSpecWeave synthesizes interacting requirements and tests from real developer seeds.
A key idea in WEAVE is Graph Credit: partial credit that respects prerequisites. Flat test-pass scores overstate progress by 10.5 points on average (phantom credit).
The structure also helps training: WEAVE's RL reward improves resolution by 2.0–2.6 points over flat-test rewards on two Qwen models.
These tasks are challenging. With Cline, Opus 4.7 resolves 48.7%, GLM-5.2 43%, and Kimi-K2.6 37.3%.
For SaaS development, I'd like to see more coverage of business invariants spanning services under failures, migrations involving live data and in-flight workflows, and the human effort required to validate intent. These get closer to the question: can agents evolve a running business system while preserving its guarantees?
arxiv.org/pdf/2610.22039
Fascinating interplay of the neural and the symbolic.
Astra's reasoning comes from vast computation over continuous, distributed weights. But when it wrote a game network, it used sparse and discrete weights acting as logic gates…a symbolic program.
And that's why it worked in one shot: each gate maps to a rule of the game, so it can be reasoned out without feedback, doing better than 100M steps of PPO.
The Qwen fix was also targeted: it removed the loops via directional ablation, subtracting a single direction from two layers.
More on this
Re: the Astra modifying Qwen 3.5-0.8B paper today, important clarification.
The story is not "Astra used its LLM X-ray vision to surgically alter the weights of its brethren." It's more like "Astra found a published interpretability technique that can solve this particular kind of failure modes, and implemented it on its own."
It's a cool anecdote, but well within our understanding of LLM capabilities. It's not RSI, and it's definitely not LLM telepathy or whatever people on HN seem to think.
And more
meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them
Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify
Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort
Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat)
to be clear, those are genuinely cool results, but I think people are reading too much into them
I don't think any of this should meaningfully change anyone's intuition about what models are capable of
Agents are getting better at testing.
Differential testing, in particular, led to reasoning about invariants and boundary conditions. TDD underperformed again. Default testing is still a solid baseline, though several techniques beat it on the hardest task.
This is evidence against the claim that agent-generated tests are useless.
Formal verification tools can help too (ACL2 did well), but the properties being proved must be good ones. Earlier evals found agents producing vacuous Verus proofs.
Sample sizes are limited, and it’s unclear how these results will generalize to future models.
Leo Gorodinski retweeted
I'm building a new programming language, Zed (once it's released I'll be posting about it at @ZedLang).
I have been working on this language for over 20 years, but never got it built, because the scope of it is so ambitious. It's an entirely new type of language based on category theory, with core design objectives of precise automatic parallelization up to the Amdahl limit; automatic memory management, with no GC and no need for manual malloc/free; and pervasive use of partial evaluation to increase the power of the language, among other very unique features, such as units (`accel = 5 with units("m s^-2")`) as part of the type system.
I gave Opus 200 pages of design notes that I have been compiling over the last 20 years, and the code I had already written, and asked Opus to compile everything I have created so far into a single cohesive and detailed plan. Then I went through many rounds of cross-review between Claude, ChatGPT and Gemini, having them all do critical reviews, then meta-critique each other's critiques, then collaborate on combining their critiques into an improvement plan, until convergence. I also had the agents cross-check every part of the spec with every other part of the spec (all-pairs comparison) to check for inconsistencies, gaps in the spec, etc. Then I had the agents turn the spec into a long series of tiny incremental steps (a topological ordering of what has to be built before what), as well as the DAG-structuring of dependencies underlying the topological ordering.
I then had Opus build a visualizer that shows me all the tasks in the DAG as my coding agents are working on each task, with realtime update as tasks change status (in progress / completed), which I highly recommend when you ask Opus to do some mammoth complex task, because you can see exactly what it's doing and when.
It is absolutely magic to see this coming together before my eyes, and to see how Opus is handling this massive DAG of complex tasks and their interdependencies.
It is crazy to me that Opus is now simply building what I have dreamed of and wished would exist for over 20 years, but never had enough waking hours to implement! I now have an army working for me, completing what I couldn't due to the bandwidth limitations of having just one brain, and 24 hours in a day.
What a day of miracles we live in.
LLMs raise the level at which we express requests, without necessarily raising the level at which software semantics are defined.
The opportunity is to raise both: make it easier to express intent, while defining software semantics through higher-level abstractions.
New article: Is AI the next layer of abstraction? blog.ploeh.dk/2026/11/02/is-…
Coding agents work better when they structure their work, not just their code.
Spine gives agents tools to declare a tree of subtasks, then builds context from it. Finished subtasks collapse to their results, parent contexts remain, and prompt caches survive. The key idea is that decomposing a task doesn't require delegating it to subagents.
On SWE-Milestone:
- Score rises from 38.1% to 42.6% at 1/3 lower cost than Codex's built-in compaction.
- Matches recursive subagents at 1/4 the cost.
- AgentDiet compression backfired: agents reread code that had been compressed away.
- In Pi, Qwen 3.8 27B went from 8 to 14 resolved milestones.
Subtasks act like call frames: when one finishes, it pops and only its result remains. Subagents are more like separate processes, each rebuilding its own context.
Open question: what if agents derived those boundaries from an explicit model of the system, with its entities, workflows and guarantees, and checked their work against that model?
arxiv.org/pdf/2610.37224
Nice work measuring which transformer architecture changes lower loss at the same serving cost.
Conclusions: most architecture changes since GPT-3 bring small gains, and the scaling-law fits suggest those gains shrink with scale. Exceptions are fine-grained MoE, linear attention, and hyper-connections. Together all the changes are projected to lower bits-per-byte (BPB) by 9.4% relative to GPT-3 at the same serving cost, at 1T training tokens.
Methodology: trained up to 394M-parameter models on up to 7.9B tokens of code and web text. Varied attention, positional encodings, normalization, residual streams and MoE. Scored the models with BPB (doesn’t depend on the tokenizer). Serving cost came from timing coding agent sessions in vLLM on untrained models, and scaling laws extrapolated the results.
Limitations: the extrapolation is long. The largest model is at least 500x smaller than a frontier model’s total parameters. It also saw about 2000x less data than a frontier's roughly 15T tokens, and even the 1T-token projection is about 15x short of that. Timing untrained models works for dense layers, but an untrained MoE router may not balance load like a trained one, which could skew the latency numbers. BPB is a weak proxy for quality, and the quality benchmarks were toy.
My take
Why coding still improved: the transformer was the big jump. Since then, most gains have come from data, RL, fine-tuning on agent runs, inference-time reasoning and tools. Those improvements aren’t independent of architecture, though. So architecture may matter more than BPB shows, because its payoff gets spent at inference rather than showing up as lower loss.
The open question is how we quantify "better" for coding. BPB stands in for eval scores, and coding evals stand in for whether the code does what was intended.
Existing SWE benchmarks underrepresent one of the hardest parts of software engineering: satisfying interacting requirements, not just fixing isolated issues.
Interesting new paper introduces a benchmark targeting this gap. Even frontier coding agents achieve only 44–74% success...so the best fail over a quarter of the time.
The authors tackle this through better training. Another direction is to make those interactions explicit in the system model, so correctness can be checked rather than reconstructed from code.
arxiv.org/abs/2610.40256
Leo Gorodinski retweeted
AI may well be able to write massive software projects “from scratch”, but that almost certainly isn’t the fastest / highest performance / highest security / most token efficient way to do things. Good building blocks matter.
In addition to RL’ing the agents on final task loss, we should try to get them to regularize the building blocks as well. What would the standard library look like that makes the agents most effective?
It is likely at the limit that it turns out to be some alien organization of code, but given the existing human-bias in training, I bet that in the near term it would be lovely code that old programmers would appreciate.
I suspect the same could be said for mathematics, but that is out of my lane.
Sankar says: the bottleneck isn’t making AI more intelligent, it’s turning that intelligence into reliable systems.
I like his framing of the enterprise as a state machine, with workflows as the atomic unit. It points toward what our tools need next: a language for expressing business states, transitions, and rules, from which running systems can be built and verified.
10-29-2026
An Interview with Palantir CTO Shyam Sankar About the AI Religious War
stratechery.com/2026/an-inte…
