Information from Documents

Let's test some variants on Qwen3-8B to illustrate why. Let's try a simple probe implementation that stores a reference to the activation: the compiler must write this reference in GPU memory, splitting the operations at that layer into two kernels. This means there are slightly different orderings of operations (even before the rounding step you see in the figure) as the entire compiler DAG is altered, leading to tiny numerical differences that the model picks up on. Let's also try just splitting into two kernels, with no probe. Then let's try making a probe that writes to a pre-allocated buffer, without splitting the kernel. You can see that all three interventions change activations. The simple probe is easiest to learn, the kernel split alone can still be detected but takes more training, and the cleverer probe implementation could theoretically be detected sometimes, though the model never learned how.

something very interesting is happening i’ve been monitoring user sentiment on new model releases and i have now seen 3 latest frontier models in a row where a significant amount of users chose to roll back to use a previous version 1. opus 5 - very negative feedback in general can’t communicate. eager to jump into conclusions without sufficient context. keeps making mistakes it even earned its own website now opusfived.dev (it’s really funny) many people (myself included) abandoned opus 5 and went back to 4.6 and 4.8 2. gpt 6 astra seeing more and more people reporting going back to 5.6, both sol and luna main reasons are: too expensive to use as primary model; spiky performance - sometimes very smart but sometimes very dumb; not great at understanding user intent and knowing when to stop vs keep going 3. grok 4.6 i like the grok model family in general but there’s now more and more evidence that 4.6 is not strictly better than 4.5 it’s slower, more expensive, and doesn’t seem to offer meaningful increase in intelligence see vulcanbench.com/leaderboard.… if this happened to just one model, it’s probably just a bad checkpoint and i wouldn’t find it interesting but all these latest model releases across companies all seem to be running into a wall? that’s something worth understanding it seems the common themes are: - the cost vs quality tradeoff no longer makes sense to most people perhaps the models are already smart enough for most tasks, and people don’t perceive enough benefit from the additional intelligence to justify the extra cost perhaps the model companies should focus more on making fable tier models dirt cheap, rather than inventing something better but even more expensive - training has gone off rails opus 5 having better benchmark than fable was the most clear example of this - i haven’t seen a single person say they agree opus 5 works better than fable in practice it’s clear that the benchmarks and post training has started optimizing for the wrong things people don’t actually care about - fable 5.1 is the only release i don’t hear much complaints if anything, people just want the 50% cap removed so they can use fable more now with all the evidence that most high performing open source models are just distillations of claude, it seems to show that anthropic still has a moat what do you think? what do you want the most from the next model release?