ib retweeted
I hope this discussion on evals for evals makes clear that we don’t (just) need better benchmarks. We need better tools for eval engineers to build, examine, improve, and maintain benchmarks. This feels so obvious and so critical.
ib retweeted
Eval developers are the next analytics engineers; people who will set standards that ultimately govern how models are trained and deployed. I expect to see many more companies hire for this role.
ib retweeted
A number of talks at @aiDotEngineer lands on the same conclusion: Agents need a shared ontology.
An ontology tells an agent what things mean: this is a Customer, that field is Revenue, these two relate. It's necessary, but also static.
What agents actually lack is what follows from what: if this changes, that breaks; this spiked because of that. The casual relationship, not category.
Ontologies ground meaning and the layer above grounds reasoning. Scaling agents at the enterprise level requires both.
✨ontologies✨
thanks Prof @coyle_frankp, @emileifrem, Jesús Barrasa, @steveonjava et al for reintroducing these concepts to the AI Engineering world
ib retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
ib retweeted
Kimi's CEO🇨🇳 says every AI lab has the wrong obsession
Zhilin Yang, on why K3 beat the frontier: "Every lab, like Claude, thinks the model matters most. That's wrong. It's how you organize the people building it that wins."
Then Moonshot proved the philosophy. K3 sold out every plan on purpose, cutting their own revenue instead of throttling existing users. When Anthropic hit that wall in April, they cut users' usage 50% at peak.
His one big idea: long context is the AI era's RAM. The 128K-to-gigabytes jump, compressed into 2 years instead of 40.
The real moat: "your biggest advantage, perhaps your only advantage, is your organization."
Model, or the team behind it: which actually wins?
ib retweeted
Our first model, Inkling. Trained from scratch, weights are open, fine-tunable on Tinker today.
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
ib retweeted
I actually feel strongly that the "learning" companies will want to do with their data will mostly not be to train models.
Let's say you have a bunch of valuable data about your customers or employees.
Your best bet to make that IP useful is to turn it into skills or artifacts that models can use in-context.
If you go through the effort to train it into the model:
1. that is hard to get right and takes a lot of time + effort
2. you will have to do it all over again when the next base model comes out
3. most importantly, it is irreversible... when things change, you cannot untrain what you did
+ the brain drain is making the future look pretty bleak. On the other hand, there's a promise of more effective personalized education with AI, but not sure when that would be realized.
the destruction of American education over the past decade is an incredible self-own
competence is objective. a child can either do the math or they can’t. but in the u.s., a lot of people have reasons not to say that!
parents don’t want to hear their kid is struggling. teachers don’t want scores used to manage them. districts don’t want embarrassment. progressives worry accountability will create inequity. conservatives don’t want federal authorities. so we end up with process, weak standards, and excuses to explain away bad outcomes.
people object that it’s phones, covid, demographic change.
ok! but we fail globally when others have phones, covid too — vietnam is much poorer than the u.s., yet performs well in international math comparisons. some countries treat math as a basic skill everyone needs to master. here, it is part of a fight about fairness, autonomy, and feelings
in the age of AI — if people can’t do basic math, read closely, or think through problems, ai won’t make them more capable. it will become something they rely on without understanding. the countries that come out ahead in the global race won’t just have better technology. they’ll have people who know how to use it, question it, build on it. we need the national ability to decide something is worth doing coherently (teaching math!)
the US has the money to teach math well but it has not shown the will. we are failing the next generation
Tip of the day!
Yes! binary judges are far more practical for most people, because likert scales (or scores) have too many footguns
All the flashcards are here (inspired by @chrisalbon ‘s flashcards) maven.com/parlance-labs/o/54…
Starting to notice more conversations from engineers about how to measure the ROI of coding agent usage.
ib retweeted
A knowledge graph stores facts, while a Wisdom Graph stores causal judgments.
MEGA turns data (traces, context, etc) into verified wisdom and feeds it into the next task, so agents get measurably better the more they run. It decomposes data into a causal triplet it can reason over (deduction, abduction, induction), and every conclusion is auditable back to its causal steps. Then a statistical gate admits only real gains, so it compounds signal, not noise.
And wisdom compounds at every level, across a team, an org, and globally, so each task starts on the shoulders of everything learned before it.
