Richard Valtr retweeted
Good Will Hunting GPU Edition
Ep. 2: My Boy's Wicked Smaht
Ben Affleck reveals he writes Python, understands convolutional neural networks, worked extensively with GPUs, and used his celebrity status to get private looks at Google and OpenAI’s video models
“I’ve always been kind of into computers since I was young. Then, when film started to move from analog film to digital, I became more interested in that aspect of it. The visual-effects workflow for many years has included machine learning, so I can write pretty shitty Python scripts and stuff like that.
“With convolutional neural networks, which were the precursors to what the transformer can do, which is much more computation simultaneously, you would do things like look at what’s called a tensor. That’s the numerical translation of a visual image in numbers, like the batch number, the frame number and the red, green and blue values of each pixel in each frame. It’s just that simple. That numeric is called a tensor.
“You’d use a convolutional neural network to identify patterns that reveal what’s called edge detection or feature extraction, which is identifying patterns well enough to know, this is where the window ledge is, so we can more easily take the green-screen image out and replace it with something.
“That was familiar to me early on because, prior to Artists Equity, I had a small visual-effects company. I’ve worked with GPUs a lot too. The visual-effects guys said, ‘Hey, you should see. There are a couple: Google and this other company, OpenAI, are doing really interesting stuff with transformers in video.’
“I’ve learned that I can actually just call up and go, ‘Hey, it’s Ben Affleck. Can I come see what you’re doing?’ Sometimes people say yes, to my astonishment.”
Richard Valtr retweeted
Physicists may have found the origin of gravity. And that would finally explain why gravity is different from the other forces of nature.
Richard Valtr retweeted
A very teleological view. But yes, this is one reason why only America really could have invented LLMs: you need a technosphere which is large enough to have social media (lots of natural language dialogue tokens) and you also need a large pool of software engineers and capital markets and domestic B2B demand to sustain cloud companies (for datacenters). There are five technospheres: America, China, Russia, Japan, and Korea. Only America, China and Russia have social media, search + cloud, and Russia obviously can't get chips / has other problems. Along these lines, European "sovereign AI" can't really be taken seriously as long as they don't sustain their own technosphere. Like they want to have sovereign AI but they need to have their own technosphere first. (As is oft pointed out, it's Google's monopoly on search as a money printer that allowed them to sustain the research largesse which led to lots of core infra incl the Transformer itself but also lots of research up to that.)
Prev comments on: Korea's lack of LLM development despite an early read [0] and China's shallow b2b market [1]
[0] xitter.cf/distributionat/status/…
[1] xitter.cf/distributionat/status/…
the US SaaS market is about 10x larger than the Chinese SaaS market. the larger american SaaS market trains american companies to be far more willing to pay for tokens, which is why US AI labs make so much money on B2B. there is an interesting analogy to the industrial revolution here.
(I) reasons the US SaaS market is larger:
(a) the biggest one is simply that engineers in China are cheaper to hire, so it just makes sense to build software in house vs buy b2b. keep in mind that this is what americans used to do too, back in the day. but now a good senior software engineer at a big tech co in SF is like $400-600 k all-in, so american companies just pay the saas bill instead.
(b) SaaS vendors in the US have economies of scale + cultural advantages. if you start a tech company one of the first things you do is open a stripe account, open a slack account, open a chatgpt account, etc. this is considered best practice and the cultural norm because most engineers will copy over the stack from their previous employer. in contrast, chinese companies, especially large SOEs, want a lot of customization, are used to things being done in house, etc.
(II) consequently, american companies are pretty used to the idea of a large monthly bill for b2b providers and this translates neatly to being willing to pay for tokens. there's of course a lot of grappling with how to allocate the token budget, and how much to pay, but the transition is smoother.
(III) there is a very interesting analogy to the industrial revolution. as you may know, one argument for the industrial revolution occurring in 1700s britain and not, say, mughal india or song china is the price ratio between labor and energy. britain at the time had very high wages and very cheap, accessible coal deposits. today, the important ratio is the price ratio between labor and compute, or labor and tokens. in the US, labor is very expensive and compute is relatively cheap (or was, a few years ago). in China, labor is much cheaper, and compute is much more constrained. so the economic conditions for AI are much more favorable in the US than in China.
(a) one might object that OAI and Anthropic were raising billions before their b2b businesses took off. for example, OAI's consumer to enterprise mix was heavily dominated by consumer after ChatGPT's release. this is definitely true but, per OAI, 47% of all ChatGPT messages were related to work in June 2024. so it was definitely not all consumer qua non-productivity. and Anthropic's recent valuation jumps are definitely because of their productivity use cases, not because of consumer ones.
Richard Valtr retweeted
Arsenal—23 years
Knicks—53 years
Two long barren runs come to an end in 2026 🤝
Richard Valtr retweeted
Anyways, all this is to say I think a lot of people underestimate the extent to which AI is much better now than it was last year, and a lot of people overestimate the extent to which this necessarily means we're going to get 20% GDP growth or mass unemployment.
Richard Valtr retweeted
⚡ Investment Announcement ⚡ And also a BIG MILESTONE 🇨🇿 Czech Unicorn @MewsSystems, founded by @valtrese, raises a record $300 Million in a Series D round, led by @eqt, @atomico, @harbourvest_Ai, @KinnevikAB, @BatteryVentures, and @Tiger_Global.
startupkitchen.community/cze…
Richard Valtr retweeted
Vážení spoluobčané, jako prezident republiky mám tu výjimečnou možnost promluvit k vám hned první den nového roku a popřát vám, aby byl dobrý a aby se každému z vás dařilo.
Richard Valtr retweeted
Do humans learn like transformers?
It's a question that sounds almost philosophical, but Pesnot Lerousseau and Summerfield turned it into a rigorous experiment. They trained both humans (n = 530) and small transformer networks on the same rule-learning task, then manipulated a single variable: the statistical distribution of training examples—from fully diverse (every example unique) to highly redundant (the same items repeated over and over).
The result is striking. Both humans and transformers show nearly identical sensitivity to this manipulation. Train on diverse data, and learners generalize rules to novel situations ("in-context learning"). Train on redundant data, and they memorize specific examples ("in-weights learning"). The transition between strategies occurs at the same critical point (Zipf exponent α ≈ 1) in both biological and artificial systems. Neither can easily do both—until you give them a composite distribution mixing diversity and redundancy, at which point both humans and transformers become "double learners."
But here's where they diverge: humans benefit from curricula. Present diverse examples early, and people discover the generalizable rule without losing the ability to memorize later. Transformers, by contrast, suffer catastrophic interference—whatever they learn second overwrites what came first.
The implication for AI and education alike: the structure of training data matters as much as its content. And while transformers may match human learning in surprising ways, they still lack the flexibility that lets us benefit from well-designed curricula.
Paper: nature.com/articles/s41562-0…
Richard Valtr retweeted
Some deep thinking about the frontier-model business model. All of this is grounded in numbers leaked by The Information, NYT, etc.
🔵The Core: It’s a Compute-Burn Machine
At its heart, the model is brutally simple: almost all costs come from compute – inference, and especially training. Training follows something like a scaling law. Let’s assume costs rise ~5x every year; and ROI on training costs is 2x.
That creates a weird dynamic:
Year 1 training cost: 1
Year 2 revenue from that model: 2
But Year 2 training cost for the next model: 5
Net: +2 - 5 = -3
Run it forward and it gets worse:
Year 3 revenue: +10
Year 3 training cost: -25
Net: -15
Frontier models, as currently run, are negative-cash-flow snowballs. Every generation burns more cash than the one before.
For this to ever flip to positive cash flow, only two things can logically change:
A. Revenue grows much faster than 2x, or
B. Training cost growth slows from 5x a year to something like <2x
Anthropic’s CEO Dario Amodei has broken down scenario B (“training costs stop growing exponentially”) into two possible realities:
youtu.be/GcqQ1ebBqkc
1/ Physical/economic limits: You simply can’t train a model 5x bigger — not enough chips, not enough power, or the cost approaches world GDP.
2/ Diminishing returns: You could train a bigger model, but the scaling curve flattens. Spending another 10x stops being worth it.
What OpenAI and Anthropic’s Numbers Reveal:
Both companies’ leaked financial projections basically validate this framework.
OpenAI: OpenAI’s plan effectively assumes total compute capacity stops growing after 2028.
Translation: margins improve because training costs flatten. This is scenario B.
theinformation.com/articles/…
Anthropic:
1/ They assume the ROI per model increases each year. Spend 1, get back say 5 instead of 2.
2/ Their compute spend growth is also much more muted. From FY25 to FY28: OpenAI compute cost growth >> Anthropic's
Using the framework above, they’re counting on both A revenue ramp and B slower cost growth.
theinformation.com/articles/…
🔵 $NFLX Is the Closest Analogy
In tech, capital-intensive models are rare, though not unprecedented. $NFLX is a good analogy: for years it had deeply negative cash flow that worsened annually. They had to pour money into content upfront, and those assets depreciated over four years. In many ways it resembles data-center and model-training economics.
Peak cash burn in 2019: -3B
2020 cash flow: +2B
Why the sudden swing positive? COVID shut down production. Content spend stopped growing. Cash flow instantly flipped.
🔵The Endgame: Margins Arrive When Cost Growth Slows
$NFLX didn’t stop investing in content entirely – it just stopped *growing* that investment aggressively once it reached ~300M global subscribers. At that scale, stickiness is high, and they only need to maintain their position, not expand content spend 10x a year.
I don’t think OpenAI or Anthropic will ever stop training entirely. But they won’t need to grow training spend by multiples forever. At some point:
ROI per model goes up, or scaling limits kick in, or both.
And the moment annual training spend stops growing 5x a year, profit margins show up almost immediately.
That’s the strange thing about LLM economics:
It’s a burn machine…until suddenly it isn’t.
Sources:
theinformation.com/articles/…
theinformation.com/articles/…
theinformation.com/articles/…
theinformation.com/articles/…
++
Full article:
open.substack.com/pub/robono…
Richard Valtr retweeted
Sharing an interesting recent conversation on AI's impact on the economy.
AI has been compared to various historical precedents: electricity, industrial revolution, etc., I think the strongest analogy is that of AI as a new computing paradigm (Software 2.0) because both are fundamentally about the automation of digital information processing.
If you were to forecast the impact of computing on the job market in ~1980s, the most predictive feature of a task/job you'd look at is to what extent the algorithm of it is fixed, i.e. are you just mechanically transforming information according to rote, easy to specify rules (e.g. typing, bookkeeping, human calculators, etc.)? Back then, this was the class of programs that the computing capability of that era allowed us to write (by hand, manually).
With AI now, we are able to write new programs that we could never hope to write by hand before. We do it by specifying objectives (e.g. classification accuracy, reward functions), and we search the program space via gradient descent to find neural networks that work well against that objective. This is my Software 2.0 blog post from a while ago. In this new programming paradigm then, the new most predictive feature to look at is verifiability. If a task/job is verifiable, then it is optimizable directly or via reinforcement learning, and a neural net can be trained to work extremely well. It's about to what extent an AI can "practice" something. The environment has to be resettable (you can start a new attempt), efficient (a lot attempts can be made), and rewardable (there is some automated process to reward any specific attempt that was made).
The more a task/job is verifiable, the more amenable it is to automation in the new programming paradigm. If it is not verifiable, it has to fall out from neural net magic of generalization fingers crossed, or via weaker means like imitation. This is what's driving the "jagged" frontier of progress in LLMs. Tasks that are verifiable progress rapidly, including possibly beyond the ability of top experts (e.g. math, code, amount of time spent watching videos, anything that looks like puzzles with correct answers), while many others lag by comparison (creative, strategic, tasks that combine real-world knowledge, state, context and common sense).
Software 1.0 easily automates what you can specify.
Software 2.0 easily automates what you can verify.
Richard Valtr retweeted
The unbridled joy of listening to someone smart who’s not trying to sell you anything.
The @karpathy interview
0:00:00 – AGI is still a decade away
0:30:33 – LLM cognitive deficits
0:40:53 – RL is terrible
0:50:26 – How do humans learn?
1:07:13 – AGI will blend into 2% GDP growth
1:18:24 – ASI
1:33:38 – Evolution of intelligence & culture
1:43:43 - Why self driving took so long
1:57:08 - Future of education
Look up Dwarkesh Podcast on YouTube, Apple Podcasts, Spotify, etc. Enjoy!
Richard Valtr retweeted
Like everything @jburnmurdoch makes, this chart is amazing.
The sharp decline in conscientiousness and rise in neuroticism among young people is astonishing.
But also of note: literally every age group has gotten less extroverted in the age of the smartphone
Richard Valtr retweeted
This is the most inspiring thing I’ve read in AI in the last two years: storage.googleapis.com/deepm…
What a beautiful future ahead, just happy to take part in it.
Richard Valtr retweeted
Exciting news from our portfolio company, @MewsSystems! 🙌 Mews has raised $75 million to fuel expansion in the U.S. and Europe while continuing to innovate in hospitality tech. Congrats to Richard Valtr (@valtrese), Matthijs Welle and the team!
Read more: axios.com/pro/retail-deals/2…
Richard Valtr retweeted
Man like @PeterOnSports playing Kendrick's 'Humble' at FT 💀
Richard Valtr retweeted
New randomized, controlled trial of students using GPT-4 as a tutor in Nigeria. 6 weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions.
And it helped all students, especially girls who were initially behind
