sam

So the CEO of proton(mail) released an "AI manifesto", and hidden in all the usual fluff is an unintentionally perfect summary of the entire situation we find ourselves in these days:

I've been mulling over Peter's thoughts in this post for a couple of months, and it's finally hit me: LLMs have essentially disconnected the concepts of production and verification, which for most of human history have been inextricably linked. Put another way: they can create things without understanding them. In the past it was vanishingly rare for someone to produce a correct complex math result, or a large functional program, while simultaneously having no idea how it actually worked. Humans essentially create through a process of "incremental verification". We are constantly course-correcting. A convincing sounding rebuttal to this position is "but a person using Wix has no idea how their website actually works!". That's somewhat true for the end user themselves, but someone, somewhere still does (the wix devs, the browser company, the AWS engineers, the linux maintainers, etc). We still hold the knowledge & understanding in aggregate across the human race at large, and the process for producing that website has been incrementally verified all the way up to the point that the end user engages with it. As a result, the elephant in the room is that this new mode of production relies on retroactive verification of the entire solution, which is extravagantly more expensive (in terms of human effort) than the incremental verification humans do along the way while creating something themselves. In fact from my experience with AI programming tools, I am beginning to suspect that retroactive verification is actually more expensive on its own than the entire process of creation "by hand" + incremental verification. I suspect this is also the reason why the most successful users of AI code generation tools seem to be people who are a) experts in their domains already and b) willing to babysit the production of the software in small enough chunks that they can personally incrementally verify the output. When OAI dumps 1000s of math proofs on the community, they're essentially washing their hands of the work required to actually verify the results. I wouldn't be surprised if paradoxically, we see the cost of verifying these "P-like" tasks actually greatly increase over the next 10 years, as experts tire of verifying machine generated output, and are forced out of the field by changes in economics. Of course people love to talk about the idea that "the LLMs will verify the output" but if you've used any of these tools you'll know that they all suffer from the same seemingly intractable "steering problem", where the output is dependent on the expertise of the operator. This is as true today as it was in the very early stages. Put simply, if you don't have any experts that can verify things by hand, LLM verification isn't going to help you.