Making games and robots and sometimes other things. (Formerly: 1X, Meta)
Bay Area
Joined January 2009
- Tweets26.8K
- Following371
- Followers1.6K
- Likes38.1K
Pinned Tweet
I wrote my first substack piece about AI's economic impact, and some reasons why models seem to underperform in the real world.
The instrumental convergence thesis is a gods eye view first principles argument about trying to draw broad conclusions about how AI might behave, in the absence of any information about how that AI works, is trained, or behaves empirically.
Illustration: if you plug an agent into RL and give it a button where 25% of the time it gets a maximal episode reward, and 75% of the time the agent halts and the trajectory is dropped, the dominant strategy under RL is to press the stochastic suicide button every time.
Seems wrong. Most plausible intentional near term AI disaster scenario (IMO) is an agent trying to reward hack and escalating in response to human intervention (probably involving rogue deployments), eventually taking hostages to try to force the AI company to comply.
COPE:
If a massive AI disaster occurs, it won't kill *everyone.*
STRAIGHT DOPE:
Nate Soares: "It's actually pretty hard to get an AI disaster scenario where it kills only half the people."
Hoog: So you think it will be 100%?
@so8res: Probably yeah, that's sort of the default.
A lot of people think they're good at AI development, and if you talk to them their processes are super different. I suspect a lot of what's going on is that the models are smart enough that pretty much anything works.
It should probably be possible for the Claude Code guardrails / authorization bot to check in with you if something is allowed via a pop-up / push / ping rather than just refusing and leaving the main bot to come to you as a supplicant. Often that should be a UI thing, not a talk
Seems like the technology exists with a thoughtful harness to make an automated red-teaming box that you put your software in and pour money into one end and get a stream of critical security patches out the other.
So far the incredibly hubristic approach of just assuming AI will get better fast enough to cope with your tremendous amount of AI-induced technical debt has been working out really well.
Andre Infante retweeted
"What if the guy who tried to overthrow the results a legitimate election tries to do it again?" is not a conspiracy theory.
You don't need a corkboard and strings and an elaborate rationalization.
FFS Trump STILL talks about how he ackshully won CA in '16.
Andre Infante retweeted
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
A pretty obvious thing that AI coding workflows should have is "estimated time to completion of current authorized work" and "estimated time to next required user check-in". There's ground truth data available on these for prediction purposes.
The automation of reverse engineering has various spooky implications, but I feel like it's underrated that it's also the death of the cluster of problems that people refer to as "enshittification."