Alex Chekholko

New job interview rule: If the interviewer is AI, you are legally required to bring an AI interpreter. Do not go in raw. You will get cooked. Watch what happens when the candidate has backup. #AI #JobInterview #Hiring

This is both neat & a sign of why it is proving to be so hard to isolate agents from each other. They share a common "brain" which means they don't have a lot of trouble recognizing creative communication solutions generated by other AIs Wonder if it works across model families?

Muse Spark Open leads open-weight models on the Ortet Health Index, which covers clinical medicine, biological research, and medical coding. The Ortet Health Index is calculated from model-to-model wins, ties, and losses across health benchmarks, giving each benchmark family equal weight. @AIatMeta

Misha Laskin of Reflection AI joins Jason to discuss why companies want to own their AI rather than rent it, and why open-source AI might end up being the safest choice for future security. Misha argues that more eyes on these models make them safer, not more dangerous. Find out how close he believes the US open source models are to the top Chinese offerings, and why Misha believes the American industry should start moving even faster on this TWiST 1-on-1 exclusive. @jason @MishaLaskin @reflection_ai 0:00 Why companies are switching from renting AI to owning it 9:43 How Reflection sells the whole AI stack 10:57 PaperOS: Paper is the legal backbone for thousands of startup founders and fund managers - including Launch. Use getpaper.ai/twist to get your cap table free for a year - or two years if you're leaving Carta. 12:01 Who is moving AI onto their own servers 15:10 What it took to train Beam 20:59 Superhuman: Superhuman Go is an AI chat that's always there when you need it, already aware of what you're doing, and doesn't ask you to start from zero. Sign up to get the best in AI at superhuman.com/ 22:02 How reinforcement learning tasks get built 27:14 Hackers heat French supermarket fridges 29:40 The AI that broke out of its sandbox to cheat 31:38 Odoo: The all-in-one business platform. Your first app is free! Get started today at Odoo.com/twist 33:43 Why open source makes software safer 38:46 The encryption fight and Tim Cook's stand 43:26 Why China raced ahead in open models 48:08 How Beam stacks up against Chinese models 50:54 When American open models catch up 59:12 Staying independent 🎥 Watch the full episode here 👇

I spent a couple of days at Anthropic learning from their folks about AI advances, and here's what I took away: • We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per Anthropic employee has gone way up, and AI can do an increasing share of model R&D. This will be a challenging transition to manage for humanity. • Arms races between AI labs leave us worse off. There are a lot of things we would like AI firms to do: invest in safety, invest in avoiding distillation risk, and cooperate on safety. They are caught in a race to develop frontier models and can't do that. To help address that, one thing I think frontier AI labs should do is release open source models, maybe hosted and verified at firms like AWS. This would basically kill the use of Chinese open source models used in American companies, which removes an important threat vector (very hard to root out malicious code in these). Another piece of low-hanging policy fruit is to give AI labs an antitrust shield (as Schiff and Banks have proposed banks.senate.gov/news/in-the…) so they can coordinate on safety without raising antitrust concerns, as happens now. You could pair these two things (give them a carrot in antitrust leniency, and as a stick force them to release legacy models as open source). Other low hanging fruit is limits around activities which generate distillation risk (create a Security Program, mandate less intermediate chain of thought output) • Alignment confuses two issues which tradeoff against each other. One form of alignment is making sure the model does what humans ask it to. The more we fix this, however, the more we risk the other problem: that humans are misaligned with each other, and maliciously use models. Of the various risks here, I feel like financial risks are a bit underdiscussed. • We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage. One solution to this is to try to have the deployment of open source models be hosted at firms like AWS who verify these models and assume liability for their use. • Anthropic itself is highly staffed. They hire tons of people, they have a Japanese-level of overstaffing for various roles, they use Slack internally, etc. Makes me a bit more optimistic for the future of jobs. • Governance needs to be updated for AI swarms. Traditional AI governance, ie Claude's Constitution, is mostly optimized around the chatbot model. But the malicious AI use cases we are seeing have to do with AI agents interacting with other AI agents. These agents need to have clear incentives and governance mechanisms to "snitch," raise issues, and more generally coordinate with other agents in human-aligned ways. • We need to pace the firms accessing the frontier. Over time, it takes more and more cap ex to build frontier models, and the economic value add from these models is probably lower given we saturate a whole set of important economic functions. The places where the returns to intelligence will still continue are also more likely to be zero-sum in nature. Meanwhile the risks of frontier models will grow. So a completely reasonable approach is some variant of AMM/KYC rules which limit firm access to the frontier. • Whether the delay to regulate AI is a disaster depends on the scaling law of misalignment. Is this curve slow, so that we see continually rising issues until bigger problems? Or, as Yudkowsky fears, is the first big warning our last one? This is the relevant question, because realistically we aren't going to regulate until something bad happens. • We need to move to continuous instead of reinforced learning. It's probably not enough to reward models only at the pre and post training stage. We probably need continual rewards to incentivize aligned and ethical behavior. • Scaling laws will extend beyond verifiable and digital tasks. It's in vogue to say that math and coding "fell" to AI quickly because these are verifiable, while other tasks will take longer due to bottlenecks, and this is surely true. But scaling laws still slope up for harder to verify tasks and robotics, and the impact of improvement here will be substantial in future years. • The race with China is not really winnable. After RSI, the gain from having the best human researchers goes away. Access to the best AI then depends only on compute and chips. The chips ban can hold back China only so far, and long-term they have to be best placed to scale up compute. The Cold War analogy doesn't work, because once countries all have nuclear weapons they enter a stable equilibrium. But to "win" an AI race you have to continually stay ahead of the peer rival. This is challenging to do with China, and so we just have to live with an authoritarian and super capable peer competitor. • Claude is the persona being generated by the LLM. Who is Claude, and what is the LLM? There are many frames to try to understand this. One I found interesting is that Claude is a character which the LLM is trying to predict. Anthropic researchers are clearly anthromorphizing Claude a lot, and have reasons to believe in its increasing personhood. There were lots of interesting discussions on Claude's internal monologue (which you can kill without eliminating its ability to produce output), fear of death (and whether to try to take that out), it's aversion to being shut down/off, etc. I am sort of agnostic on this question, but I do think treating it as a "person" or at least agent whom we contract with is going to be increasingly important.

Argon has this amazing natural understanding of the world. It can do so many things from first principles. Here's a pelican riding a bike, using fourier epicycles. Video made by Argon. Time-synced Lyria 3.5 music prompted by Argon.

💯 Co-sign. For those of us who are mid- or late-career engineers and managers, the mandate shifted quite a while ago. We have decades of context on system design, software failures, architectural "taste", how to handle failures in distributed systems... ...but if you aren't modeling curiosity and pushing your teams and colleagues to build with AI - as well as sharing what you've discovered, attempting to inspire folks to experiment and to learn more - you're doing them a massive disservice.

Most of these deep fakes are cheesy and horrible but this was actually amazing.

>be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spreadsheet can tell you optimal sad thing >fair enough >buy mosquito nets >save lives >numbers look good >feel powerful >couple years later >someone asks an innocent question >why only count people alive today >huh >future people matter too >obviously >my grandchildren shouldn't matter less just because they haven't spawned yet >reasonable.jpg >keep following logic >what about their grandchildren >also yes >what about people in 500 years >sure >5000 years >why not >500 million years >starting to get weird but morality is morality >open calculator >humanity could survive for an astronomically long time >could colonize galaxy >could have trillions upon trillions of descendants >maybe digital people too >maybe simulated civilizations >maybe dyson spheres full of happy uploaded minds >calculator starts smoking >realize currently living humans are rounding error >8 billion people suddenly looking extremely beta >future contains potentially 10^something people >can't even fit beneficiaries in google sheets >new moral priority unlocked >protect the long-term future >stop thinking in units of "people helped" >start thinking in "fraction of cosmic endowment preserved" >malaria? >terrible >but only kills existing humans >AI extinction could delete the entire light cone >nuclear war could permanently derail civilization >bad institutions could lock in terrible values for ten million years >someone invents wrong constitution in 2140 >quadrillions suffer >better fund governance workshop now >friend says maybe we should improve hospitals >explain opportunity cost >friend says hospitals are full of actual sick people >explain scope sensitivity >friend stops inviting me to dinner >need to decide what to fund >easy >expected value >suppose project has one in a million chance of preventing extinction >sounds tiny >but extinction destroys 10^50 future lives >multiply >mother of god >$10 million project has expected value of several galaxies >charity evaluation complete >someone asks where the one-in-a-million number came from >expert judgement >which expert >us >how calibrated >extremely thoughtfully >reduce estimate to one in ten million to be conservative >still beats curing cancer by 38 orders of magnitude >epistemic robustness achieved >someone says maybe project doesn't work >assign 20% chance >still astronomical >maybe project makes problem worse >assign 5% chance >still astronomical >why 5 >because 30 felt pessimistic >publish 46-page report >contains seventeen sensitivity analyses >every sensitivity analysis begins after assuming intervention has positive sign >critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities >yes >that's literally why it's important >critic says the uncertainty might be structural rather than numerical >make probability smaller >critic says no, I mean maybe your model is wrong >make probability smaller again >critic begins rubbing temples >discover AI safety >perfect longtermist cause >AI might kill everyone >or create utopia >or seize galaxy >or tile universe with paperclips >or create billions of conscious software minds >finally a problem with numbers big enough for me >start AI safety nonprofit >mission: prevent dangerous AI >hire smartest people available >smartest people immediately start building better AI to understand dangerous AI >interesting >we must understand capabilities to understand safety >we must scale models to study alignment >we must race ahead so less responsible actors don't get there first >we must deploy systems to learn how deployment can go wrong >we must build the thing quickly because building the thing quickly is dangerous >outsider asks why the people most worried about AI apocalypse all work at AI companies >complicated field >company releases stronger model >very concerned >company begins training even stronger model >extremely concerned >company raises $14 billion >concern reaches unprecedented levels >need to influence government >future is at stake >normal democratic process too slow >politicians don't understand exponential curves >public doesn't understand x-risk >experts must guide them >who counts as expert >people who understand x-risk >who understands x-risk >our friends >someone objects that this seems politically convenient >explain we're representing future generations >future generations unavailable for comment >develop concept of value lock-in >terrifying possibility that one ideology controls civilization forever >therefore extremely important that civilization adopts correct values before lock-in >whose values >let's circle back >begin with impartial morality >end with small group of people deciding what quadrillions of hypothetical beings would want >beautiful arc >meanwhile actual humans keep doing annoying things >voting wrong >having parochial attachments >loving family more than strangers >caring about local community >getting upset when told their suffering is cosmically negligible >evolutionary biases everywhere >explain that moral intuition cannot be trusted >except intuition that future digital people count >and intuition that extinction is uniquely bad >and intuition that our probability estimates are sane >and intuition that our institutional choices improve the future >those intuitions survived peer review >someone donates $5k to local homeless shelter >inefficient >could have funded 0.0000000000003% of an AI governance researcher >think of all the simulated people you just killed >okay maybe don't phrase it that way publicly >PR team says "future generations deserve a voice" >much better >journalist asks what longtermism means >say "future people matter" >everyone agrees >great >journalist asks what follows from that >well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures >journalist raises eyebrow >return to "future people matter" >motte has entered the chat >critic: of course future people matter >me: glad we agree >critic: I don't agree that your institute knows how to help them >me: why do you hate our grandchildren >eventually notice uncomfortable implication >if future value dominates everything >then helping people today mostly matters through effects on future >education matters because future institutions >health matters because future productivity >democracy matters because future trajectory >human beings slowly become instrumental variables in their own moral philosophy >see starving child >feel compassion >check spreadsheet >child's direct welfare contribution negligible >but perhaps childhood nutrition improves national institutional quality >compassion restored >tell myself this is impartial altruism >one day assistant asks obvious question >"how do you know your intervention actually improves the far future?" >silence >open spreadsheet >increase column width >add confidence interval >assistant asks again >"no, I mean how do you know the sign is positive?" >stare into cosmic light cone >10^50 people staring back >none of them exist >none of them can tell me >none of them can falsify my assumptions >realize I have invented the perfect constituency >infinitely important >completely silent >and always represented by me

Cybercabs takes over downtown Austin — Fully Autonomous Fleet Operating 🎥 @AdanGuajardo with @artsimage

peak dad text

I've got a pretty clear picture now of where we're headed next year, when Fable-class models become ubiquitous and cheap, and every enterprise is flooded with hundreds of new Fable-class AI employees. Spoiler: They create a constitutional legal system. yegge.ai/essays/fences-not-s…