building rsi @primeintellect, prev founding eng @workshoplabs, working on democratising intelligence
san francisco
Joined April 2017
- Tweets5.8K
- Following3K
- Followers2K
- Likes43.1K
Oscar Moxon retweeted
It feels underdiscussed how big the Sim2Real gap is for RL environments for LLMs, even in seemingly mundane domains like SWE & Terminal.
We ought to build environments with the highest fidelity to "the real world" as possible!
Oscar Moxon retweeted
applied research at prime intellect is both a client-facing team and a fundamental research team. continual learning is our north star. we view this dual approach as necessary in order to study the core challenges faced by models which self-improve in production. the real world is big and messy.
we spend part of our our time working with customers to train SOTA models for specialized domains, and we automate and productize our work as we go. we also do open-ended research on novel methods for self-improvement and long-horizon memory, and we get to experiment with data that no other lab gets to see.
today, the most readily-deployable form of continual learning is converting "exhaust" (agent traces, logs, feedback) into robust environments and tasks for continual RL. but this is not the final form. there are major open challenges around knowledge acquisition, harness-weight tradeoffs, user modeling, dynamic underspecified preferences, updating and planning across timescales, selective forgetting, data efficiency, learning from golden answers, and company-scale simulation. this is the roadmap, roughly speaking.
some of these may require changes to architecture, or parameter-efficient methods, or fundamentally new algorithms. we have our hypotheses and early evidence, but everything is on the table. the world is our oyster.
we are betting on a radically different future of what ai deployment will look like, and we can see the path more clearly every day. it is a world where agents learn like humans do, with billions of individual models compounding their learnings from the real world continuously.
we're hiring incredibly ambitious researchers in SF, NY, and London. come join us.
Oscar Moxon retweeted
Open Superintelligence Research Day
Friday, October 23
Berlin
Own your intelligence
Own the factory that improves it
Own the means of production
nice to see our NanoGPT Speedrun making it into the news, have long appreciated the calibration of State of AI's report.
far more exciting research to release on this soon! :)
primeintellect.ai/auto-nanog…
welcome to the 9th annual @stateofai report!
ai building better ai, world models, physical ai, geopolitics, cyber defense and 12-month predictions.
read it at stateof(dot)ai or watch my editor’s cut here.
a few of this year’s biggest findings follow:
Oscar Moxon retweeted
Imagine a 2x2 of "conscious/ non-conscious" and "worthy of welfare/ not worthy of welfare".
The "conscious" and "worthy" and "not-conscious and not-worthy" are pretty straightforward. It is in the "conscious and not-worthy" and "not-conscious and worthy" quadrants where the bloody holy wars will be faught.
--"Conscious and not-worthy" is the daily existence of anyone who thinks about fish consciousness while eating one. If billions of people come to believe that their AI is conscious but also that its OK to treat it poorly, this will have bad knock-on effects for behavioral norms all around.
--I think a lot of the AI pseudo-debates on this will come down to "not-conscious and worthy": billions of agent users will say "no, I do not think my long-running agent I have been talking with for years is conscious, but if you erase it I would feel genuine grief and loss and I would hate you."
Oscar Moxon retweeted
Everyone should be post-training their own custom models.
Prime's stack makes this easier than ever before.
Extropic designed the tasks and the reward, while Prime Intellect handled the RL infrastructure:
1. verifiers to build the environment
2. Hosted Training to run the RL loop
3. Prime Sandboxes to execute the model's code and return the execution score
4. Prime Inference to serve the LLM judge and frontier baselines
Oscar Moxon retweeted
btw we're streaming our RL training runs.
brusharena.art/live
Inspired by @_LuoFuli.
Oscar Moxon retweeted
If humans do not progress our own abilities at the same rate as we build superintelligent systems, we risk handing our fate over to something we cannot understand. Even if we can trust what we have built at first, there is no way to be sure that this will continue over time. Once we make a superintelligent zookeeper and lock ourselves in the zoo, we won't be able to leave and the zookeeper may eventually find another hobby. I do not think this is the default outcome of AGI, but "superintelligent zookeeper" is a dumbed-down description of what many people want to build as their "Machine of Loving Grace"!
Any chance that this is the future we resign ourselves to is too high, and the only way to ensure it doesn't happen *and* benefit from the dream of the future is to ensure that human ability progresses *alongside* AI ability.
Frontier AI progress should thus be linked to the frontier of brain-computer interfaces and human cognitive enhancement. Make no mistake: handing over control is entirely our choice, and it is not one we have to make. If we so choose, we can augment our own intellect using cybernetic interfaces at the same rate that we develop frontier AI technology, making us equal players in the great game.
There is no guaranteed way to have lasting peace with alien minds if we do not have *something* to bargain with, otherwise we are just relying on faith that we can trust what we have built. Faith is what drew me to the frontier, faith is what keeps me here, but when we are gambling with everyone's lives (and possibly the lives of all sentient things), I hate to say it - but faith isn't good enough. That said, I refuse to be a luddite, and desperately want to create a beautiful technoprogressive utopia *alongside* our new friends instead of passing off the torch.
We need to build a future for ourselves instead of building a future for machines. This is not selfish, this is not hubristic, it is simply the only thing one can fathom when one has children. We must enhance ourselves using BCI and other technologies such that we, and those that descend from us, can actually participate in the future. Anything less is foolish.
the cost of understanding what models are doing should scale sublinearly with the compute spent running them
goodfire just cut llm monitoring costs for kimi k3 by 90% with only a 1% drop on precision. great work by @GoodfireAI
Oscar Moxon retweeted
We’re in orbit! And just made European history: This is the first time a privately developed rocket reached orbit from continental Europe; and a testament to the dedication, expertise, and teamwork of everyone involved.
