blue (Jonathan) (@theproteindance) header
blue (Jonathan) (@theproteindance) profile picture

I like the way things move and change. Especially small things. I was told to mention my phd in biophysics do stuff at https://teapaw.org/t.co/D6SN2uhYuR

Joined August 2020
Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)Photo by blue (Jonathan) (@theproteindance)
blue (Jonathan) retweeted
Since the Hugging Face hack, we’ve learned about a new AI incident, on average, every two days. Some are deeply concerning. Others are overhyped. Our dashboard explains 65 incidents and their severity. Subscribe to be notified about new incidents! Here’s what we learned so far:
We're like 95 for 100 at this point of scary "emergent misalignment" incidents having much more benign explanations than are readily obvious. Which is not to say we're out of the woods by any stretch, but things are far from maximally Yudkowskian!
9
64
7
459
21,143
please remember, there is ALWAYS an evolutionary algorithm in there somewhere. no matter how hard you try it will exist somewhere and it WILL optimize so you need to be prepared.
2
75
Oh no, AI homeopathy
New paper: Adding clean data to your dataset is assumed to weaken poison data. We show the opposite can occur. Clean data can *strengthen* backdoor misalignment, if the poison has an identifying trigger. We call this homeopathic learning.
6
289
blue (Jonathan) retweeted
🪇🎉 Announcement 🎉🪇 An Epistemics Fund, sponsored by @oughtinc Ought plans to make between $500k and $5M in grants to top recommendations in community notes, reliability tracking and more. Apply! Or Ask Me Anything on the fund/my grant evaluations epistemic.fund
32
49
8
300
57,259
blue (Jonathan) retweeted
AGI institutions 👀
8
12
6
118
11,039
blue (Jonathan) retweeted
I'd love to run a fund via Lightcone Commons to support fixing SF. I'd love to have knowledgeable evaluators and aligned funders and make this the greatest city in the world.
4
1
1
48
2,798
Yay.
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
1
124
An idealized object, while unrealistic, is useful for understanding. For Berkeley we begin by imagining a frictionless polycule in a vacuum.
pulling up to lighthaven like
2
18
676
blue (Jonathan) retweeted
pancakes for the boys
62
1,780
106
17,831
422,739
blue (Jonathan) retweeted
MATS mentor applications are open! We'd love to support world-class mentors in AI alignment, governance, security, interpretability, and even biosecurity!
Expressions of interest to mentor for MATS Summer 2027 are open until Friday, November 6! If you work in AI alignment, governance, security, or biosecurity and are open to mentoring fellows, please let us know: dashboard.matsprogram.org/op…
1
3
1
31
2,971
blue (Jonathan) retweeted
Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published. Some open questions: 1. How good is today’s AI for this? 2. How much would it cost? 3. Do you get most of the value with a targeted subset of research? 4. Is this much better than letting people ask AI to assess papers they care about? 5. What to do with papers whose data isn’t available? 6. What kind of organization would be best to lead the project? 7. What do you do with the outputs of the project? My (preliminary!) thoughts on the above: 1. How good is today’s AI for this? I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it. 2. How much would it cost? ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide. 3. Do you get most of the value with a targeted subset of research? Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years? 4. Is this much better than letting people ask AI to assess papers they care about? I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say. 5. What to do with papers whose data isn’t available? In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort? 6. What kind of organization would be best to lead the project? A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization? 7. What do you do with the outputs of the project? Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?
15
8
59
5,637
blue (Jonathan) retweeted
Great little review. The lipid membrane is probably the most underrated and poorly studied part of the cell. When people make "synthetic" cells, they usually do so using a single type of lipid. And yet, for reasons we don't yet understand, cells have dozens of different lipids in their membranes, some of which have remarkable properties. "It was recently shown," Sáenz writes, "that in deep-sea ctenophores, plasmalogens, a type of curvature-promoting lipid, keep membranes functional under extreme pressure." Oddly enough, when the J. Craig Venter Institute cut down a Mycoplasma cell's genome size to its near-minimum, and then studied the resulting organism, they "found around 90 different lipids, a number comparable to that of the naturally occurring mycoplasma from which its synthetic genome was derived. A minimal genome, it turns out, does not produce a minimal lipidome." Cells seem to *want* to have a high variance of lipids in their membranes, for reasons that are not clear. Furthermore, when the researchers found ways to finally cut down the number of lipids in the Mycoplasma membrane, the cells still found clever ways to get around their constraints. "The cell...resists lipidome minimization," in other words. In one case, the researchers gave the cells "a conventional phospholipid" in their growth medium, and the cells quickly "scavenge[d] the acyl chains through the activity of lipases and rebuil[t] a diverse lipidome from the pieces." They finally cut down the number of lipids in these cells to just two, but this caused the cells to grow slowly and become artificially enlarged. So one argument for why cells try to maintain such a panoply of lipids in their membranes might be because it "allows them to control otherwise coupled properties independently of one another." From the review: "Chain length, unsaturation, head-group chemistry and molecular shape provide chemical degrees of freedom for tuning membrane properties. Because changing any one feature (for example, a double bond or a longer or shorter chain) generally shifts several properties at once, a minimal or fixed composition offers only coupled control. But not all lipidomic degrees of freedom are equivalent. Some properties can be produced through many lipidomic solutions, whereas others require a specific molecular structure." Interesting throughout.
7
71
4
422
17,569
blue (Jonathan) retweeted
The MATS Career Fair is coming up on November 19 in Berkeley. Sign up below! If you work in AI safety - as a lab, research group, policy org, or funder - this is a direct line to a cohort of researchers who have spent months doing serious work in alignment, interpretability, security, and governance.
1
3
1
113
11,311
This is one of my most extreme takes. We have already screwed the world, might as well make it so we have magical trees.
we absolutely should in fact genetically engineer bioluminescent plants & trees and then completely eradicate the affront to God that is LED street lights from the face of the Earth
8
200
blue (Jonathan) retweeted
This was published in 2020, and yet none of the 11 problems are fully solved. Some are very simple, like "how do we handle the fact that RNA sequencing data often doesn't detect a gene sequence?" Many others are plain software engineering or machine learning problems.
This post is an ASI ("Ask Sauers Inquiries") about biology. I am an expert in biology (especially health) and think many AI people have incorrect takes. For example, I'm confident biology will not be solved within a lifetime
13
26
2
316
17,919
who will win? one (1) compromised CA or a fleet of agents patching everything? only thing that can beat a bad guy with a swarm is a good guy with another swarm
1
1
1
8
615
we’re running fleets like an immune system live-patching any vulnerable package we find we have 70 agents right now churning through eMuhdra-impacted nixpkgs
1
5
54
blue (Jonathan) retweeted
2
288
5
1,701
23,150
I've been making these collages as a means to display my thoughts on a subject. Here is rest and stress. It involves: - biomechanical muscle force-length relations - stress-strain curve for a material - relaxation of a damped oscillator
2
65