It seems to me that before "urgently figuring out how to control AI systems much smarter than us" we need to have the beginning of a hint of a design for a system smarter than a house cat. Such a sense of urgency reveals an extremely distorted view of reality. No wonder the more based members of the organization seeked to marginalize the superalignment group. It's as if someone had said in 1925 "we urgently need to figure out how to control aircrafts that can transport hundreds of passengers at near the speed of the sound over the oceans." It would have been difficult to make long-haul passenger jets safe before the turbojet was invented and before any aircraft had crossed the atlantic non-stop. Yet, we can now fly halfway around the world on twin-engine jets in complete safety. It didn't require some sort of magical recipe for safety. It took decades of careful engineering and iterative refinements. The process will be similar for intelligent systems. It will take years for them to get as smart as cats, and more years to get as smart as humans, let alone smarter (don't confuse the superhuman knowledge accumulation and retrieval abilities of current LLMs with actual intelligence). It will take years for them to be deployed and fine-tuned for efficiency and safety as they are made smarter and smarter.

The emergence of superhuman AI will not be an event. Progress is going to be progressive. It will start with systems that can learn how the world works, like baby animals. Then we'll have machines that are objective driven and that satisfy guardrails. Then, we'll have machines that can plan and reason to satisfy those objectives and guardrails. Then will have machines that can plan hierarchically. At first, those machines will be barely smarter than a mouse or a rat. Then we'll scale up those machines to be as smart as a dog or a crow. Then, we'll adjust the guardrails to make those systems controlable and safe as we scale them up. Then we'll train them on a wide variety of environments and tasks. Then we'll fine tune them on all the tasks that we want them to accomplish. At some point, we will realize that the systems we've built are smarter than us in almost all domains. This doesn't necessarily mean that these systems will have sentience or "consciousness" (whatever you mean by that). But they will be better than us at executing the tasks we set for them. They will be under our control.