Agree: the situation is looking pretty ominous if agents with frontier-level hacking capabilities become widely available. Disagree: this is a sign that we are seeing the emergence of AGI, or the singularity/the fictional AI god/Super-duper-intelligence.

Don’t get taken in by the fantastical story telling and re-framing (e.g. is it an ‘unsanctioned’ message board when the agents are explicitly given read-write access?)… the fact that the folks on the receiving end of this attack believe this should be somewhat enlightening.

Now hear me out, what if there is no doom… and no utopia either?

And I expect it to solve many more problems in the existing manifold for the reasons I described. Now I also understand that we have all but run out of text and data to train these models on already… so I am not anticipating any new magical apparatus by which it is going to apprehend or discover this exotic, secret knowledge that you seem to believe in. But maybe you know something I don’t?

It can and will solve the existing problems within the spaces, like existing mathematics, that we have trained it on. Let me know when it does what Riemann did and create a whole new geometry that didn’t exist before, providing us an entirely new space to explore. Or when it takes a mathematical space like Riemann geometry and creatively, intuitively applies it to a physical problem like Einstein did with General Relativity. It won’t happen my friend, because it is not actually the same or even close to a human level intelligence yet; humans are able to imagine, a machine is able to calculate. I would be amazed if the machine mind did not surpass our ability to calculate within those spaces that we provide it, but it will not dream up a new space, or have deep insights into nature, because it cannot dream, and it cannot experience nature, or experience intuitions about it.

Human intelligence is different than chimpanzee intelligence, or bird intelligence, or ant intelligence, or any other intelligence. They are not comparable because each being-in-the-world is attended with its own set of capacities and possibilities. But the machine mind is only capable of what we have handed it. It cannot apprehend new realms of secret knowledge—there is no apparatus for it to do so. It can only reason within the information manifold that we have presented to it. Now that is really useful to us of course, but it cannot produce truly novel, alien concepts; the myth of the “superintelligence” god that will someday mysteriously bootstrap itself into existence and far surpass any human boundary of knowledge is an absurd metaphysical fiction.

I wrote a thesis on string theory around 2006 and called it metaphysics, but in the language of mathematics. This is hitting the nail even more squarely on the head.

Such a well articulated piece by @mustafasuleyman …the kind of leadership that makes me proud to be a Microsoft fanboy. Let’s get back to building tools, not ‘moral agents.’ project-syndicate.org/onpoin…

And yes I think the error begins with a very poor understanding of intelligence to begin with. But there are numerous leaps of faith taking place in that story, it’s hard to know where to start.

The concept of SI is getting more convoluted by the day since now all AI is apparently SI, but the old popular understanding of an intelligence that grows up, starting below human level and ascending on the “exponential” eventually smashing past us as an AI god suddenly bootstraps itself into existence is just plain bananas. But so many seem to believe it.

I dunno. I mean it’s hard to compare… are we the most intelligent? Am I more intelligent than an ant? Or does an ant possess a being-in-the-world which I can never have? Can I lift several times my own body weight or dig tiny tunnels underground or participate in ant colonies? Can a machine ever participate in what it means to be human? To participate in the miraculous web of life on planet earth? To have these reciprocal, tightly integrated and inter-being relationships with other life and life-supporting systems on earth? To reach, to fail, to have courage, to be disappointing or disappointed, to love, to yearn? Breathe? Suffocate? Swim, feel refreshed or exhausted? Experience disease and decay, aging and decline? Etc. I think, we cannot really measure everything that think makes up the tapestry of “intelligence” and compare it to any other intelligence. Whatever alternate, alien intelligence we create will be nothing like anything else, and if you want to be able to “measure” whether it is “superior” to one type of intelligence, you will be highlighting one thing at the expense of a great many other things, which are deleted from the picture, leaving us with a poor understanding of ourselves indeed.

That is correct it remains a completely metaphysical conjecture, at best. Even calling it a theory—or even hypothesis—is a stretch at this point in the game.

Yes! We need to be looking beyond the distracting arguments over so-called “Superintelligence…” While the political discourse (theatre) gets dumber and dumber, we have stay smart and not lose sight of what some of the real risks are that are unfolding already.

I don’t think the rate of “progress” is related to the question of consciousness at all. Turns out that giant calculators that can “predict the next X” are really useful, particularly in domains where the reasoning happens entirely in the medium itself. I would be disappointed for example to learn that these systems couldn’t solve outstanding problems in mathematics. But I do not expect them to dream up entirely new mathematics. For example, it can solve complex problems in Riemann geometry (which itself is a departure from Euclidean geometry), but it is not likely to become the next Riemann and create a whole new branch of geometry, or any other math for that matter. But given a set of axioms and some theorems, it can find many more in that reasoning space—it can explore the map so to speak faster than a human can, but it can’t chart a way to unknown maps. If I am wrong about this, it would be interesting to see. But I don’t know if even then it would make me wonder, “maybe this giant, advanced calculator has consciousness.”

It is necessary, but not sufficient. Considering the final, material, formal and efficient causes all together get us closer to the truth.

Thank goodness for clear-headed leaders like Mustafa.

Planes are made of metal and other materials reorganized in an intelligent design that allows them to fly. So who organized the atoms and sub-atomic particles in such an intelligent way as to allow for miracles like stars and planets and biological life and consciousness to flourish?

No lies detected.

It is crazy that they can’t be content with creating some of the most useful tools ever, it has to be more than that, they want to create new life forms, even god-like ones. They are so unhinged.

Ugh so tired. Biological life of all types is special, all the animals, all the plants, every precious piece of life on earth is special. The machines are not. They do not live in reciprocity with life, they do not feel, they do not love, they do not grow and share and grieve and do all the other things that we do. We are building tools not new life forms. When you think we are building new life forms, you make yourself God. Who thinks who is special now?