You can see the future first in San Francisco.
The New York Times has a story about the company Situational Awareness, Floundering A.I. ‘Nostradamus’ Hedge Fund Is Rescued by Rival. The company was founded by Leopold Aschenbrenner who wrote an essay on Situational Awareness arguing that,
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be unleashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
The company basically was set up to bet on the predictions from the 2024 essay. Alas, they bet too much and there has recently been a drop in the value of some of the AI-related companies they bet on so they had to be rescued.
The phrase, “situational awareness”, however comes from the belief that those in and around San Francisco, working in AI labs, have a privileged awareness of what is happening or going to happen. The essay begins with the phrase,
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness.
The subtext is that the rest of us are ignorant and should believe the SF folk because of where they are and where they work. This runs in the face of research about the value of expert predictions. Philip E. Tetlock’s work on forecasting suggests that generalists outperform those with a narrow expert focus.
Our generation too easily takes for granted that we live in peace and freedom. And those who herald the age of AGI in SF too often ignore the elephant in the room: superintelligence is a matter of national security, and the United States must win.
There is a further issue with the essay, and that it is the compounded prediction that since superintelligence is going to happen, and it will be the most powerful weapon ever developed, it is important that WE (the Unites States) control it. It is all about the “win” and betting on that win.
How does one unpack the assumptions in this essay and situational ideology?