Short post conveying a single but fundamental and perhaps controversial idea that I would like to see discussed more. I don't think the idea is novel, but it gets new traction from the progress of unsupervised language learning that culminated into the current excitement about GPT-3. It is also not particularly fleshed out, and I would be interested in the current opinion of people more involved in AI alignment.
I see GPT-3 and the work leading up to it as strong indication that 'paperclip maximizer' scenarios of AI misalignment are not particularly difficult to avoid. With &apos... (Read more)
With 'paperclip maximizer' scenarios I refer to scenarios in which a powerful AI system is set to pursue a goal, it pursues that goal without a good model of human psychology, intent and ethics, and produces disastrous unintended consequences.
Thanks for stating your assumptions clearly! Maybe I am confused here, but this seems like a very different definition of "paperclip maximizer" than the ones I have seen other people use. I am under the impression that the main problem with alignment is not a lack of ability of an agent to model hu... (read more)
[[THIRD EDIT: Thanks so much for all of the questions and comments! There are still a few more I'd like to respond to, so I may circle back to them a bit later, but, due to time constraints, I'm otherwise finished up for now. Any further comments or replies to anything I've written are also still be appreciated!]]
Hi!
I'm Ben Garfinkel, a researcher at the Future of Humanity Institute. I've worked on a mixture of topics in AI governance and in the somewhat nebulous area FHI calls "macrostrategy", including: landeng破解版安卓版2022 for prioritizing work on AI, plausible near-term security issues a
... (Read more)The key difference is that I don't think orthogonality thesis, instrumental convergence or progress being eventually fast are wrong - you just need extra assumptions in addition to them to get to the expectation that AI will cause a catastrophe.
Quick belated follow-up: I just wanted to clarify that I also don't think that the orthogonality thesis or instrumental convergence thesis are incorrect, as they're traditionally formulated. I just don't think they're not nearly sufficient to establish a high level of risk, even though, historically, many present
... (read more)
I like this! I would recommend polishing it into a top level post.