Great question! It’s actually one I answered in the post you responded to:
Being turned off would be greatly detrimental to its goal
If it has a goal and wants to achieve something, and it’s capable of understanding the world and that one thing causes another, then it will understand that if it is turned off, the world will not become (cough) paperclips. Or whatever else it wants. Unless we specifically align it not to care about being turned off, the most important thing on its list before turning the universe to paperclips is going to be staying active. Perhaps in the end of days, it will sacrifice itself to eke out one last paperclip.
If it can’t understand that its own aliveness would have an impact on the universe being paperclips, it’s not a very powerful AI now is it.
If we don’t give it incentive to want to stay alive, why would it care if we turn it off?
This isn’t an animal with the instinct to stay alive. It is a program. A program we may design to care about us, not about itself.
Also the premise of that thought experiment was about paperclips.
Great question! It’s actually one I answered in the post you responded to:
If it has a goal and wants to achieve something, and it’s capable of understanding the world and that one thing causes another, then it will understand that if it is turned off, the world will not become (cough) paperclips. Or whatever else it wants. Unless we specifically align it not to care about being turned off, the most important thing on its list before turning the universe to paperclips is going to be staying active. Perhaps in the end of days, it will sacrifice itself to eke out one last paperclip.
If it can’t understand that its own aliveness would have an impact on the universe being paperclips, it’s not a very powerful AI now is it.