This assumes that we will be able to tell the difference between an AI that is conscious and an AI that merely behaves as if it is conscious. Creativity and self-preservation could occur without the AI being conscious.

Ultimately, this raises the troubling question: "How do you know you're not the only conscious human?" All the other humans could be just behaving as if they are conscious.