I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.