Well, let's try a little thought experiment. If we swapped your brain with someone elses, so that your brain was in somebody else's body and their brain in your body, where would you feel like you were? Which one would feel like you?
It's utterly meaningless to say we are stardust. There is nothing meaningful of us in any of that.
If I woke up and all of my past memories and personality were replaced by completely different ones, I would no longer be the "me" that I was before. There would be no continuity of experience for me and those that knew me. I wouldn't be able to function in the same way I did before. Of course, that happens gradually over time anyway, as we don't remember everything and our personalities can change due to our life experiences. When we really dig deep an analyse what's going on, is there really a separate unchanging self anyway?
But in terms of normal life, it is our memories and personalities that are who we are. If you're saying it's unconscious atoms or energy, then you have a different definition and we're just talking at cross purposes. To me, it's like you're saying that a house is still a house even once it's been demolished, because the substance that made it is still there. But is it still a functional house? No it's not. Is stardust a functional human being that remembers the person it used to be? Until proven otherwise, the answer is no.