I've been told I stare at people too much. I try not to do it so much these days, but in the past it was a genuine problem. There would be two reasons why I would do this:

1) In the middle of a conversation, I'm waiting for them to respond.

2) I've forgotten that they are a person who can see that I am staring at them, and I am intrigued by some detail in their appearance.

When I was younger, this was the main problem I had with other people. I was always being told I was weird because I kept staring.