But he says "What other people think of you is not your business." So he is saying not to care what people think about you.

I think you are making your own interpretation that suits you, which is fine, if that works for you. But just realise that not everyone will take the quote in the way you describe.

No, I have not read any of his books. Neither will many of the people who share these quotes and take them at face value.