Absolutely! It's up there with "with all due respect" and "I was only joking". If you're being respectful, or you are clearly only joking, you won't need to use those phrases.