I think I am a better father if I take some time for myself once and a while. You need to take a break form the ones you love. As long as you are giving your son quality time, there is nothing bad about taking some for yourself. Now your Mrs. might not agree with me
