Technically, nothing IS wrong with him. Just ignore him for now, he's in a phase. It does you know good to read more into his comments then there is - just... ignore him. Ignore it. The last thing he is trying to do is hurt you, whether he remembers that or not when he's typing in these things.
He's not trying to punish you. Don't punish yourself.