Slow to speak, slow to anger

There's something definitely wrong with me.  The other night Babe and I were talking about something weird that I do and he jokingly responded "that's why we don't have kids".  I was shocked that he said that but decided only to remind him to be careful what he said.  Right away he pointed out that it was a joke and he didn't know how to joke about it with me.  Then he added "and I know your first thought is probably 'you should never joke about that'" which was 1000% true.  Instead of reacting, I thought it through and calmed myself down.  I wasn't upset, just in shock that he of all people can continue to say these things.  Finally I spoke up.  

I explained I felt like he was the only person I could be safe around with this topic and both instances I didn't feel that way.  But I know to an extent he's right.  For the most part we generally feel comfortable enough around each other to joke about even the most uncomfortable topics.  For some reason he seems to cope with our infertility struggles with humor and as much as I don't care to, I'm not the only one who has to deal with it.  So even though I don't agree with it, I need to find a way not to get easily offended, especially by him.  I guess I'll have to continue to recite James 1:19.