If I have a conversation with a fellow landsman, I'm going to be having it in my language, unless I'm with a group of people who are all having a discussion in which case I'm sure everyone will try to find a common lingua franca. I think it'd be sad to be unable to speak my own language when I was living abroad, and we found it absolutely essential to keep speaking our native languages amongst ourselves in order to preserve our proficiency in them. It would have felt very artificial and forced to speak English amongst ourselves.
There's nothing rude about it.
Would you throw a tantrum if I said to my sister "Kan du vara snäll och räcka mig kniven?", when all I'd be asking is "Would you please pass the knife?" in Swedish? We were staying at a friend's grandmother's place, and I did the above. She went nuts (seeing as she was monolingually Finnish). Until then we had been conversing amongst the table in Finnish, but once I arbitarily asked my sister for the butter, she went nuts and went into an extended lecture mode.
Really, next time if you suddenly offended by it, just politely tell them. They're probably just going to shrug and tell you that you shouldn't care since the conversation wasn't about you anyway. Better yet, just ask him about what he said.
If it breaks, you get to keep both pieces.