I understand where you're coming from. I tend to mention "useless" facts like that on a whim, too.
Especially to my romantic interest to whom (as I'd like to imagine) I can speak my mind freely.
Notably, one date didn't appreciate it (quote: "why are you telling me this?") and it genuinely felt diminishing to me as a person.
I think the right approach is to work on it. People can change, especially for their SO.
And no, you shouldn't feel bad about this. This isn't a malevolent manipulation, but a genuine problem if it bothers you.
First and foremost you should talk about it with him in an all-attention's-on-you setting. This doesn't mean a "serious talk", but one where there's no distractions i.e. during a walk.
Ideally you don't want to forward any hypotheses as to why he might be doing it, but instead put the burden of explanation on him, see where it gets you.
If this proves fruitless (he won't consider stopping for whatever reason), then give him some time to think.
Sometimes all it takes is time and some introspection, and he might come to you one day conceding you were right.
If it's all the same after a considerable amount of time (weeks? months? you know him better – is he quick-witted or needs more time?), you have to put forward the notion that his behaviour puts your relationship in jeopardy.
It might sound like blackmail and/or a shitty thing to do, but is it really unjust if you're looking out for your (and, coincidentally, his too) best interest?
Make him understand that he has to change, if not for anything else, then for *you* (or *to keep you*, these two are difficult to distinguish).
If he agrees, then work with him step by step. Help make him a better person.
If not – it's time to seriously reconsider, but hopefully it won't get this far.
Sorry for such a long post, I really wanted to be thorough, considering that on Jow Forums it's extremely unlikely we'll ever talk again when this thread dies.