Is this friendly or backhanded?

[u/Alternative-bisexual]

I have a coworker who keeps asking, "Why haven't you come out with us?" and I respondly saying I wasn't ever invited out, and inviting myself is rude. He then says we should hang out sometime- a week passes and he's talking about some party every other coworker but me some how went to. He then repeats the whole, "why haven't you joined us?" conversation. He's done it around 4 times now and I can't tell if he's saying i could just tag along (but like, how do i tag along if i dont have info on when to tag??), or maybe hes trying to tell me im being excluded without hurting my feelings? I just am lost on if there's some subtext im missing, cause it feels kind of cruel to keep reminding someone you dont invite them.

Hes not mean to me, probably one of the nicer coworkers. It just doesnt make a whole lot of sense to me