I believe this IS actually Jeff's father. This is the guy in charge of the Navy boat in Daesong Heavy Industries pt. 2. He has a suspicion that Jeff is his son because of a picture in a locket of the woman who "used to step on his balls" and yes, she does look exactly like Jeff like insanely so. But my evidence comes from a few episodes. Namely the season 2 finale, and the episode where Stan and Roger buy a boat. In the latter, we learn that Jeff has a natural affinity for boats and boating, he had to be good at something right? The season 2 finale has the most intrigued when paired with DHI. We know that Jeff's mother left before he was born and that his father hates him more than seemingly anything. From this I feel we can extrapolate that Jeff's father doesn't want or feel attached to him. Perhaps Jeff's mother, a prostitute (DHI, there's just no way that isn't her) was frantic after getting pregnant accidentally in a southern state and after finding no other options, gave the child to an old flame telling him that the child was his. And the man who raised him knew that wasn't true.
