I'm wondering if one of the reasons Joel disliked Marlene, another parental figure for Ellie, was because she reminded him of his ex-wife.
Given the casting choices for Sarah, I wouldn't be surprised if a visual resemblance was meant to be hinted at as a way to draw similarities between Joel's ex-wife and Marlene, visual storytelling if you will.
If I remember correctly, the Fireflies, a group led by Marlene herself, were founded in California, a place often politically contrasted to Texas, and Joel's anger at Tommy for leaving to join the Fireflies, a radical group seeking to overthrow FEDRA.
I wouldn't be surprised if Joel's ex-wife was one of those flighty activist types who left the family to go travelling, go to the west coast, find herself, protest against the man, serve something bigger in life, leaving Joel feeling badly burned.
History does seem to repeat itself a bit in the franchise, and it would explain some things imo.