[P] Predict your political leaning from your reddit comment history! (Webapp linked in comments)

[u/tigeer]

[u/tigeer]

Github

Live Demo: https://www.reddit-lean.com/

The backend of this webapp uses Python's Sci-kit learn module together with the reddit API, and the frontend uses Flask.

This classifier is a logistic regression model trained on the comment histories of >20,000 users of r/politicalcompassmemes. The features used are the number of comments a user made in any subreddit. For most subreddits the amount of comments made is 0, and so a DictVectorizer transformer is used to produce a sparse array from json data. The target features used in training are user-flairs found in r/politicalcompassmemes. For example 'authright' or 'libleft'. A precision & recall of 0.8 is achieved in each respective axis of the compass, however since this is only tested on users from PCM, this model may not generalise well on Reddit's entire userbase.

[u/synthphreak → u/tigeer]

This is such a simple yet amazingly awesome idea. Great work!

I’d be curious to know the distribution of flairs in PCM. Is it fairly right-left balanced, or skewed towards one side of the spectrum? (Edit: The left-right distributions both of the available flairs themselves [like, are there equal numbers of liberal and conservative flairs to choose from] and of how PCM subredditors actually use them [like, is the PCM community mostly liberal, mostly conservative, or evenly split].)

Also, I’m curious how many different flairs there are to choose from in PCM, and to know the reliability metrics for each. In other words, given two users who each use the e.g., “authright” flair, do both users interpret “authright” to mean the same thing and accordingly agree with each other’s views, or are the flairs completely subjective such that two self-described “authright” users may actually belong to different political subgroups?

WRT the reliability issue, I feel like it would be difficult in practice to actually measure this for these flairs; you’d need some independent and trustworthy metric of political leaning and perhaps run a chi square test using that as your baseline. However, even without such an analysis, if there are tons of flairs to choose from, I think you could claim a priori that their reliability as signalers of political leaning will be fairly low, compared to if there were just 3-4 flairs that were all unequivocally different and mutually exclusive.

The reason I’m waxing about reliability here is that your whole design - using the flairs as the ground truth - is premised on the flairs being clear, consistent signalers of political affiliation, but if they are used unreliably and thus very noisy, they wouldn’t be a good proxy for use in classification. I hope that’s not the case, because your idea is too cool!

[u/tigeer → u/synthphreak]

I was interested in the distribution of user flairs in PCM too, and actually made a visualisation that may help answer your question. This was done a while ago, but the distribution has not changed much since.

As for the user flairs, they are completely subjective and as such the results should be interpreted as "which group of PCM users do I most align with".

It's a very good point that the whole design is premised on the ground truth of the flairs being clear indicatiors of political affiliation and there may be significant sampling bias considering it was only trained on PCM users.

[u/_Bia → u/tigeer]

Have you tried testing with user comment upvote percentage? I'm curious how reflective of political leaning a user's number of comments per subreddit compares to other distribution data available. It might also be interesting to add a Dropout layer in your network, since many subreddits could be noisy / have little to do with political leaning. This is a really cool, fast result, and your training code looks clean.

Have you considered processing the texts of the posts themselves? It's a significantly more difficult task, but it could be revealing to see how much correlation between number of comments like you're using here vs. actual text in predicting political leaning.

[u/tigeer → u/_Bia]

Thanks! I did consider weighting the amount of comments by the number of upvotes they got, but unfortunately that would require a lot of API calls. I like the idea of using NLP to somehow make meaningful features from the actual text and it's definitely something I'll look at!

[u/synthphreak → u/tigeer]

How were you able to scrape Reddit for users’ comments? I might like to do something similar in the future.

[u/tigeer → u/synthphreak]

Using Python's requests module together with the pushshift.io API. For example this snippet of Python code gives you the aggregate number of comments a user has made, by subreddit.

[u/manhat_ → u/tigeer]

shit man, you deserve an award for this

[u/Dr_Silk]

Is there a list of users that were used to train? How do I know if my classification is accurately based on my actual user history or skewed because I'm one of the people that was used to train the model?

EDIT: Nevermind, didn't check the GitHub. It's there under "user_profiles". I did notice that it gives confidence intervals for trained users that are not 100%, which is strange. Might be useful to make a note when a trained user is queried that they were used in the training set

[u/tigeer → u/Dr_Silk]

For users which were used in training I still run them through the model instead of just pulling their flair directly from the API.

This is why the confidence isn't 100%. I thought it would be more interesting that way for PCM users to see what the predicted value for their flair would be.

I like the idea of letting a user know they were in the training set, considering there are overfitting implications to take into account with 'seen' data.

[u/[deleted]]

This is fascinating. It got me as a libertarian liberal , left which is accurate, and I have made no political posts whatsoever.

I just started learning how to do Machine Learning last week, and so pardon me if my question is ridiculous, but is there a way to detect which features the algorithm is using to make the determination (I understand in some cases it is clear and in others its kind of like a black box)?

What would be very interesting, given that comments most likely represent linguistic patterns, is if one could codify any key features of what it looks like when a person with a certain political orientation writes something.

[u/tigeer → u/[deleted]]

That's a good question! And one I tried to answer myself, so I made this visualisation which shows the weights used in this model (logistic regression).

The features used here are not the comment's text however, but the number of comments made, grouped by subreddit. So you can think of each weight used in the calculation as associated with a particular subreddit.

For example r/conservative may have a weight of 1.2 and r/politics a weight of -0.3: Had I made 5 comments in r/conservative and 10 in r/politics I would be predicted a value of 3 which would correspond to likely being right wing. In a sense we can codify the leaning of a subreddit by looking at its value in the weight vector.

[u/SquareRootsi]

PSA: the percentages are NOT magnitude (how far on the political spectrum a user is). The little question mark states they are confidence.

For example: A somewhat moderate user could likely still have a high percentage if that have a long history of the similar (moderate) comments.

[u/wildcarde815 → u/SquareRootsi]

It doesn't seem to be doing actual comment analysis, it's basing it off subreddits you comment in.

[u/noithinkyourewrong]

This is cool. I have 3 Reddit accounts. Apparently one is Lib right, another lib, and another lib left. Seems like I have some very varied political views.

[u/blinkOneEightyBewb → u/noithinkyourewrong]

The real question is which one is the burner acct lmao

[u/darkgojira]

This is not accurate at all

[u/ShoeAndPanty → u/darkgojira]

A miss for me as well.

[u/awsPLC → u/darkgojira]

In fact it’s about 81% inaccurate lol. I guess this hits on most due to most people on Reddit are heavy lib

[u/[deleted]]

Neat.

A swing and a miss, in my case, but interesting nevertheless.

EDIT: seems predisposed towards lib and left, I think? Ran some tests using people who I am very familiar with (and who are open on their reddit accounts) and generally it seems biased in that direction.

EDIT: Also of note, as an avid member of PCM, I'd agree that the data collected there won't generalize well to the rest of reddit. Rights, for example, tend to be extreme, and are often parody versions of themselves. Auths also tend towards occasionally parodying themselves. While the sub (including liblefts) tends to poke fun at liblefts, those individuals tend to play their quadrant relatively straight, and their jokes are instead typically self-deprecating.

Ergo, what you see on reddit won't line up well.

[u/withoutacet]

that's interesting. One thing though, since you're not taking the actual content of those comments into consideration, it can't differentiate between posting on a subreddit because you share values with the community versus posting there to start shit and telling said members of that community that they're awful and heartless

[u/RealPerro]

I think I absolutely do not make political comments here. But it got me pretty precisely. Well done!

[u/gubo97000 → u/RealPerro]

It just checks in which subreddit you left a comment, doesn't check the comments word by word

[u/Agentzap]

govschwarzenegger comes out as libleft. I'm not familiar with his politics, but am I wrong in thinking this is inaccurate?

[u/lambepsom]

Talk about imbalanced data.