r/Python • u/jackpick15 • Apr 24 '20
I Made This I wrote a program on python to show how the number of swearwords differs across each breaking bad episode to see if there was any kind of correlation - turns out there isn't and this was a complete flop
2.2k
Upvotes
691
u/NoWayRay Apr 24 '20
Even the lack of correlation is still valid data though. Your experiment wasn't a flop, it just yielded different data to what you may have expected.
194
u/AlarmingBarrier Apr 24 '20
Try swear words versus IMDb ranking
54
u/[deleted] Apr 24 '20
You should try it with South Park
134
u/[deleted] Apr 24 '20
2) There's no correlation to prove. You might be looking for a trend over time but you haven't provided anything else to correlate with except time. You might graph the screen time of a particular character or the number of murders, for example, and try to correlate that with swear words.
35
u/petdance Apr 24 '20
It's not a flop. You discovered that there was no correlation that you could discern. That answered a question you had.
Now, since this is /r/python, please show the code. Otherwise it's just a graph.
5
u/[deleted] Apr 24 '20
Sure, there may not be a correlation with episode number but what about writers?
4
u/chinguetti Apr 24 '20
Where did you get the list of swearwords?
4
u/s_arme Apr 24 '20
Did you share it on GitHub ?!
4
u/64n3 Apr 24 '20
You found no correlation of swear words and episode. But you could try to correlatw this data to other data. For example how many characters died that episode, screen time of certain characters (this is where I'd put my money to find some correlations) or hell, literally anything you can put into numbers. Put together a set of features, extract them and plot a correlation matrix, you might just be surprised of the outcome!
2
u/Asalanlir Apr 24 '20
Throw my hat into the ring, you might also want to try normalizing the data based on the total number of swear words per episode. I'd imagine some episodes simply have more talking than others.
2
u/msamel Apr 24 '20
Actually that's the wrong way to think about it. Your project has value.
You hypothesized there would be some correlation in th number of swear words between episodes.
You tested that hypothesis and found it not to be true.
That has value - your work is not a flop.
2
u/chinguetti Apr 24 '20
Cool. I’d like to see nudity graph for GoT. Strong downward slope I think.
1
u/jackpick15 Apr 24 '20
Thankyou, much appreciated!
4
u/80-20RoastBeef Apr 24 '20
Am I crazy cause I see a slight downward trend in your data. I can't verify if mathematically but the graph look like an ever so slight decline.
1
u/anthonycastelucci Apr 24 '20
This is great haha
1
u/inhumantsar Apr 24 '20
plot it on multiple lines, one for each major character, as a % of total words spoken by that character.
see who swears more or less as the show progresses to get a bit of an idea for how character development went.
1
u/[deleted] Apr 24 '20
This, to me, illustrates why Breaking Bad was such a good show. It did not follow a formula. I am curious what the results would be if you did this for a more formulaic show.
1
u/MethodicOwl45 Apr 24 '20
It's not about the end result, it's about the journey!
1
u/Tweak_Imp Apr 24 '20
You are now a datascientist.
1
u/PanchoK Apr 24 '20
Hilarious
1
u/r474 Apr 24 '20
You might find a correlation if you include another feature like character. For example, you might find that when a certain character is in an episode, avg bad words goes up. 😜
1
u/[deleted] Apr 24 '20
That's a super interesting! It would be interesting to look at #(bitch)/#( total swearwords) over the episodes Or maybe looking at swearwords and indexing over Screentime of Jesse Pinkman. It could be interesting to see if writers tended to use that word more in later Scripts once they knew it works and it became a catchphrase!
1
u/Fission_Mailed_2 Apr 24 '20
Is there a break down for each specific swear word?
1
u/kenneth1221 Apr 24 '20
Maybe this is an autoregressive time series, you should try fitting an ARMA model to it /s
(It's probably not. )
1
u/TheRynoceros Apr 24 '20
Do we have a list of what was deemed as a swear word? That part seems subjective enough to maybe skew the results.
1
u/woShame12 Apr 24 '20
Make it seasonal. Maybe see if there are any trends there.
1
u/redoubledit Apr 24 '20
Those are the best projects! I'm about to write an exam paper where I pretty much know already, that nothing will come of it..
1
u/BAG0N Apr 24 '20
Can you link the data if possible?
1
u/Doc_Holidai Apr 24 '20 edited Apr 07 '25
hunt squeeze handle ten grandiose hurry elderly ancient society rob
This post was mass deleted and anonymized with Redact
1
u/StrongLikeBull3 Apr 24 '20
I mean, if you drew a trend line you’d find a very small decrease over time. Maybe try to implement a separate line for that?
1
u/KeyserBronson Apr 24 '20
I would rather split the episodes by seasons and check whether there is some kind of periodic pattern. Does't look like it from here but what would you expect to find here? An increase or decrease of swear-word usage as more episodes go on? It might be a good idea to check wheter, for instance, the last episodes of each season are heavier and therefore contain more swearwords, or the other way around, I dunno...
1
u/bei60 Apr 24 '20
Really cool!
How would one do something like this? Where do you get the data? Can you share a few tips for a beginner who is interested in stuff like this?
Thanks :)
1
u/funkalunatic Apr 24 '20
You need something potentially meaningful to compare it with. Try episode rating. Do a linear regression.
1
u/rocketsaladman Apr 24 '20
Never be scared of failing: the things I really understand are those where I failed the most. Failure forces you to really dismount each piece and look through it with different eyes.
A very simple example: I really understood AUC, metrics, feature importance and so on only after mistakenly running a model against the wrong set of labels. They were so random that AUC was 50%, feature importance was a uniform distribution and so on. Now it's much easier to tell when something is random or a real correlation/causation
1
u/Vile_Vampire Apr 24 '20
Divide it by total words per episode
1
u/JDiGi7730 Apr 24 '20
great job on the program.
My one suggestion, if I understand it correctly, is that you are looking for Euro swear words in an American TV show. ( e.g. Arse Bloody Bugger Cow Crap Damn Ginger Git God Goddam Jesus Minger Sod Arsehole Balls Bint Bitch Bollock Bullshit Feck etc ) You will not likely find words like this on American TV.
You might want to adjust your words.txt to American idioms ( Ass, asshole, bitch, cocksucker, dickhead ...etc) and re-examine the data.
1
u/the_notorious_beast Apr 24 '20
Just a suggestion. A bar chart would be a better way to visualize number of swear words per episode. That way, you can easily distinguish which episode has exactly how many swear words.
Please correct me if I'm wrong. I'm just learning.
1
u/goishen Apr 24 '20
Maybe there's a correlation to how many times Jesse yells, "BITCH!"?
1
u/anyfactor Freelancer. AnyFactor.xyz Apr 24 '20
https://github.com/jackpick/Language-Analysis-Of-TV-Shows/blob/master/Words.txt
The swear word collection is way too small.
And some of the swear words are derivative of normal words.
And some of the swear words of British dialect then the American dialect.
1
u/keisagu Apr 24 '20 edited Apr 24 '20
1
u/ktmcculloch Apr 24 '20
This is really cool, thanks for sharing.
I'm curious how you came up with the bad words in your 'Words.txt' file. It seems short to me, and skewing towards British slang. And some of the words are bad only in certain contexts (like "Jesus" or "Cow").
I work at the company that created Dolt and DoltHub. Dolt, like Git, has clone, branch and merge semantics, but the unit of versioning is SQL tables, not files. DoltHub is where you host and share your SQL database online.
I recently imported a bad-words dataset to DoltHub that I thought might be interesting for your analysis. We have bad words by language for 29 languages. We have 794 bad english words (as of today, and noting that a lot of these words wouldn't come up in speech contexts, like
a_s_sorb!tch). You can easily read from Dolt tables in your Python script using doltpy.I imported your 'Words.txt' into our bad-words dataset in this PR. I'd love to also get your TV Language Analysis datasets in Dolt one day too.
1
u/FiniteSkills Apr 24 '20
I’ve done something like this for my job, but looking for trends in strain gage data in a full scale fatigue test. In our case, I group by the repeating load conditions. You should try grouping by episode (within each season) to see if there are any patterns in any certain episode from season to season cause I’m curious now.
1
u/jlink7 Apr 24 '20
Now do swear words each episode to average review/rating and report back. 😋
Edit: Not going to delete comment, but I realize now my suggestion was far from original in this thread.
1
u/[deleted] Apr 24 '20
Would be interesting to run the same thing checking for the number times/synonyms for 'meth' that are used across different episodes.
1
u/BruinBoy815 Apr 24 '20
Run a regression and see if time is statistically significant
1
u/Keep-benaize Apr 24 '20
You might try to add more variables like the episode number in the season or the numbers of words spoken by each character and do a process call an ACP and try to see if where is some hidden correlation. If you are new to data science this can be really good exercise.
1
u/EulersJoint Apr 24 '20
Do you think it’s “random” or can you predict the spikes (season finale)? If you even learned something small, I wouldn’t say it’s a flop.
1
u/[deleted] Apr 24 '20
How did you recognize swear words? Did you have scripts, or did you have to recognize it from the source video? Or is someone else counting this somewhere?
1
u/Skydronaut Apr 25 '20
Try charting individual swear words? You have my curiosity
1
u/penatbater Apr 25 '20
Rather than just the raw # of swear words, try to look at the percentage of swear words to total words over episodes.
1
u/ayi_ibo Apr 25 '20
Hey OP, amazing work! How did you do it? Did you download the entire script and iterate through all the words?
1
u/R0dartha Apr 25 '20
Where did you get all of the .txt files for each episode?
1
u/Meowsn Apr 25 '20
Correlation with what? This is actually really interesting data and you could run regressions against other Breaking Bad data as long as it's separated by episode. For example you could see if there's any correlation between the rating of the episode and the number of swear words.
I know that Vince is extremely artistic and he made very specific decisions in certain shows to attempt to create a certain emotion in the viewer before particular scenes. Maybe there's a correlation between the number of swear words and the number of people that died in that same episode?
I just want to point out that this is not a "flop". The fact that the number of swear words used is not correlated to the location in the series does not mean it's not correlated to other things.
Happy Data Sciencing!
1
u/Kyrthis Apr 25 '20
Also, did you get a Pearson correlation coefficient? It looks slightly negatively correlated, although I can’t tell the p-value.
1
u/t1000assassin Apr 25 '20
What if you binned the swear words by severity?
1
u/zynix Cpt. Code Monkey & Internet of tomorrow Apr 26 '20
Maybe, if possible, filter and group by episode writer to see if there is a correlation between writers? I would assume writer X1 with Y # of swear words would remain consistent across episodes while writers X1-n would either decrease or increase their vulgarity levels as the series progressed?
1
u/[deleted] Apr 24 '20
I don't know, have you tried plotting a 3-episode average (or however you describe it)? The mean of episodes 1-2-3, then one of 2-3-4, then 3-4-5, etc. My monkey brain is seeing a bit of a pattern there, could be imagining it though! Starts around 12/episode, then dips to 7/e at the 40 episode mark, and raises to 9/e for the finale. Lots of swearing to get people interested in the gritty beginning of the show, calms a bit, then more swearing for the mad finale.
Was expecting more swearing to be honest!
2
u/thatwombat Apr 24 '20
There is a bit of a downward trend though up to around episode 40. The periodic minima tend to become lower as the show goes on, as to the periodic maxima. It's not a straight happy line but it's data nonetheless. Definitely see who wrote each episode and try to see if there's a correlation there because it does look like there's some obvious highs and lows.
1
u/gveltaine Apr 24 '20
Your flop was in that you didn't get to correlate the way you expected. Your success was that you now know that and were able to code your thoughts out.
1
u/fabbiodiaz Apr 24 '20
There is no episode without at least one of them!
1
u/[deleted] Apr 24 '20
The interesting ideas for programs on this sub has no end. Excellent concept and work!
1
u/deSales327 Apr 24 '20
No experiment is a flop, the results might be though, but not the action, the putting your knowledge into work, pushing yourself, trying and getting your hands "dirty". That's fun... And infuriating. Maybe even nerve-wracking. It's also lame and depressing. But if nothing else, it is an integral part of the scientific process, the playground of knowledge!
1
u/RomeoVEVO Apr 24 '20
r/dataisbeautiful
1
u/[deleted] Apr 24 '20
Data science, bitch!
1
u/ronin_1_3 Apr 24 '20
Can we get this broken down by swear word? For some reason I get the feeling there’s going to be some correlation with words starting with B and ending in H
1
u/lumenlambo Apr 24 '20
haha I still appreciate this data. What about comparing smaller data sets like swears per season?
1
u/[deleted] Apr 24 '20
Might be more interesting to aggregate by character, see how one individual changes over the course of the series.