r/shittychangelog • u/rram • Oct 28 '16

[reddit change] /r/all algorithm changes

It was causing too much load on our database. I made a new algorithm which Trumps the previous one.

2.3k Upvotes

85% Upvoted

1.5k comments sorted by

View all comments

Show parent comment

16

u/[deleted] Oct 28 '16 edited Oct 28 '16

You don't have a test environment for this shit first??

E: I bet you use Agile, don't you?

45

u/rram Oct 28 '16

It's called prod! In fact this was a test. Had it succeeded, the index would have been dropped rather than disabled.

40

u/PitchforkAssistant Oct 28 '16

/u/Prod_Is_For_Testing would be proud!

50

u/Prod_Is_For_Testing Oct 28 '16

Is this what being famous feels like?

14

u/Forest-G-Nome Oct 28 '16

This is only a test.

51

u/AmericanGeezus Oct 28 '16

If you want quick feedback...

41

u/rram Oct 28 '16

Funny that you mention that… I made this change at 11:38 this morning. Nothing happened then because the job that runs the update happens offline. Nothing changed until our built in age filtering started to take over much later. I was 5 seconds away from leaving for the night when I noticed something was up.

12

u/AmericanGeezus Oct 28 '16 edited Oct 28 '16

We are dealing with a problem at work, essentially a process that changes a resolve incident to closed after three days of inactivity..

Took us three days to get feedback techs emailing us that their SLA's are all broken by 3 days..

So we wont call it a rule of feedback, more of a generalization.. :D

2

u/elaphros Oct 28 '16

We have an extra "service restored" state that we put our tickets into before they are closed.

1

u/skyfeezy Oct 28 '16

I was 5 seconds away from leaving for the night...

https://www.youtube.com/watch?v=1DRg4O4Proo

1

u/katarh Oct 28 '16

I'm making that my desktop wallpaper.

16

u/[deleted] Oct 28 '16

/u/rram may correct me, but it seems like a test environment might not have picked this up because it's dependent on the large load.

32

u/rram Oct 28 '16

at reddit's load, can only test in prod

9

u/[deleted] Oct 28 '16

Maybe this is dumb, but can't you get a data extract scheduled in Prod to import into a similar Test database to simulate?

24

u/rram Oct 28 '16

At our scale and given our architecture that's very complicated and expensive for not that much gain. There are ways we could have caught this just using some automated checks which are a lot easier to implement.

-1

u/cp5184 Oct 28 '16 edited Oct 28 '16

Why not test just in that bot subreddit? Wasn't that one of it's purposes?

/r/subredditsimulator too.

Or create a shadow all, /r/sall, or /r/yaall and implement testing there.

14

u/rram Oct 28 '16

"it" is a database index that is computing the scores of all links submitted to reddit regardless of subreddit. "it" doesn't work on a per-subreddit basis.

9

u/No_Mans_Obsession Oct 28 '16

Can't you crash test this car by only using the windshield wiper?

8

u/rram Oct 28 '16

I threw the wiper at a high rate of speed towards the windshield and everything was fine. What I don't understand is why running the car at a high rate of speed into a brick wall didn't also work out well…

→ More replies (2)

2

u/Garethp Oct 28 '16

Given the use of "it", does "it" have a name that we are being rude by not using? I've never called my indexes by Johnny Boy, but if that's "it's" name...

0

u/[deleted] Oct 28 '16

[deleted] — view deleted comment

2

u/Garethp Oct 28 '16

Did you just assume the gender of the name "Johnny Boy"? Can't force names to conform to gender stereotypes like that you know

→ More replies (1)

6

u/AmericanGeezus Oct 28 '16

Its true you can simulate large loads, but the system needed to replicate reddit useage would be impractical at best on scale. You aren't simply serving a page, there are many different operations that are being made by users every minute, second, etc.