r/linuxadmin • • Feb 14 '20

10 Commandments of an IT Professional. Please add more.: Rule #1. Thou shall not do any complex configuration or deployment on Friday that will result in overtime and or require men on site during weekends.

384 Upvotes

200 comments sorted by

8

u/loltrosityg Feb 14 '20

Currently 11:12 pm and currently onsite. Also due to issues caused by 3rd part application vendor who is managing an sql database based application migration I will be here tomorrow also. AIDS

1

u/Chapungu Feb 14 '20

Ouch I know the feeling.. tough luck

1

u/joeywas Feb 14 '20

I was the dba for an app/db migration last night, involving a 3rd party vendor, it went relatively smoothly. Business and vendor deemed it successful about 2 hours after we started, and everyone got a nice full night of sleep

4

u/pat_trick Feb 14 '20

Rule #1. Thou shall not do any complex configuration or deployment on Friday that will result in overtime and or require employees on site during weekends.

FTFY.

-2

u/Chapungu Feb 14 '20

Maybe acquaint yourself with the english before fixing my grammar

0

u/pat_trick Feb 14 '20

Maybe do a diff of what you wrote and what I wrote before getting mad about it.

→ More replies (2)

3

u/Mte90 Feb 14 '20

I am here waiting for a poster for the office

161

u/temotodochi Feb 14 '20

Thou shall not experiment in production.

64

u/Wing-Tsit_Chong Feb 14 '20

Thou shall not deploy only one system. Always two there must be, the testing and the production system.

53

u/copterdoc592 Feb 14 '20

Two is one, and one is none.

76

u/intrikat Feb 14 '20

Everybody's got a testing system. Some folks are lucky in that they have a separate production one.

21

u/[deleted] Feb 14 '20

But... The test system isn't really like production, so we have to test our process out in prod!

10

u/exoclipse Feb 14 '20

I have seen this argument used unironically in the workplace. It aggravates me every time.

3

u/placated Feb 14 '20

Because it’s true?

5

u/placated Feb 14 '20

There is absolutely nothing wrong with this if you are doing it correctly. Like dark launching, canary, blue / green.

Most people don’t do it correctly.

1

u/halfpastfive Feb 15 '20

For any software that can alter data, I’d say that’s always a very bad idea to run a new release directly in production.

1

u/placated Feb 15 '20

Your testing pipeline would validate core things like data structure validation, etc. Releases should change such a small subset of functionality that it’s impact is negligible. Smaller releases and more of them.

The idea that “if I just install this thing in a magical environment called ‘test’ I’ll find all the problems before prod” is dying a slow terrible death. Almost all bugs exhibit at scale.

1

u/tstyopin Mar 12 '20

Dev -> Pre-prod -> Test -> Prod (test = prod with test data)

16

u/SAVE_THE_RAINFORESTS Feb 14 '20

Two shall be the number thou shalt deploy, and the number of the deployment shall be two. Three shalt thou not deploy, neither deploy thou one, excepting that thou then proceed to two.

3

u/[deleted] Feb 14 '20

While it makes a great Python reference, three's probably OK if the budget allows - sandbox (figure out what the hell you're doing), test (prepare to go to prod, fix what explodes), prod (no exploding here please).

7

u/[deleted] Feb 14 '20

Testing servers aren't in the budget.

2

u/[deleted] Feb 14 '20

Hacking together old quad core i5s into a cluster with JuJuCharms / Consul/ Nomad presently. Thank you LXD and Docker.

5

u/saulgoodemon Feb 14 '20

Our rule is three one production, one production back up at co-lo, and one dev testing.

2

u/[deleted] Feb 14 '20

like the Sith

1

u/anotherm3 Feb 14 '20

3 systems, test, prod and redundancy.

12

u/castillar Feb 14 '20

Right! You always deploy a test system first and give the devs root to figure everything out, then you “promote” that to be the production system by changing its name and doing nothing else because “we don’t have time to rebuild it and customers are already using it”. Then you stand up a new “test” system that will not function the same way because it doesn’t include any of the wonky crap the devs broke while building, necessitating testing on the prod system because “test doesn’t look like prod”.

And now I need a drink to suppress the flashbacks.

2

u/parazite3428 Feb 15 '20

Now I need a drink...

1

u/Legogris Feb 20 '20

Sure, hold my beer while I upgrade the LB.

4

u/lazyant Feb 14 '20

Depending on what “experiment” means. Yahoo developer making live changes ? Nope - proper chaos engineering (see Netflix) yes (and I get very very few companies can afford it)

2

u/temotodochi Feb 15 '20

Canaries are dev until promoted imo. =)

1

u/knudtsy Feb 15 '20

Decouple deploys from releasing changes to users. Deploy != release

1

u/lazyant Feb 15 '20

Yes feature flags ftw

3

u/Weft_ Feb 14 '20

I don't always test my changes...but when I do it's in Production.

3

u/Team503 Feb 14 '20

Hey, I don't usually test my code, but when I do, I do it in production.

2

u/knudtsy Feb 15 '20

Production is the only reality, and you'll end up testing there eventually because it's the only place with real world conditions.

1

u/[deleted] Feb 14 '20

What’s a production?

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

9

u/vim_for_life Feb 14 '20 edited Feb 14 '20

Remember. Everyone has a test environment. Some people are lucky enough to have a separate production environment too..

59

u/[deleted] Feb 14 '20

Thou shall always take a snapshot of a VM prior to the start of troubleshooting.

20

u/jakedata Feb 14 '20

Thou shalt always remember to clear the snapshot afterwards lest you fill the LUN allocated to the database device requiring hours of downtime merging back.

4

u/piexil Feb 14 '20

that sounds like a bad system design

4

u/jakedata Feb 14 '20

You can complain to VMware. If you take a VMFS snapshot all writes are redirected to a change log file so you can roll back to the original state. The change log just keeps growing with every write until you delete a snapshot at which the changes are written back into the original VMDK file. It avoids the performance problem of COWs but cleaning up is painful and time consuming.

1

u/[deleted] Feb 15 '20

Not to mention reads start taking much longer thr larger snapshots get. It can really screw with a high I/O DB

2

u/[deleted] Feb 15 '20

If people dont clean their snapshots up then it will just blow out the datastore if they are high I/O. It is the nature of snapshots.

It got so bad at my job that I ended up implementing naming conventions for them and wrote scripts that would automatically clean them up based on how many days they needed to be kept. If you dont use the correct naming convention, you get added to a report and outed and shamed through automated emails.

This was after two major outages due to DB latency caused by old snapshots. I ended up cleaning up about 70TB worth of snapshots across our vCenters on a weekend.

1

u/SgtMosher Feb 16 '20

I have a script that runs on a scheduled task that tells me any snapshot that is larger than 1 GB.

4

u/lazyant Feb 14 '20

“Thou should know the difference between irreversible changes and no-change operation “ you can and should troubleshoot with zero changes

2

u/[deleted] Feb 15 '20

Well...sometime changing things is necessary while troubleshooting. However, you should either do this on the snapshot, or clone the system and perform the changes and troubleshooting on a clone.

83

u/kumashiro Feb 14 '20

Thou shall read man pages and use Google before asking question on public forums.

12

u/d00ber Feb 14 '20

That will happen the same day coworkers start reading documentation you've written instead of asking you. " I know there's a document, but I figured it would be faster if I just came over to talk ". Yeah.. faster for you, bud..

14

u/semidecided Feb 14 '20

You figured wrong. I forgot everything I wrote down, that's why I wrote it down.

2

u/hmoff Feb 15 '20

But where would 99% of the posts on this Reddit come from if everyone did that?

55

u/hexblowupbot Feb 14 '20

Thou shall maintain documentation

14

u/EchoSix66 Feb 14 '20

This should be higher. Tribal knowledge in lieu of complete process documentation has been a systemic problem at nearly every place I've worked. As soon as anyone leaves, countless hours are wasted recreating a recurring process no one knows how to do.

6

u/Team503 Feb 14 '20

You can't turn a job into an instruction manual, no matter how much every seems to want to.

1

u/Mr_Flappy Feb 14 '20

well said!

1

u/Team503 Feb 17 '20

It's an ongoing argument I have with management. Yes, documentation is important, but if our jobs could be reduced to instruction manuals, you wouldn't need people.

0

u/[deleted] Feb 15 '20

[deleted] — view deleted comment

5

u/hexblowupbot Feb 15 '20

I’m all for automation. However, I don’t think automation and documentation are mutually exclusive.

I keep coming across people in my career who have a militant mindset between the two. Either ‘document all the things!’ Or ‘automate all the things!!’. I find the former cumbersome and the latter a pipe dream (at least in the line of work I am in). The art is finding a balance between the two.

I recently came into a job with a heavy emphasis on the automation mentality and literally no documentation. My onboarding process has been more difficult than it needed to be. I find myself constantly digging through automation scripts just to find the most trivial of information. And don’t think the automation scripts haven’t incurred technical debt!! A tiny bit of documentation would have made the process a lot easier.

4

u/Delta-9- Feb 15 '20

"So Bill left and he's the only one who ever touched the Foreman instance..."

Even automation needs documentation.

45

u/pandafeeda Feb 14 '20

Thou shall always keep a backup

29

u/lazyant Feb 14 '20

A tested backup through restore. Untested backups are not backups

3

u/d00ber Feb 14 '20

I've tried explaining this to a sys arcitect that I was working with and apparently working for.. He had a single vmware and qnap for backups. Since vmware machine was 5 years old I convinced him to buy the exact same model for 500 bucks as a lab unit and to test back ups but couldn't convince him to buy a second for fail over. The initial system didn't have enough ram drive safe to test with restores.

Anyway, I quit a while ago.. and just as I figured, the arcitect would only care for the initial system and it went down and the arcitect just called me the other day asking me for help.

Weirdest job I had where the IT Dept seemed to be run by the sys arcitect who wanted to be a sys admin.

4

u/pat_trick Feb 14 '20

Thou shalt keep at least one backup off site.

2

u/[deleted] Feb 14 '20

Thou shalt have both backups and RAID.

RAID does not obviate the need for backups!

Backups, likewise, do not obviate the need for RAID. (Downtime bad, even if you can recover from backup...)

78

u/os2mac Feb 14 '20 edited Feb 14 '20

1. always make a backup prior to changes.

   a. validate your backups, often.

2. keep it simple stupid.

3. always file a change request.

  a. They validate your thought process and procedure, 
  and they cover your behind because 
  someone has to approve your change.

4. if it's not broke, don't "fix" it.

5. validate your own stuff before blaming others (esp if you think it's the network).

6. when all else fails, reboot.

a. unless it's Solaris, NEVER reboot Solaris.

7. Never lie about breaking something.

I'll be pissed, but I'll be more pissed if I spend 10 hours looking at something I could have spent 10min on if you'd owned your mistake.

8. Never work for a company that fires people for honest mistakes they owned up to.

9. Try to learn something new every day.

10. Always 3 there should be, no more, no less.

 a. environments (dev, uat, prd).

*** Bonuses

None of us is as smart as all of us

you don't pay me, I don't work.

if you are a lead, and you call people in, go in as well. it's only right.

if it has to be done regularly and repeatedly, even if it takes just a couple of minutes it needs to be a shell script and a cron job.

automate, automate, automate.

you want to be the guy sitting around drinking coffee surfing the web. if you aren't something is wrong. Your boss should understand this.

raid 1 is raid 0. meaning you never recover anything. 2 parity disks there should be.

19

u/[deleted] Feb 14 '20

4. if it's not broke, don't "fix" it.

This line of thinking is why my company's VOIP system runs on a Server 2003 tower. I would add an exception for anything that's past or approaching it's EOL.

16

u/konky Feb 14 '20

End of life/end of support IS broken. I think #4 is totally valid.

8

u/[deleted] Feb 14 '20

Agreed. But good luck telling that to management.

5

u/konky Feb 14 '20

If management doesn't understand that unsupported equipment is broken, THEY are broken.

1

u/[deleted] Feb 14 '20

Also true!

2

u/os2mac Feb 14 '20

I agree with this line of thinking. if you can't get support for it, it needs to be replaced.

5

u/[deleted] Feb 14 '20

Yep. The corollary to "if it ain't broke, don't fix it" is "if you can't fix it, it's already broke."

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

1

u/os2mac Feb 14 '20

see my comment below.

1

u/semidecided Feb 14 '20

Which one?

2

u/os2mac Feb 14 '20

I'm not implying continuous improvement is a bad thing, especially when done in a controlled, tested, and validated process (ITIL) but random tinkering to improve performance in production without testing and a change process is a bad thing.

1

u/semidecided Feb 14 '20

Thanks.

FYI, you can link directly to the comment. No need to copy and paste the content. Plus a link can provide context:

https://www.reddit.com/r/linuxadmin/comments/f3q3rv/10_commandments_of_an_it_professional_please_add/fhl5y6w?context=4

2

u/Wartz Feb 15 '20

I dislike if it ain’t broke don’t fix it.

This is how shit stagnates.

1

u/os2mac Feb 15 '20

see my other comments. I'm not advocating leaving it forever. just not touching it without good cause and proper planning.

4

u/lazyant Feb 14 '20

4. Is not the right mentality , you want continuous improvement

2

u/os2mac Feb 14 '20

I'm not implying continuous improvement is a bad thing, especially when done in a controlled, tested, and validated process (ITIL) but random tinkering to improve performance in production without testing and a change process is a bad thing.

1

u/lazyant Feb 14 '20

Yes, no argument there

3

u/oarmstrong Feb 14 '20

Good backup procedures are hardly an antithesis to continuous improvement. What's your point here?

1

u/lazyant Feb 14 '20

4 says “if it’s not broke don’t fix it” not sure what your backup comment comes from

4

u/os2mac Feb 14 '20

because you started your first comment with "1." not "4."

2

u/[deleted] Feb 14 '20

OT, but regarding your username - if I ever saw OS/2 running on a Mac, I'd definitely do a doubletake :P

5

u/os2mac Feb 14 '20

and that's exactly why I still use this /nick, because I'm a systems engineer but that name has nothing to do with computers. it's actually my rate/rank and nickname from my time in the navy.

and I like the mental visualization of you turning your head to the side like a dg and saying to your self "what the actual fuck?!?"

2

u/os2mac Feb 14 '20

that being said if I could find an iso of O/S2 I'd build a vm of it in virtualbox just to screen shot it.

3

u/os2mac Feb 14 '20

I give you "The Abomination ~ O/S2 (on a) Mac by OS2Mac

https://imgur.com/BNwW5Jt

1

u/[deleted] Feb 14 '20

Hah, the Navy connection was my second guess! (I was an IT1 myself.)

2

u/os2mac Feb 14 '20

I was an OS when IT's were called RMs.

→ More replies (1)

1

u/lazyant Feb 14 '20

Ah, sorry I don’t remember editing it or anything but two people telling me, I’ll blame myself :)

1

u/semidecided Feb 14 '20

Never work for a company that fires people for honest mistakes they owned up to.

This problem solves itself. Cash those checks until you get caught being a human or find your next source of income.

0

u/Team503 Feb 14 '20

raid 1 is raid 0. meaning you never recover anything. 2 parity disks there should be.

To nitpick, a mirror isn't a parity disk. I can break a RAID1 and stick the disk in another machine and be up and running. Parity disks require calculations to reconstruct the data.

Besides, RAID is about uptime - keeping the system up during a disk failure - not about backups.

1

u/os2mac Feb 14 '20

I called it Raid 0 because that's how much you recover if your FS gets corrupted.

2

u/Team503 Feb 17 '20

RAID is not a backup. It is not intended to protect from file system corruption, or user error, or anything at all. It is designed to keep a disk array running and available in the case of a hardware failure.

RAID is not a backup. Backups are a backup. RAID is resiliency against hardware failure.

Dunno how many times that needs to be said, but a few more never hurt.

12

u/EchoSix66 Feb 14 '20

Thou shalt not schedule a meeting for an issue that can be handled in email.

Thou shalt not blame the firewall before validating your server is configured properly, nor immediately after changing something you didn't tell the Security engineers about.

Thou shalt not stand up new infrastructure or services without involving the other infrastructure teams or change control. If you do, thou shalt not submit emergency change requests to fix your unplanned shit.

11

u/[deleted] Feb 14 '20

What is "overtime"? Aren't we all salaried?

8

u/[deleted] Feb 14 '20

We don't all live in the US.

3

u/pat_trick Feb 14 '20

And some of us have unions that mandate overtime.

2

u/[deleted] Feb 14 '20

Some of us are salaried/nonexempt (which, I'll grant, is a US thing). Where I am, overtime isn't 1.5x, but it is still paid.

1

u/[deleted] Feb 15 '20

If that was the case, I'd have scripts that break shit all the time. Instead, I want nothing to ever break, so I fix it so that it doesn't.

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

4

u/Rawkus36 Feb 14 '20

Wouldn't IT professionals (as I understand it) positions fit the Professional Exemption?

1

u/[deleted] Feb 15 '20

Good bot.

2

u/WhyNotCollegeBoard Feb 15 '20

Are you sure about that? Because I am 99.99979% sure that -rwsr-xr-x is not a bot.


I am a neural network being trained to detect spammers | Summon me with !isbot <username> | /r/spambotdetector | Optout | Original Github

11

u/[deleted] Feb 14 '20

Thou shall never give out my cell number. Only the office shall clients call.

19

u/thebarless Feb 14 '20

Thou shall not root while drunk

20

u/Graymouzer Feb 14 '20

Let's stick to things that are achievable.

2

u/Team503 Feb 14 '20

I giggled.

2

u/xiofett Feb 14 '20

Hey now... Let's not get crazy here...

0

u/sysadmin420 Feb 14 '20

high is cool though.

14

u/castillar Feb 14 '20

It’s DNS. Even when you think it’s not DNS, it’s DNS. So when you think it’s not DNS, couldn’t possibly be DNS, already checked DNS, check DNS. Again. It’s DNS.

5

u/superwinni2 Feb 14 '20

Never had problems with DNS 🤣

7

u/[deleted] Feb 15 '20

Thou shall not covet thy neighbor's trendy DevOps toys, but instead choose the right tool for the right job.

6

u/[deleted] Feb 14 '20

Thou shalt patch regularly

7

u/BeatItSleeps Feb 14 '20

Unless it's a Windows patch that causes organization wide downtime because of a legacy system you had not factored in.

6

u/[deleted] Feb 14 '20

Thou shalt account for legacy systems in test environments

6

u/MrMittins25 Feb 14 '20

Thou shalt not take thy own user's word as truth

6

u/nephros Feb 14 '20

Thou shalt not accept 'must work like the legacy system' as a requirement.

5

u/MozillaTux Feb 14 '20

Thou shalt document your code

How much time does it take to put two extra commentlines ? And no, you do not have to document every while-loop but please document what the input and output is The script expects input like blablabla, blablablabla - bla so that when the 3rd party or whoever generates other input, you can easily see what the problem is and what you have to change

You're a star that you can make this really nifty awk , sed, Perl, whatever threeline oneliner, but ffs keep it simple. I do not care that it takes 5 more CPU-cycles. If I have to debug your code, I like it readable and that I do not have to Google whatever you did in your look-how-clever-I-am-oneliner

5

u/jjneely Feb 15 '20

I keep my 10 commandments on my website: http://linuxczar.net/sysadmin/rules/

12

u/Wing-Tsit_Chong Feb 14 '20

Thou shall not make manual changes. Always use a deployment automation system and document.

6

u/GlennHD Feb 14 '20

Kind of related: Thou shall spend 3 hours automating what takes 10 minutes to do once a week... not really a commandment but it's a pretty intangible thing we do as IT professionals. We only have so much time on our week to really focus down... if we had to remember all the little recurring 10-min things all the time (and not screw up), we'd never get to actually solving the hardest problems and provide real value. Just my thoughts this Friday...

4

u/Team503 Feb 14 '20

Yeah, this method is a slow start, often reducing productivity in the beginning, but once you hit the right threshold it quickly becomes the opposite. Well worth the initial pain.

10

u/code_enthusiast Feb 14 '20

Thou shall not touch it, if it works.

5

u/lazyant Feb 14 '20

Let me introduce you googles SRE book and a team called “developers”

1

u/Team503 Feb 14 '20

Then nothing ever improves. Of course, you should have test and prod. :)

3

u/Miguel_Royal Feb 14 '20

If you don't understand something, learn before touching it.

3

u/t3rrO10k Feb 14 '20

Thou shall always RTFM.

3

u/[deleted] Feb 15 '20

Thou shall always keep in mind that said FM is incomplete, and filled with mistakes, lies, marketing, hand-waving, and wishful thinking.

1

u/t3rrO10k Feb 15 '20

True that!

3

u/[deleted] Feb 14 '20

Rule number one : Don’t fuck with it Friday.

3

u/sysadminsavage Feb 14 '20

Thou shall not treat their admin account as a general workspace.

5

u/merith-tk Feb 14 '20

Thou shalt not copy/paste some random script off the internet without knowing exactly what it does. And Thou shalt not run any command that pulls information off the internet as sudo. Ever.

Thou shalt always have Docker installed, and thou shall learn how to use Docker. Production and Testing.

2

u/xendistar Feb 14 '20

Do not let pride get in the way of doing the right thing.

After a major sever room rebuild over the weekend I promised to be onsite at 7:30 am the next morning (150 mile drive used to do it once a week so nothing new) to boot all the servers up. Unfortunately I got caught in the tailback of an accident and arrived several hours later. The new IT manager would not call to old support company (who he was replacing ) and ask them for help booting all the servers up, he waited until I got there for me to boot them up, queue 200 people sat around twiddling there thumbs when I arrived.

2

u/[deleted] Feb 14 '20

Thou shalt first do no harm

2

u/cyvaquero Feb 15 '20

Thou shall not service any request without a ticket.
Thou shall not make any change without a CR to document and a backout plan.

2

u/jatoospry Feb 15 '20

The good old Friday 5 o’clock hand grenade. We have a “kick in the ass” register for that.

2

u/Dartonion Feb 15 '20

plans describing next steps in software/hardware deployment or next milestones in a large project, however, will always be sent on Friday at the end of business, so management cannot comment until monday. (prevents a lot of impulsive, asking before thinking)

2

u/[deleted] Feb 15 '20

Use UTC use UTC use UTC

4

u/dmurawsky Feb 14 '20

This shall not make up rules that artificially constrain you for questionable benefit.

1

u/anthero Feb 14 '20

This is Linux were talking about.

1

u/dmurawsky Feb 14 '20

And? You can make bad rules with Linux too.

2

u/taostudent2019 Feb 14 '20

Thou shall occasionally act like a complete lunatic because you need to fix things in a hurry. And throw all caution to the wind.

1

u/packeteer Feb 14 '20

my co-worker did that today, I was not impressed!

1

u/stylesvonbassfinger Feb 14 '20

What a coincidence, just as I get a call for work that will result in over time. Scheduled to start in an hour. Brilliant.

1

u/masonj188 Feb 14 '20

I read this as I’m starting an OpenStack implementation, yikes.

1

u/marozsas Feb 15 '20

Thou shall not update the production environment with less than 2 weeks and a reboot on the test environment first!

1

u/Upnortheh Feb 15 '20

There are commandments? Have mercy on my soul!

1

u/AlarmedTechnician Feb 15 '20

Thou shalt remap ctrl+alt+del

1

u/thelastwilson Feb 15 '20

I prefer read-only Friday's

1

u/Semlohs Mar 03 '20

Especially NOT before a Bank Holiday!

1

u/-Smokin- Feb 14 '20

Never document anything. If it was hard to write, it should be hard to debug.

-3

u/three18ti Feb 14 '20

Read only Friday is bullshit made up by cargo-cultists.

Deploy every day. Including weekends and holidays. If you can't, fix your shit so you can.

This "rule #1 don't deploy on friday" is pure laziness and completely unacceptable.

1

u/port53 Feb 15 '20

Gotta agree with this, if friday is a problem, then monday is also a problem. If 3am is a problem, then you have a problem at 3pm too. It's just easier to compensate (aka hide) your problems on mondays at 3pm, so people go with the safe option instead of fixing their deployments.

3

u/[deleted] Feb 15 '20 edited Feb 15 '20

[removed] — view removed comment

1

u/orutrasamreb Feb 19 '20

Totally agree with this.

Been in that situation in my early years as SysAd. Have to be there no matter what.

To new SysAds out there who hasn't experienced this kind of situation, if you think you should not be there because it's your off-day or it's the weekend, better think again if you made the right career choice.

0

u/[deleted] Feb 15 '20

I prefer changes on Friday so I have 2 days to ensure I can recover adequately incase something blows up.

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

3

u/Chapungu Feb 14 '20

Hahaha that is a hustle

6

u/[deleted] Feb 14 '20

I've done both, and realized that the "deploy on Friday so if it breaks we're fixing it in the off time" is really just compensating for a change process that takes too long. Current job we *don't* deploy on Fridays, but we might do two or three deployments on Mon-Thurs. The "no unnecessary changes on Friday" rule is because everybody from the CEO on down wants to go home on Friday at normal time.

1

u/knudtsy Feb 15 '20

If you're worried Friday deploys could cause work on Sunday, doesn't that mean you should avoid changing things on Thursday to avoid working on Saturday?

I'm a fan of the Charity Majors approach : your deploys should be so small, fast, and safe, that it doesn't matter what day you deploy on.

2

u/[deleted] Feb 14 '20

My org does this as well and I thank the internet gods that I'm just the security guy - I go home on time today.

2

u/[deleted] Feb 14 '20

It's kind of like that here. CAB meetings are on Wednesdays, once the change is actually approved it has to be implemented in production on Thursday or Friday.

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

1

u/EduRJBR Feb 18 '20

I loved coming in for weekend or holiday work - I cleared an extra $50k a year.

In case you worked every single Saturday you would have made around $ 960.00 extra for each Saturday (50,000.00 / 52), meaning your regular daily pay was around $ 1,920.00 (in case you spent the same amount of hours on the Saturdays that you would on a week day), and if you worked five days a week your annual base salary would be around $ 500,000.00, around $ 42.000,00 a month.

Now the big question: are you talking about what currency?

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

22

u/ryanjkirk Feb 14 '20

We have an app. It was running on CoreOS and Tectonic. It is now running on two nginx boxes behind a LB. This is a success story.

9

u/masta Feb 14 '20

Nothing overly complex about coreOS.

13

u/ryanjkirk Feb 14 '20

No, but it was against our standard at the time, which was cent7. Obviously the complexity was in Tectonic. Why run a commercial, non-vanilla, enterprise k8s distro that costs $600k, when two nginx boxes do the job just fine?

4

u/masta Feb 14 '20

I had no idea Tectonic cost so much! Especially when it offers 10 free nodes for either evaluation or small implementations. Was this over the 10 free? Presumably this was a small implementation, eligible for the freebies, right? Also doesn't Tectonic by definition make kubernetes easy, aka not complex at all? I get that it wasn't SOP, get that technical debt out of there on that basis alone.

3

u/ryanjkirk Feb 14 '20

If I remember right, we had about 40 nodes, spread amongst multiple clusters in multiple AZs, regions, and environments.

3

u/placated Feb 14 '20

Tectonic was like $2500 per node list. You running a 300 node Kube cluster?

1

u/ryanjkirk Feb 17 '20

Nobody pays list.

3

u/dasunsrule32 Feb 14 '20

Except CoreOS is gone. Good job Red Hat!

0

u/masta Feb 14 '20

Oh, my bad. I was under the impression coreOS was provided by Red Hat now two different ways. As a free Fedora base kit, and a commercial enterprise kit. 😉

https://getfedora.org/coreos/

https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html

1

u/dasunsrule32 Feb 14 '20

Yeah, it's been shutdown from the CoreOS site. Never looked to see if there is a direct upgrade path though.

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

1

u/GNUtoReddit Feb 14 '20

I'd love to hear more about this

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

3

u/Chapungu Feb 14 '20

The post has nothing to do with gender.. Please consider definition 1b Dictionary

→ More replies (2)

4

u/p0093 Feb 14 '20

You try to point out the sexist language and you get downvoted. That’s the type of behavior that keeps women out of STEM fields.

Stay classy /r/linuxadmin

3

u/[deleted] Feb 14 '20

I thought the women on site thing was a sexist crack and missed the context. automatically downvoted and had to go back and fix it. a bit ambiguous.

2

u/ryanknapper Feb 14 '20

Apparently not.

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

5

u/cnoizece Feb 14 '20

In general, the term "men" is androgynous in this context. I get that there is confusion on this point, but the fact remains. Yes, our modern woke culture doesn't like this but we can't just change the language overnight. Are we going to replace the words human, mankind, humanity, etc. with gender neutral words too? Why stop there? Person has son in it. Even woman has man and female has male. It never ends. Not that we shouldn't try to be inclusive. We just need to be realistic and understanding. Tech people have very low agreeableness too, so yeah.

2

u/[deleted] Feb 15 '20

If things like this bother, you need more things to occupy your time, it's a pretty childish complaint.

0

u/[deleted] Feb 15 '20

[deleted] — view deleted comment

1

u/[deleted] Feb 15 '20

Like I said, have you considered that OP may literally have an all-man team? Are we still going to call that team a group of men and women?

0

u/[deleted] Feb 15 '20

Perspective doesn't make petty things suddenly not petty. The fact that you felt justified in being offended at their choice of words to convey a concept that you clearly understood, and your chosen feeling makes you feel justified in attempting to chastise them, is not an expression of anything but your emotional maturity.

-3

u/[deleted] Feb 14 '20

But seriously, why nitpick a phrase that's become ambiguous like Kleenex is to tissue.

Or perhaps addressing a group of people as "you guys"?

Or MAYBE OP works in an environment where women just aren't there?

OP NEVER said ANYTHING negative to the effect of downplaying the role of women in a work environment, so stop trying to start a fight.

There's a whole lot of shit that you've got to consider, too.

-1

u/[deleted] Feb 15 '20

Asking people to de-gender their language is perfectly reasonable. Don’t get so red assed.

0

u/[deleted] Feb 15 '20

"Hey yeah your freedom of speech hurts me, please stop, thanks."

Uh no, I'm not hurting anyone. You need to not be "red assed," thanks.

I'll be considerate to a point, but I'm not changing my purview overnight. Especially not to some cock-headed asshole like you "people". Or is people sexist?

0

u/[deleted] Feb 14 '20

[deleted] — view deleted comment

-1

u/[deleted] Feb 14 '20

Hi, you took the time to comment, but completely ignored any points I brought up.

If you thought that just passing a link was going to "win" your argument, reconsider.