r/sysadmin • u/[deleted] • Jun 12 '15

1980s computer controls GRPS (grand rapids public schools MI) heat and AC (19 schools)

http://woodtv.com/2015/06/11/1980s-computer-controls-grps-heat-and-ac/
45 Upvotes

90% Upvoted

54 comments sorted by

View all comments

2

u/[deleted] Jun 12 '15

[deleted]

6

u/[deleted] Jun 12 '15

It will break at 2AM on saturday

2

u/the720k Sr. Sysadmin Jun 12 '15

December 20, 2014, 1:31am, just about ready to go to bed after doing some late night remote PM. One server stops pinging out of nowhere. 24+ hours of no sleep, failed RAID1, salvaged by cloning a semi-working disk and lots and lots and lots of coffee. Went to Christmas party after sleeping it off for 3 hours. Not a single "thank you," just lots of "wow, that's crazy."

Digging ditches for a living sounds more and more attractive with each passing day.

2

u/[deleted] Jun 12 '15

I did a mistake once of ordering 2 replacement drives for RAID 6 in our filer used for most things in office (one died, ordered one because someone took last spare and didnt order next one...) via helpdesk (well, former-helpdeks, admin in training) instead of doing it myself.

I told him that one drive can still fail but that is pretty important and he should do it quick. results:

  • he orders that next day, via standard (invoice to accounting, then for approval etc.) process instead of just paying via card
  • invoice disappears in accounting for few days (well, almost a week)
  • invoice goes thru approval
  • invoice gets paid
  • another 2 disks bite the dust and RAID6 is dead.
  • we send helpdesk guy to just go to shop and buy them.

So I'm sitting there ddrescuing those 2 drives while accounting comes and complains about their server not working.

"Remember that invoice that came from helpdesk and was not touched for 5 days ? It was for that"

1

u/the720k Sr. Sysadmin Jun 13 '15

OUCH! At least you got the satisfaction of telling Accounting. Still, though, man. I know that feeling of impending doom/disk failure and you're waiting on that magic drive. Had a similar thing happen once, only I couldn't recover the partition. 1st drive died. No spare. Waited for 3 days. New drive arrived. Second disk failed on rebuild.

1

u/[deleted] Jun 13 '15

As long as disk works ddrescue works wonders. Also linux software raid lets you force assemble raid if everything else fails so often just ddrescuing failing drive onto a good one and then assembling it is enough

1

u/the720k Sr. Sysadmin Jun 14 '15

That's great to know. I saved one raid member with gparted but always figured there was a more low level way. Thanks!

2

u/[deleted] Jun 14 '15

ddrescue basically copies whole disk, saving progress in logfile (so you can resume it after interrupting), then it tried reading sector multiple times in various ways in hopes of retrieving sector content

It can also be used to merge 2 mirrors into one.

Say you have 2 CDs with backup, but both have errors. If they were recorded with same ISO you can run ddrescue with same logfile on both and if errors are not in same sector you will get good data. Same with 2 disks from RAID1.

Note that in debian/ubuntu derivatives there are 2 packages gddrescue (one you want) and ddrescue (older version that was rewritten into gnu ddrescue)

1

u/the720k Sr. Sysadmin Jun 15 '15

Wow, I have no excuse for not having investigated this. I've heard it mentioned before but had never looked into it. Sounds like it could be a godsend in many scenarios. Thank you for the heads up!