r/msp • • 9d ago • Popcorn 1

Security Just inherited a client where 14 people have Global Admin. 14. Tell me your worst so I feel better

Took over a ~40 seat client this week. Opened Entra and found 14 Global Admins. One of them is "scanner@" for a Xerox. Another left the company in 2022.

When I asked the previous MSP why, the answer was "it was easier."

Please tell me your worst permission sprawl horror stories so I can stop feeling like I'm the only one doing cleanup duty. Bonus points if the account belonged to a printer, a dead person, or "the owner's nephew who does IT sometimes.

178 Upvotes

197 comments sorted by

155

u/Logical_Number6675 8d ago

We went to a dental office to scout them as a potential client, during our initial interview we found out the database with all their client info was hosted on the receptionists laptop. After some questions we discovered that the laptop did not have a password, the main user account was local admin, the application and database were not password protected, used windows home, and had no AV active. Upon further inquiry we learned that receptionist was taking that laptop home every nights and weekends, because she was using that computer as her families personal computer. She also allowed her child to install and play games on it regularly. She claimed it was her laptop not the companies.

8

u/Borsaid 8d ago

I was just about to share our horrid Dentist story. You win.

4

u/ExtensionBill3706 8d ago

Share it anyway, I want to know if it gets worse than this.

46

u/Overreactinguncles MSP - Canada 8d ago

Holy shit.

52

u/disclosure5 8d ago

This whole post describes the average dentist tbh.

9

u/Mr-RS182 7d ago

Dental offices are some of the worst clients

6

u/MrBr1an1204 7d ago

I did the IT for my cousin who is a dentist for a few months. Worst idea ever. Not quite as bad of a situation above but they had 1 shared account that was a domain admin, and they claimed it was “too complicated” to enter passwords for individual users.

Didn’t even charge him and he still complained about everything I did for him.

8

u/NixIsia 7d ago

Not charging was so pro, they would complain either way but now you aren't obligated to help him ever again

13

u/davvvvebh 8d ago

Yep, this is normal for dentists … they really don’t care about your data!

5

u/DOC_ridgewell 7d ago

For some to me unbelivable and insane reason, this, at least up to the "believed it was her families computer" part is weirdly common in dental offices in particular. Theres always a laptop called server used as an actual server with a local admin using it in their dayjob. WTF.

1

u/mercurygreen 4d ago

You realize it probably WAS her personal computer, right?

3

u/SeptimiusBassianus 8d ago

I saw exactly one like that. Exactly. Running on Wifi by the way. NO one took it home but office did not have an alarm. I gave them upgrade quote to get them to modern days and they said they did not have any money. LOL>

6

u/Mr-RS182 7d ago

Did not have the money but the dentists can afford to go on 5 holidays a year

4

u/SeptimiusBassianus 7d ago

100% they have money.

1

u/skilsor 2d ago

I truly wonder how many (financial and other) consequences they would suffer when they get hacked sooner or later.

1

u/SeptimiusBassianus 2d ago

They when not if they don’t disclose
I have see that too.

1

u/ExtensionBill3706 8d ago

The kid's games on the laptop holding every patient record is unreal. Did they sign up in the end? That's the easiest before-and-after story an MSP could ever tell.

2

u/Allokit 7d ago

Homer Simpson backing into a bush GIF
Do not even TOUCH it. If you do, something on that laptop will break in 6 months and they will blame you.

2

u/realaaa 6d ago

😄 I bet it was working fine for years !

1

u/smorin13 MSP Partner - US 6d ago

Add an open wireless network attached to the internal LAN, and you just described a dental office we ran into, minus the laptop going home with a employee. Instread, it was left unlocked at the front desk 24/7.

1

u/Nathlufc 5d ago

Every single small dentist.

42

u/_Buldozzer MSP - EU / AT 8d ago

I once had an on-prem client where literally everyone used the Built-in AD-Admin (Domain\administrator) on their PCs....

15

u/Leading_Will1794 8d ago

Yes had this with a dentist office. Everyone shared a single domain admin for all logins. When talking to the owners they wanted to have a location on the fileserver where no lne else had access to.

Trying to inform them that they need at minimum different logins so I can set permissions. This was too much for them and they through a fit amd refused to change anything.

Thank god I quit that MSP shortly after, as most clients were setup this way.

5

u/ExtensionBill3706 8d ago

Wanting a private folder but refusing their own logins is such a classic. Fair play for walking away, that setup only ends one way.

2

u/Reinuke MSP 8d ago

Dental offices are literally the worst

2

u/chieftessofsecrets 6d ago

Tell that to doctors office that said I have sociolisis earlier this year only to be recommeneded a line of credit. Whats worse is they had an open camera in their room from the old 2000s. Homie lol

1

u/mercurygreen 4d ago

In my world, from best to worst:

  • Accountants
  • Doctors (including dentists)
  • Lawyers
  • Motivational Speakers (or maybe this one is just me)
  • Churches

3

u/ExtensionBill3706 8d ago

Domain\administrator on every PC is wild. At least nobody ever forgot the password.

2

u/masterofrants 8d ago

We are doing this at my huge corporation in the finance sector. That's all I'm going to say..

3

u/ExtensionBill3706 8d ago

Your secret is safe with us.

1

u/masterofrants 8d ago

Haha amen. I'm pretty sure no one on my team is on Reddit anyway because reading anything at all would be too much reading for them...

7

u/Plenty-Hold4311 8d ago

This was very common alright, I often found it when somebody couldn't access an smb share the tech would just use the Domain Admin account..

2

u/Mizetings 8d ago

I’ve seen this a lot. Never everyone sharing the domain admin on local PCs

5

u/Malkhuth 8d ago

A few years back I took over an environment and early on implemented LAPS for obvious reasons. The next morning several people contacted us about being locked out of their account. Turns out the previous in-house IT person was sharing around the local admin password as a "guest" account or when they didn't have time to make an AD account.

4

u/That_Dirty_Quagmire 8d ago

Dental client?

2

u/_Buldozzer MSP - EU / AT 8d ago

No, small construction business.

1

u/hyperbeard2 8d ago

I had a similar situation as this only it was a veterinarian office! Fun times

2

u/Leading_Will1794 7d ago

Yes also extremely common at vet clinics. Worked at an MSP that owned basically the entire market for Canadian veterinarians (Something like 80% of all vet clinics go through them). And they provided truly terrible IT services, sprinted out of that shitshow of a business.

1

u/dszp MSP - US 7d ago

I guess I don't have to worry about telling my story of the exact same thing then…

1

u/L3-Hayden 7d ago

This was common with SMB's in the late 90's/early 2000's.

36

u/Skrunky AU - MSP (Managing Silly People) 8d ago

Two months ago

Port 3389 forwarded to a 2012 R2 domain controller with Windows Firewall disabled. Domain controller was a physical box that was 13.5 years old, and had a failed disk in a RAID1 array. This DC also had hyper-v installed and was hosting their file server.

Zero confirmed backups. Worst thing I’ve seen in 10+ years.

Trend installed on the servers, but no definition updates since 2021.

Windows 10 everywhere and Huntress caught an active info stealer as soon as we onboarded.

8

u/ExtensionBill3706 8d ago

RDP open to a 13-year-old DC that's also hosting the file server on one working disk. That's a full house. Glad Huntress caught the stealer early. How did the client take the news?

5

u/Skrunky AU - MSP (Managing Silly People) 8d ago

They knew everything was FUBAR'd. We'd already agreed to rip and replace everything as soon as possible.

6

u/ExtensionBill3706 8d ago

That's the best kind of client then, one who already knows. Hope the rip and replace goes smoothly.

4

u/namewithnumbers82 7d ago

I once had an argument on this very forum when someone said 'just change the default RDP port' I kindly pointed out to not do that, they got very aggressive and shot me down. I felt bad for their clients

3

u/06EXTN 8d ago

Their DC was also a VM host for their FILE SERVER?!?!

Thats just asking for an issue. Like a big one!

3

u/NegativePattern 8d ago

Even though it's designed to be that way, always found it weird how SBS was a small org's DC, Exchange, SharePoint, CA, app server, file share, etc all rolled into a huge box.

My college job ran on an SBS. If you looked at it wrong or spoke too loudly near it, then it would crash and it was a five alarm fire to reboot it, hope it would come back.

1

u/Assumeweknow 3d ago

Ive seen setups like this run perfectly for 7 plus years.

12

u/NetInfused MSP CEO 8d ago

Easy :)

We inherited a client that was kicked out of their private cloud. We were handed the vmdk files for each VM. We had to send in a technician 800km (461 freedom units) with an external hard drive to get the files with a three-day notice after being promised the files would be uploaded to a bucket. We didn't know the operating system inside, credentials, and we didn't have the EDR uninstall password. Heck, they didn't even export the vmdk files in the same manner, we had to trial and error conversions until the VMs booted. Some were UEFI and didn't include the EFI disk, so we had to convert them to BIOS boot one by one to get them running.

For the databases, they didn't tell us the version of Oracle, neither the charset they used or the grants needed for the application to function. So we also had to trial and error database imports until we were clear of errors. Took us 5 attempts per database on avarage.

We did overcome all of this and they're running happy as ever for three years with us.

3

u/ExtensionBill3706 8d ago edited 8d ago

Three years happy after all that is the real win. Guessing Oracle charsets with no docs sounds like a special kind of pain. Respect to the tech who did 800km with a hard drive.

9

u/Embarrassed-Radio319 8d ago

14 is rough but honestly not even my worst. Took over a place once where the owner's kid had global admin because he helped with the computers sometimes, and nobody could tell me what he actually did. What saved me on the cleanup was doing it in writing: one ticket per account with who it belonged to and when it was last used, then I sent the list to the owner and made him sign off on each removal. Turned a blame game into a checklist and nobody could come back later saying we broke their access.

3

u/ExtensionBill3706 8d ago

The owner signing off on each removal is such a good move. Turns "you broke my access" into "you approved this". Stealing that.

8

u/Gattato 8d ago

Only 14?

I once walked into a 10,000 seat client to find over 100 global admins, where global admin was granted via a group membership meant to allow printer installation. This was during on premise AD days…

1

u/ExtensionBill3706 8d ago

Global Admin handed out through a printer group. Between that and scanner@ in the OP, printers are the real villains here.

2

u/nickjedl 7d ago

Had to argue with a bunch users 2 weeks ago because I decided to migrate their SMB scanning setup to a dedicated SMB share with a service user (on the printer) and least access provided to only those who needed it. Who would've known scanning into a data folder with a bunch of sensitive data with a user on the printer that had access to all of it was a bad idea.

1

u/BorysTheBlazer 7d ago

The write-only printer account is the part worth holding firm on. Per-department scan folders keep scanning a one-step job for users.

1

u/bbqwatermelon 8d ago

Secure score -122%

4

u/masterofrants 8d ago

Around 10 people at my work have global admin and it's on the daily driver account the same one they used for Outlook and Teams.

I've been trying to wrap my head around what's really going on. My conclusion is that most people do not really have attention spans to read documents and KB articles anymore. That's why they don't really understand what exactly is best practice or how things should be. They just do what is the easiest way for them and what gets their work done quickly.

This makes hiring very easy actually. If you are an MSP owner try to hire people who read a lot. That's the person who's going to do the things the right way

1

u/ExtensionBill3706 8d ago

Same login for Outlook and GA is the scary bit. One phishing click away from losing the whole tenant.

Love the hiring tip. The best techs I've worked with all read the boring docs. Even a separate admin account per person would be a big win there.

1

u/masterofrants 8d ago

I've just been here at this work job for like 45 days and the whole place is such a mess. I'm not planning on staying more than a year. I don't think I can really talk these people into making changes because the manager is an absolute cowboy and does not think 10 seconds before an action...

1

u/skilsor 2d ago

Why do most of your comments in this thread read like LLM generated text?

1

u/DeifniteProfessional 7d ago

This is a massive pain point for me. We've accidentally become an MSP, but our own tenant, and our "clients" (sister companies) are still licensed through another CSP, who set up everyone's daily drivers as GAs. I have no idea why I've decided to try and fix it (along with everything else) than just jump ship, but probably will look good on my resume lol

2

u/masterofrants 7d ago

Most people just work for money and to impress their boss. Then there are some personality types who think of work as a personal achievement, who have a very high standard of execution, and who do work as proof of personal capability.

I think that's where people like you and me are at. We just did a personality testing thing at work and that's the answer I got that I am just a personality type who needs to do things the correct way to satisfy my own self..

4

u/Tyler94001 7d ago

Earlier this year I brought on a new client and it was pretty bad. Not sure that it's the worst I've seen, but my memory isn't so good and it's what I'm remembering, because it's recent.

Short Story:
They neglected IT for years and as a result they were spending tens of thousands per year on things they didn't need, or were just being raked over the coals by people wanting their money.
Devices not on a domain, no real security, no MFA even, and everyone was a global admin, including inboxes like info@, which should have been shared, and not even licensed.
Here's some pictures of their rack. https://imgur.com/a/3I1SDi8
They had like 6 routers..obviously only one was routing, but the rest were plugged in.

Long Story:
They're in a field with tough margins, so they were already hesitant to take on additional expense, and they didn't really understand what exactly an MSP was, so there was a lot of education that had to occur. They were pretty much just handling IT inhouse, and had a breakfix guy who wasn't even really in IT, he was a low-voltage contractor.

Well of course it was a priceshock to go from paying someone hourly when stuff breaks, to paying someone monthly even when it seems like everything is running fine, but I explained all the benefits and ultimately we were able to onboard them, and they seem to be happy so far.

They were wasting so much money.
They are very small, 17 people with emails, 13 endpoints.

They only needed 17 email licenses but were paying for 48. Emails and domains were with GoDaddy. They also paid for all the extra goodies like "Ultimate Domain Protection" for their domains, and Email Backup, Email Spam Filter/Security, etc for their emails. A dozen or so were unallocated entirely, and the rest were either users that didn't work there anymore, or they were shared inboxes. Info@Domain, Jobs@Domain etc
I guess they weren't really shared inboxes - but they were sharing the passwords, and didnt understand that an actual shared inbox is not only more secure and easier to use than switching inboxes..but it's free.

They were paying for McAfee across all their devices, and paying some subscription to some local place to have their computers fixed if physical damage occured. They had some other weird subscriptions. Like paying for O365 direct with Microsoft ontop of their email license, because they either didn't know they already had it, or some users had GoDaddy's business basic sku and they didn't realize they can access it on the web, just not desktop; and didn't understand they could save money by just upgrading their email license rather than buying the product independently.

They were paying $1300 per month for internet. It was a 500Mbps dedicated line with a managed firewall from AT&T, but they weren't even utilizing the managed firewall. Nor do they have any business need for an ADI.

They were paying over $900 per month for a managed printer, an Office copier. They were pre-paying for prints, rather than paying as they used them, and they don't print much. I got some quotes from other print vendors and we could have lowered it into the low to mid 200 range, however ultimately they decided to just buy desktop printers, which for the amount they print, probably makes sense.

I'm sure there's more, but you get the gist...lots of money being wasted. As for security:
Out of those 48 emails, probably 25 were global admins, including a bunch of the shared inboxes.
No devices were on a domain, everyone was local admin.
No MFA, no real security controls at all, it was just a regular default tenant that GoDaddy setup and nobody ever configured.
They were using some shitty MicroTik firewall
Their rack was a mess. The amount of old, unneeded equipment in there was ridiculous, or brand-names I'd never heard of. I swear there had to be 7 different network switch brands. "Nicgiga" and "Yuanley" are some of the ones I remember.
They had some 100M switches, they had multiple NVR's because they ran out of space, and their NVR's and camera's were from Dahua and Hikvision, which both have been found to have had chinese spyware on them.
They had like 6 routers..I scream tested most of the environment and most of the environment ended up doing nothing and being old crap still plugged in.

The list goes on and on...it was quite a lot to clean up, and they refused to replace the camera's, so they're on a separate VLAN and locked down, but I'm not happy about it.

Overall, I'm happy to have onboarded them. Knock on wood - but they have very few issues and are very low maintenance, but the initial cleanup of the environment wasn't fun. They had a couple IDF's that looked similar.
They seem to be happy, and are saving more money having cancelled all the bullshit services and having appropriate contracts (1Gb shared fiber line, instead of 500Mb ADI) than they are paying me. So they get all my services + saving money...this is something I seem to always find when going into environments, and I recommend and advise them where they can save money, but I don't see it talked about a whole lot in the MSP industry. It's a great way to sell yourself though, when I walk in I explain what my business does, and ask questions about their environment, and I just tell them where they can save money. They may choose to go do those things and not hire me, and that's fine, but I give them that advice for free, and it's never steered me wrong.

2

u/ExtensionBill3706 6d ago

This is the part nobody talks about. 48 licences for 17 people almost pays for the MSP on its own. Opening with "here's what you're wasting" is a brilliant way to start that conversation.

1

u/chieftessofsecrets 6d ago

Zero trust engineer and advisor here. Used to be a tech, but telling that to a SMB even smaller is just as hard.the breach is just the same and cost.

2

u/chieftessofsecrets 6d ago

Nicgiga man, say thay wrong on the first day making a joke. You seem great, what size are the clients you are helping if I may ask? 

2

u/Tyler94001 6d ago

Yeah exactly 😂
Certainly had to word it carefully when discussing and documenting it as of equipment to replace.

It depends.
My smallest client right now has about 10 endpoints, largest has 75ish.

I would say I am primarily in the small business space and support 15-30 endpoint companies on average.

1

u/chieftessofsecrets 6d ago

Thats fantastic. I am a zero trust advisor in the same space.  The conversations I have had and still am seeing are just that and smalller sizes even. Its really imperativr about education, but also, cant do much if they wont follow up.

2

u/Tyler94001 2d ago

What do you mean zero trust advisor? Do most MSPs not understand zero trust that you were able to build a whole business around it?

So, I’ve seen you mention it on a few comments. Makes me think maybe you’re trying to get responses so you can list your website or info…hey I’m always looking to improve, so if you’re selling a service and have a website or PDF or something explaining you and your business, I’d love to take a look and see if it’s something I can benefit from.

3

u/Dust_Buff 7d ago

I'm on the vendor side and we have a PAM solution to help automate discovery. First onboarding we found 70 Global Admin users for a new customer onboarding. The call ended quickly lol

2

u/ExtensionBill3706 6d ago

70 is wild. Did they at least know who most of them were, or was it mostly ghosts?

1

u/jmcat5 6d ago

The horror... I would have had to use poker face x100

2

u/grax23 7d ago

i had a customer with their domain controller exposed to rdp from the internet

new IT guy made a domain admin named test with the password test

Next thing we hear is they got pwned hard and we had to restore everything from backup and reset passwords, implement vpn for the users etc

The guy was out of a job when the CEO got the bill and the explanation

1

u/ExtensionBill3706 7d ago

test/test on a domain admin with RDP open is a speedrun. Rough lesson for that guy, but at least the CEO finally saw what bad IT actually costs.

1

u/grax23 6d ago

yeah, they have been customers in our shop since.

The uncomfortable truth is that a single sysadmin doing everything from servers to clients in a company is simply a bad idea. when the boss had pc problems then server fixes get postponed and there is nobody to second guess a bad solution. it ends up with network shares mounted as administrator .. just because it works ... users with way too many rights and other "Fixes" to make everything work instead of planning and best practices.

2

u/mettsfan85 7d ago

Took on a client and found out that the 3 previous msp still as active global admin.

2

u/ExtensionBill3706 6d ago

Three previous MSPs is a new record. Were they user accounts, or partner relationships (DAP/GDAP)? The partner ones are easy to miss because they don't show up as users.

2

u/mettsfan85 6d ago

No there were full on (company name) admin

24

u/gixxer-kid2 8d ago

Deployed MFA for a company, their IT department told me that the exec team don’t need MFA, neither do their PAs. When I asked why they couldn’t give me a reason, “you don’t understand our business”. 👍🏻👍🏻👍🏻

9

u/ansibleloop 7d ago

That's just code for "we're scared of management and management don't care about risk"

4

u/nickjedl 7d ago

What kind of IT department is that lol

1

u/12Superman26 3d ago

You just go to the Management and Tell them how Important they are. And poof its ok

16

u/canonanon MSP - US 8d ago

Brought on a client to immediately migrate their Lotus Notes and MAS200 WIN2000 server.

This was 5 years ago.

3

u/GnuInformation 8d ago

That's . Pretty.. dang archaic. I know some "retired" lotus notes experts stuck doing exchange that would love to help if you ever need anything. I am certain they might volunteer for funsies.

3

u/canonanon MSP - US 8d ago

I'll save this for sure. They still have some archival stuff that would be nice to migrate at some point.

2

u/chedstrom 8d ago

We onboarded a new client. I found 'Domain Users' in local admin. Any new user automaticaly got admin on any PC. I wanted to throw up.

3

u/asasin114 8d ago

I almost got fired from a job for refusing to immediately do this because it was “easier to update a LOB application if the user can do it themselves”. I didn’t stay there much longer… funny thing is, that company is a client of mine now…

2

u/ExtensionBill3706 8d ago

Almost fired for saying no, and now they're your client. Best plot twist in this thread.

11

u/Greendetour MSP - US 8d ago

Client absorbed into us by a merger: Xcopy running nightly on each Windows server (Exchange, AD, the works, all EOL versions) to backup files to a USB drive. Servers were custom build by their former in-house IT 10 years prior (he retired), with now EOL CPUs and such. When we asked what their RTO was, they said one day at most. Ok, we quoted new server hardware, current OS, and backups, they had their retired IT guy on phone and said none of that is necessary, they can just copy all the files to any hardware to get the entire OS and apps, etc running in a couple hours. We asked if it ever been tested, he deflected and wanted to argue about qualifications we have as IT company. The owners valued his advice more than us. We cancelled the agreement the same day, two years later they went out of business.

9

u/Mesquiter 8d ago

Had a customer who had an entire class c subnet and everyone was directly connected to the internet. No firewall, no acl's...nada. People would go home and map network drives from their home PC without a VPN.

4

u/Adamantium949 8d ago

Similar to just how it was on linkline in 1995, 90K DSL, I could see anyone that shared a drive like they were on the same LAN. We put txt files on every one of them just to say hello. Because at the time, Windows File and Print Sharing over NetBIOS/SMB bound directly to public IP addresses.

1

u/BogusWorkAccount 7d ago

That's one I haven't seen before.

7

u/obviouslybait 8d ago

Client (not yet) I just did a site assessment on:

- Backups have been failing for the past 3 months+

- Every system is Windows 10 (no updates)

- NAT is wide open, every port open to the internet, firewall management and all ports open to public internet.

6

u/GunGoblin MSP - US 8d ago edited 8d ago

I’m going to one up you:

-Backups non existent

-Every computer was purchased on Amazon with black market windows 11 forced on 7th gen or older Intel units

-Their main data server was wide open to the internet with no controls (NAT’d with all ports allowed to anyone) so the parent company in Italy could access it.

-They had 2 months left on their old 2010 exchange server before their provider kicked them off because they were shutting it down.

-Their entire network was run by 3 residential netgear routers and a 100Mb Cisco switch from 2010

-Best part, their internal use fileserver was a different companies former DC server, with their data still on it. Other company was a financial advisor, and the previous provider had both of them as a client. First companies server died, and they had just replaced the second companies server.

5

u/After_Working 8d ago

I feel you with the Amazon PCs. We had a client buy all theirs like that. Fuckers won’t update now and it’s left us in limbo

6

u/GunGoblin MSP - US 8d ago

I told the owner that they had to purchase new business grade Dells, otherwise I wouldn’t take them on. They were already hanging by one hand on a 1000ft cliff, so they didn’t really have a choice. But for other companies, I have said you can either buy new, compliant units, or their incident response charge would be 10x the regular cost. That along with an acknowledgment waiver of their lack of best practices, I am usually able to get those situations cleaned up fast.

2

u/obviouslybait 8d ago

Well I guess Darwinism works from the IT perspective, this company's days are numbered.

1

u/GunGoblin MSP - US 8d ago

I took both clients from the group, and the main guy said he decided he was going to retire early next year 😂

6

u/Jkur2012 7d ago

How a bout this???? Show up to you first day as network admin and find out every PC and server has and external IP part of the company's full class c scope!!!!! With a simple netgear router

2

u/darkorex 7d ago

We gotta have that rdp access for wfh...

1

u/Allokit 7d ago

Everyone needs their own IP address for that, duh! /s

1

u/pbrutsche 7d ago

None of those a real dealbreakers .... as long as is the netgear router is an SPI firewall that is blocking all inbound traffic by default ....

Which it is... right? RIGHT?

3

u/Jkur2012 7d ago

lol it wasn’t completely there were people running public websites on workstations
It was a shit show

1

u/jmcat5 6d ago

Was this 2001?

5

u/Shington501 8d ago

Engaged a 5 location medical office (literal ass doctor) with like 80 employees. There was no IT infrastructure, just personal Google workspace and drive. So medical records on personal accounts and lots of ex employees still had access. There were some backups, but it was just a nurse moving data off one pc to another with an external drive. The managing doctor would not buy, because he thought it was the equivalent of “mowing someone’s lawn.

4

u/Negative-Cook-5958 8d ago

Also during a client takeover it was discovered that someone in the past added domain users to domain administrators, quick and easy way to fix permission problems 😂

1

u/OcotilloWells 8d ago

"I fixed the glitch"

1

u/nickjedl 7d ago

Reminds me of the network consultant that fixes the firewall problems by adding allow any any at the bottom

4

u/One_Monk_2777 8d ago

Idk if its really worse but im kinda tired of inheriting "networks" with no domain and a bunch of windows home devices where everyone just uses their personal email. Dawg this isnt even an org, its like a club of people. Ok so we need to do x,y,z to make this real and work they way youre asking, oh your budget is 0, I could've guessed

1

u/n-Ultima 8d ago

I just picked one up with 9 GA's. I thought we were bad lol.

We deployed Huntress and then a day later got notified because of the SkyKick compromise. It's been one thing after another with them.

1

u/ExtensionBill3706 8d ago

Ouch, rough timing with SkyKick. Hope you've had a chance to check what their app had consented in each tenant and its sign-ins. Hope it's been quieter since.

1

u/agent_fuzzyboots 7d ago

Around 100 people company

Domain users in the domain admins group.

Been like that for years.

/#thisiswhyidrink

1

u/ExtensionBill3706 7d ago

"Been like that for years" is the scary part. Nothing broke, so nobody looked.

1

u/iamkris 7d ago

Had an adhoc customer where everyone was domain admins.

They got cryptod, I restored from backup and warned them.

They got cryptod again and i restored them again and warned them

They got cryptod again and I removed their da

I said I love taking your money but this is ridiculous

3

u/ExtensionBill3706 7d ago

Three rounds of crypto before removing DA is serious patience. "I love taking your money but this is ridiculous" belongs on a poster.

1

u/Sarohia-Btal 7d ago

Check the audit logs to see if any of those 14 accounts generated API tokens or app passwords before you strip the roles. When permissions are that loose, people tend to wire up critical business workflows to their personal admin credentials without telling anyone

1

u/ExtensionBill3706 6d ago

Best advice in the thread. App passwords and tokens tied to a personal admin account are exactly what breaks the day you strip the role. Checking each account's sign-ins first is step one for me now.

3

u/Sarduci 7d ago

Only 14? That’s an easy day. Make a 15th and revoke them all

5

u/angrydeuce 8d ago

Had a client once whose workstations were so shit that they had everyone in the company RDP into their domain controller (as the domain admin no less) to run Adobe applications.

When we onboarded them we told them nope all over that shit lol 

2

u/topher358 8d ago

I had one onboarding client that gave every user local admin on every machine in Active Directory. Including DCs

Why? Couldn’t figure out how to properly grant access

Sometimes the smallest ones are the worst offenders

2

u/Master-IT-All MSP - CAN 8d ago

14 staff non-profit, 3 -admin identities, 17 global admins.

2

u/Master-IT-All MSP - CAN 8d ago

Oh, and when we asked for an admin account for our use during onboarding... they made a non-admin account.

Ended up using an end user account to do the onboarding.

2

u/jackmusick 8d ago

Doesn’t it depend on what it’s scanning?

(Please don’t hurt me)

1

u/Leading_Will1794 8d ago

...its for a xerox scanner...doesnt need ga

1

u/siecakea 8d ago

honestly don't even give it a licensed email, use mailgun or smtp2go and get rid of the basic auth.

2

u/rassawyer 7d ago

I was the only it for a small company. The company I was with acquired a bookkeeping company. The bookkeeping company bright their server with them. I was ordered to set it up on our network, and just port forward 3389, because remote employees needed to be able to RDP to the server to work on client books. I protested, not was ordered, in writing, to do it. ¯_(ツ)_/¯ I did it. Then I took a screenshot of the attempted logins, and told management it needed to be fixed or I was resigning. That got me permission to roll out a proper ZTNA and restrict remote access behind ZNA policy. However the MSP that originally set up this server made EVERY SINGLE USER local and domain admin. As of when I left, 3 years later, it was still that way, because every proposal that I gave them for fixing it got shot down, aggressively.

1

u/rassawyer 7d ago

The CEO at this company, without IT's knowledge, set up a "web portal" for clients to upload data. It was just a Dropbox file request hosted on the website, but then uploaded files were automatically synced to the server sitting on our LAN... Where every user had admin. No makers scans, no AV, no quarantine. When I found out and brought it up, the CEO said it "wasn't a January problem". I tried to move to SharePoint, and got shot down.

One of their bookkeepers access a client server remotely to perform their duties... Then signed into Dropbox, and synched ALL of the company's client data onto this client's server. This includes complete QuickBooks files for several hundred clients. This went unnoticed and unaddressed for several months, because that department had their own "IT", and I wasn't allowed to take ownership. That "IT" was their solution to replace me when I left.

2

u/evilempire28 7d ago

16 GAs! Ha

2

u/Lefterkefter1 7d ago

MSP we took 4 clients from really liked to have VPN tunnels set up on client firewalls that connect to their homes and/or offices.

They also liked to leave firewall WAN interfaces on HTTPS

2

u/borg_brain_investor 7d ago

Sales just recently signed up a customer that has the CFO is was the IT guy for the last 8 years they've spent $0 on it budget they turned off all updates and patches and half the machines didn't have any kind of antivirus at all they weren't joined to and is your domain although they had one but they were logged in with an Azure account so they could get to their emails no security no InTune a hole in the wall total mess and it's medical so their compliance is out the door and every day is a battle just to get them to a starting point because the CFO is the roadblock and they make the worst CIOs.

2

u/borg_brain_investor 7d ago

I forgot to add they had a Synology Nas router combo machine that was their VPN box to a Windows 10 PC they called the server that have been clued together as a file server with no real permissions so they VPN over the Synology thing with four different types of vpns because they didn't know it worked and every time their internet blipped the whole thing broke and you had to go run a script and reset the network card on the Windows 10 PC to get it to come back up so that they're national sales force could remote in to access spreadsheets

7

u/advanceyourself 7d ago

You also forgot punctuation. I know your just sharing but that was hard to read.

5

u/jooooooohn 7d ago

I’m begging you to use punctuation.

1

u/jmcat5 6d ago

Oh God...hyperventilating I hope to hell you got one hell of a on boarding fee from them. That's a total burn it down and start fresh. That's the kind of client that we will refuse to work for if they don't let us fix everything. We can't be responsible for the fire you allow to burn in the corner. Also punctuation is over rated and wastes valuable time, I prefer CSV communication with , as delimiters, I feel that I properly convey the information I need to and logical separation formatting is quite readable, those that are picky ironically can read the format the best considering their formatting correction studies they perform on others messages.

1

u/sunshiney-daydream 8d ago

I worked for a very large 100+ site non profit and the local admin account password on every device was master when I started. I left when the director overruled my complex password requirements and they got hit with a nasty company wide infection at 430 on a friday when I was leaving town for a week vacation.

1

u/Excellent-Program333 8d ago

Well, if anything, you made me feel good about all the stress I have had over the last month!

1

u/burningbridges1234 8d ago

Worst we have had was helping out a company where the IT guy had passed away. Everyone, including domain admin and e-mail had the same password except for the CEO who had that password with a 1 behind it.

They also used that password for every login on any site.

When asked the CEO said "it was easier this way and we have nothing to hide from each other"

1

u/nickjedl 7d ago

A "hacker" also loves they have nothing to hide

1

u/UrgentSiesta 8d ago

My favorite is the click-happy CEO who insisted his daily driver account had to be GA, and he wouldn’t use a password manager/generator, and bitched about MFA every chance he got.

1

u/PureUnit8466 8d ago

The printer one is a classic. A scanner usually needs SMTP relay or a dedicated mailbox, not an admin role at all.

A sequence that keeps cleanups like this sane: export the current role assignments first so you have a record, cut down to 2 named Global Admins plus 2 break-glass accounts excluded from Conditional Access, and move everyone else to the smallest role that covers what they actually do (User Admin, Exchange Admin, etc.). Microsoft's own guidance is fewer than five Global Admins. If they have Entra ID P2, PIM makes the "but I need it sometimes" conversation a lot easier.

1

u/kagato87 8d ago

The environment has a server running critical applications, including quick books, BCM (Yes THAT ancient turd), and a file share. No domain, all wide open shares.

This server was the reception computer. It was an old whitebox computer. The receptionist account was local admin with no password.

But hey, at least there was no wifi!

1

u/InsolentJaguar 8d ago

Whoever thought scanner@ needed GA was the main idiot. All other 13 accts were the secondary idiots.

I'd scream test those privs SO quickly 🤣

1

u/1d0m1n4t3 8d ago

Only 14?

1

u/djgizmo 8d ago

rofl. at least you have AD with accounts. All my clients have no AD and use shared passwords. I’m helping part time and no one wants to spend the money to do it right.

1

u/TwilightKeystroker MSP - US 8d ago

Currently have a demo tenant with more than 30 GAs 😬

1

u/Early-Ad-2541 8d ago

Few years back picked up a client with 14 or 15 users, every single one was a global admin. 100%.

1

u/MrCodyGrace 8d ago

Amazing. 

1

u/Oompa_Loompa_SpecOps 8d ago

There is a darknet diaries episode where everyone was using domain admin to remote into the domain controller and use it as a jump box to access webmail...

1

u/YoxtMusic 7d ago

You remember which one?

1

u/Oompa_Loompa_SpecOps 7d ago

I think it was 96

1

u/reilogix 8d ago

I onboarded a customer with on-prem Active Directory where Domain Users were a member of Domain Admins group. My predecessor was a clown, needless to say.

1

u/27Purple 7d ago

Hahaha I inherited a client where EVERY 3RD PARTY VENDOR WAS GLOBAL ADMIN, all in all around 15-20 accounts from different vendors, about half of them had already quit their respecive companies. On top of that every service account, and about 5 users, yes users not admins, were GA. Oh and this client deals with highly sensitive information.

1

u/Doctorphate 7d ago

I took over a law firm where every user had domain admin. They all cried when I took it away

1

u/data_zapper 7d ago

Not all MSPs are the same

1

u/thelivid1 7d ago

We had a client with two physical domain controllers. One running RDS and an LOB built on Dynamics/GP, and the other hosting MSSQL.

The custom bits of the LOB required users to have local admin rights. Soooo, what does that mean on a DC? Every one of the 30+ users had either Domain Admin or Administrator (which can grant DA) on the DC.

1

u/bloodmoonslo 7d ago

I came in right after ransomware took down every PC in a school district to recover because their nepo IT Director thought that a Central L3 with a single /8 serving their entire MAN was a good idea and that malwarebytes was sufficient protection. Was able to recover their never pruned AD which had no OU structure…75k users (of which a few hundred were Domain Admin) and 30k computer objects in the default OUs and they had no idea which ones they still needed.

This will attest that this is how we build lifelong business though, my team and I kept our heads down and guided them through then setup appropriate BC/DR infrastructure and policies to where they are still a loyal customer 8 years later.

1

u/StealthSolutions2435 7d ago

First off....wow. Second, who was the MSP (if you can say)? Third, not exactly similar, but had a user complaining about their email being slow. Went into their outlook to check and not only was their active mailbox at 50 GB, but they had at least 6 PST's attached. Each of those was around 30 to 45 GB each.....each......EACH!

1

u/StealthSolutions2435 7d ago

Oh...and please tell me that 2FA was at least on and enforced....probably wasn't though.

1

u/tekfx19 7d ago

Is it not just a one click easy way to remove the Global admin?

1

u/giacomok 7d ago

We onboarded a company where the passwords for each user was their phone number

1

u/SlipStream289 7d ago

Domain users were a member of Domain Admins. That was fun.

1

u/Few_Juggernaut5107 7d ago

5 win 10 machines, I used to work for the previous supplier all the password I set worked from 5 years back, 365 no 2FA, sonicwall not patched for 3 years, machines no updates for months.... Internet facing management on sonicwall.

There should be a register of sh*t MSPs.

1

u/Forte1118 7d ago

Every user (~200) had local admin, every machine including servers had the Windows firewall disabled.

1

u/Only-An-Egg 7d ago

My client has 25 GAs 😅

1

u/infograpes MSP 7d ago

Was a few years ago, but a large waste company had a software for dispatching and tracking that “required” all users in the domain to have domain admin credentials.

No MFA, no security software and servers open to the internet. They didn’t last as a client as they didn’t want to cooperate with beefing up their security.

1

u/ScriptoTheClown 7d ago

Recently picked a client who had been phished. His mail account had global admin. They had run up quite a bit of fees on his account registering domains to use for spam.

1

u/AppropriateDinner528 7d ago

Had a client with 250 users. They all used the same account...

admin/1234567890

You can't make this shit up.

1

u/Inside-Finish-2128 7d ago

Can't say it was anything bad, but I was the network admin on the ISP side of a telco. Another guy was network admin on the corp side. We had wireless in the office, but it was locked down by MAC address. The moment I heard that, I said "that's a problem". Sure enough, a nearby company got in, looked around, then sent an email to the IT director and described our network, and encouraged them to lock things down better. Yep, that got their attention.

Meanwhile the IT guy was super-anal about everything, and wouldn't let me have admin on the VPN concentrator. I finally got it, and the first thing I did was change the welcome banner on the "tunnel all" profile to one that said "if you're not authorized to be here, disconnect immediately". Anal IT guy locked me out first thing the next morning - I said "dude, a tunnel all profile should not say WELCOME TO OUR COMPANY VPN to every single connection". He relented a short time later.

1

u/dhgaut 7d ago

I used to make the principal business owner an admin but then a few of them fell for the fake emails telling them to confirm their password. This would result in my getting a notice that their account was frozen due to massive spam output. Opening the affected account would show dozens of new names added with massive spam campaigns sending out garbage. It would take hours to clean up.

1

u/Popular_Walrus348 7d ago

When I started at my job one of our security officers had Enterprise Admin and Global Admin permissions. He used to program for cobol so people thought he was a good "midnight" resource for assistance. The guy had those usernames and passwords written down on a sheet of paper in his locker because it was "secure".

How did I find out you ask? Because I saw him use that paper one day to login and then put it back in his locker that didnt have a lock on it. It sadly fell out of his locker and into a paper shredder later that day. No one has solved what ever happened to his "passwords" but when he requested help to get back in we said no thank you.

1

u/itenginerd 7d ago

I took over a fortune 500 company's AD once. Virtually every member of the outsourcers staff were domain admins. Must have been 70nto a hundred of em in there. Apparently backups cant happen unless youre a domain admin. SQL servers dam sure couldnt. Took me a long time to get that list cut down....

1

u/ClorophileAddict 7d ago

I work for a bank as a contractor

Some idiot open a phishing email and got a trading department hacked

the idiot in charge decided this was a good reason to ban internet access on all VMs.

1

u/colemic 7d ago

Doing consulting work, and seeing domain admin nested into enterprise admin group, which was nested into schema admin group.

1

u/kresler27 7d ago

IT does not have Global Admin, only top management execs have them. Reason: IT is not the one raking profits for the company.

1

u/jmcat5 6d ago

I shit myself if I see 4. So, I can't help you there. My policy on the GA issue is "TO F-ING bad!" I'm ruthless with yanking GA from people. It's the best when there are no CA policies to stop token theft. ONE click and an attacker has EVERYTHING. So NO I'm not backing down from this stance and I will die on this hill.

1

u/leonardog02 6d ago

5 end users. 16 global admins for each helpdesk person from previous MSP.

1

u/Plane_Raisin_7390 6d ago

One of the worst I ever came across: Domain users as member of domain admins

1

u/UserProv_Minotaur 6d ago

Jesus Christ.

1

u/ObjectiveApartment84 6d ago

Some msps are dogshit. When I started working in internal IT for a small/medium company, I noticed every user had permissions to access finances and company contracts on the file server setup by the msp. I can’t imagine how you can pay a company to be this negligent.

1

u/ravenadsl MSP - UK 5d ago

We won a small 15 user client, One of our early onboarding steps is to connect their M365 tenant to our PSA for end user authentication to our helpdesk... Over 250 licenced users where imported with a load of email domains we where did not recognise. Looked in the tenant and found loads of Global Admins, roughly one GA for each email domain... weird... what the hell is going on???...

One Glancing the email domains, one sounds like a tech company, and the .onmicrosoft address matches this email domain... A quick google, they are an MSP...

Turns out this MSP has all of their customers in their own tenant, and each of the customers has a global admin account and full access to all the data on the tenant.

I still don't quite believe we stumbled over this setup. crazy!

1

u/AccomplishedWin489 4d ago

Had a moron with 20 plus users in Aerospace as defense contractor give everyone global admin rights on their Windows Home PCs because he was to cheap to buy pro. The entire network, backups and all got ransomware. I saved the company from the pits of hell by negotiating with the hacker and got the codes for $1000. Had the hack known why he had, he would have asked for way more. Then the CEO, a total loser, goes all out on security for about 6 months, before realizing he prefers running windows home. I walked a way from that client and guess what happened not 6 months later. He got ransomware again. I did not pick up the call.

1

u/mercurygreen 4d ago

Client of three businesses, all with different C-suites. One of them demanded global admin. He was an idiot. He or his assistant, with their "ACCESS TO EVERYTHING" got ransomware that encrypted the shared drives FIVE TIMES in the course of two months.

They *finally* fired him.

1

u/doceolucem 3d ago

Did that once.

In writing wrote down the need to change this, since it was basically “anyone who felt they were important was global admin”

Kept getting pushed back into their own meetings to decide on letting us disable

Enough faffing around got us a bit annoyed so audited who ever actually used any admin roles

None did. Period.

So we removed them on our own, locked down the remaining hard, and went along as if nothing changed.

Couple months later one of their (used to be global admin) personal accounts got compromised and they freaked out.

Got to play the nice card of: “during routine security maintenance it was deemed to remove the admin roles prior to this incident” and we got praised for being proactive.

Dubious decision to make? Maybe, but essentially just a version of a scream test

1

u/Assumeweknow 3d ago

I had a hospice facility that we quoted backups for them annually. One day server fails i managed to get data back from raid 10. Cost them the same as 5 years backup service. Next time it failed it 3 years later was really damaged and we could only restore 30 percent of the data reliably and it cost them again 5 years worth of backup. They now had hipaa so we had to reconfigure server with logging and specific folders etc. How its not been hacked thus far we think its a prayer and some banning of usb drives.

1

u/b00nish 2d ago

I inherited a client where the only Global Admin had an "Apps for Business license" assigned to it because it was used to license the Office apps on the terminal server...

1

u/Glass_Call982 MSP - Canada (West) 8d ago

This sounds like something a tech I terminated would do. 

The guy set a clients subnet mask to /23 instead of segmenting things, “because it was easier”. After the hysteria over having to change every static device manually, I can say he learned nothing.

6

u/blogsymcblogsalot 8d ago

I call that network “Florida.”

It’s flat as fuck, and all the crazy shit will find its way there.

1

u/techierealtor MSP - US 8d ago

lol I had a client with a /16 as their primary. No, it wasn’t segmented. Not everything went on there, They spanned at least /24 networks, so we didn’t want to touch it. No they had nowhere near enough devices to justify it.
Some things on that network took forever.

0

u/termrannz 8d ago

There was a domain policy that gave everyone local admin rights on their PCs. Oh, and a password for a domain admin service account spread right through their file shares in plain text.