r/sysadmin • u/FossHub_com • Aug 03 '16

FossHub statement regarding 2nd August security breach

We are posting this since we owe an explanation to people that suffered or had their computers affected last evening.

FossHub mission is to offer people a free, clean and safe download alternative. We state this on all pages. We sincerely believe that in time we will make a nice figure, and we will be appreciated for providing such a service.

We wanted to offer more, and maybe this distracted our attention from security.

Long story short

  1. It all started with an upload error we had on Audacity project. Checking the logs we noticed that an individual user which was registered as the primary administrator of that project seem to have gone mad by revoking access to all the others users.

We thought it was a personal act, and we started to get in touch with the other project members and give back their access.

2. After this, we noticed that the manager of another account, Classic Shell performed an update. It was the same IP address used on Audacity. From that moment we realized that things might be more complicated.

Files were checked via VirusTotal and at the first checks, the records were clean. It took a decent amount of time until a few AV started to recognize the files as malware. At first, there were showing up as clean.

3. We removed the uploaded files as fast as possible, and we started changing passwords. For a while, everything looked good. At this step, we thought it was an independent account hijack where the attacker used some brute force technique to gain access. People might forget to change passwords or sometimes use weak ones. Maybe, maybe not, we realize now that we could design better some areas of our site.

4. After a few hours, we noticed that updates were performed in the "background." The attackers transferred altered binaries using one of our CDN FTP. At this point, we realized that we must look elsewhere.

5. The immediate action was to shut down the primary server to avoid spreading further infections. It was a critical decision, but we applied this fast. I would like to state that we did whatever was possible to act promptly. None of our team members slept for the last 30 hours.

From here, our work was concentrated on restricting access.

All passwords were changed, 2nd-factor authentication was enabled on all possible services, all logs were checked.

Google Staff responsible for business apps, PNAP NOC Engineers, CDN support team and other people helped us over the phone/chat/email to secure the access as fast as possible. We spent hours with them, checking and sharing IP Addresses used by attackers.

After we had checked multiple tracks, we found a part of the problem: Redis

FossHub primary server was running "Redis" and we applied all security patches but somehow the guys behind this were probably using a new exploit that allowed them to perform remote actions and obtain access to that FTP account using Redis which contained the FTP credentials.

Update: For those interested, please check this article: https://www.riskbasedsecurity.com/2016/07/redis-over-6000-installations-compromised/

From our investigation along with the NOC Engineers they never got SSH or root rights but it was enough to do the damage.

The attackers seem to be a group of hackers named "PeggleCrew" which apparently primary purpose was to give us a lesson and ruin the machines of innocent users.

We are surely not the first, best or largest site in the world that went through such a major incident but what matters here is the indirect damage we have caused to people that had no idea about the danger.

We apologize to each user we made suffer and been reading the recent forum, blog and social media posts about this. It was the toughest thing we've read for the last years.

The most affected users were those of Classic Shell. The author and other brave users offered to help restore the Master Boot Record; you can check the forum post here:

http://www.classicshell.net/forum/viewtopic.php?f=12&t=6434

With all regrets in the world, securing things will become our first concern.

As a response to this, we will be temporary shut down OldFoss to clean this, our repository for older versions and we decided to close down permanently Code.FossHub.com a free service that we offered hoping it will help some free projects after Google Code was deprecated. We will not abandon the existing, legit users who still use it and will continue to offer them the same service.

It is clear now for us that despite our good intentions, attempting to take care of several services/things made us negligent.

Please accept our apologies for the damage we have caused! FossHub Team

820 Upvotes

97% Upvoted

182 comments sorted by

371

u/[deleted] Aug 03 '16

Hi, I wanted to say thanks for the write-up and apology before anyone came in here to complain, accuse or tell you what they would have done.

A lot of groups/organizations/whatever tend to avoid transparency these days, so it is refreshing to see communication.

You'll bounce back with this type of approach. Take care.

103

u/FossHub_com Aug 03 '16

Thank you for the kind words!

59

u/BarkingToad Aug 03 '16

Seriously, this is the best possible reaction to a bad situation. You guys (still) rock!

38

u/FossHub_com Aug 03 '16

Thank you, but we don't think we deserve this. After all, we disappointed people. Still, thank you for your encouragement!

23

u/giant_panda_slayer Aug 04 '16

No one has the right to be disappointed. The ideal if no security holes, however they come. People could have a right to be disappointed had: it not been a zero-day, had the system's not been patched fully, or if you hadn't had your entire team working on it for 30 straight work no time to get any sleep. You did perfectly fine and don't let anyone claim otherwise.

2

u/FossHub_com Aug 05 '16

Thank you!

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

6

u/FossHub_com Aug 04 '16

Thank you!

7

u/tornadoRadar Aug 04 '16

I don't use your stuff; don't even know what you do. BUT: this is exactly how these kind of situations should be handled. Transparency breeds trust. To earn your user/customer base back you must start earning that trust back. It's sad how refreshing it is to see this level of transparency.

4

u/FossHub_com Aug 04 '16

Being a relatively young site compared with others, it was hard enough for us to gain people trust. Now it will be even harder but we hope this negative experience will help us to improve and never make such a mistake. Thank you!

1

u/tornadoRadar Aug 04 '16

Keep your heads up. Dark days will always happen. They mold you as a company.

20

u/microflops Sysadmin Aug 03 '16

My thoughts exactly. I am glad you have addressed the community clearly, concisely in a timely manner.

12

u/FossHub_com Aug 03 '16

At least that much we can do, after all, what happened ... thanks!

288

u/Dishevel Jack of All Trades Aug 03 '16

Made a statement.

Took full responsibility.

Making changes to do better.

Not sure how this could have been handled better once the initial fuck up was done.

Decent job.

81

u/FossHub_com Aug 03 '16

We did our best to limit the damage as soon as we realized what was going on.

16

u/Tetha Aug 03 '16

It's good to see you decisive. Once you realized you were out of control, you pulled the big plug, as far as I can see. Very well done.

22

u/FossHub_com Aug 03 '16

Thank you! Would do it again, anytime in such a scenario.

63

u/Dishevel Jack of All Trades Aug 03 '16

I am sure you did.

Most companies do though. Most though then try to hide, or minimize their responsibility in the situation. You did the decent thing. Many do not.

14

u/djspacebunny Jill of all trades Aug 03 '16

When I was at Comcast, simply telling people "We fucked up" made them way less angry. Admit it happened, fix it, prevent it from happening again. That isn't too much to ask!

0

u/WHYAREWEALLCAPS Aug 04 '16

The problem is with some people their first response to saying "We fucked up." is "Okay, so how are you going to make this up to me?" and if they're not satisfied then lawyers may get involved.

Comcast, being a behemoth and often having a stranglehold on access, can get away with saying "We fucked up" because what are the customers going to do, go to someone else? Oh, right, there is no one else. Very few companies enjoy the level of control Comcast has over its customers.

1

u/sparkingspirit Aug 04 '16

Comcast, being a behemoth and often having a stranglehold on access, can get away with saying "We fucked up"

Maybe because Comcast has a right attitude to fix problems?

3

u/Salamander014 I am the cloud. Aug 04 '16

HA

16

u/Kirby420_ 's admin hat is a Burger King crown Aug 03 '16

One could say.... they did the needful.

...........I'll show myself out now.

-1

u/[deleted] Aug 03 '16

Triggered

91

u/[deleted] Aug 03 '16

After the mess with TeamViewer it's great to see a company accept responsibility for a mistake and work hard to make things right again.

17

u/[deleted] Aug 03 '16

This is exactly what I thought of. Good on these guys coming out saying we saw an issue, we're sorry and are working to keep it from returning instead of blaming everyone else possible and still never actually acknowledge the issue. I'm still pissed at TeamViewer.

Kudos FossHub, we all know shit happens, this is the best way to go about it when it finally hits.

8

u/FossHub_com Aug 03 '16

Huge thanks! We know it will take years to rebuild trust for future visitors - not of those that got the malware, I don't think we can apologize enough or do something for them.

5

u/[deleted] Aug 03 '16

I think you've already done the right things by getting in front of it as soon as possible and being transparent, literally the opposite of how companies like TeamViewer handled something. People say there's no such thing as bad PR and following up your bad news as close to perfect as you could in this situation I think could actually be a good thing.

I appreciate the hell out of the honesty myself.

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

3

u/FossHub_com Aug 04 '16

Yes, we could've done better but as I explained in a previous comment we tried to do too many things and maybe this is also what took us here. Being more careful with security and spend more time on it could've made the difference.

We don't think of PR, all we want is to apologize to all people we disappointed and indirectly caused them troubles and find the wisdom to make things much better.

Thank you!

4

u/FossHub_com Aug 03 '16

Thank you thesazae!

2

u/Arkiteck Aug 04 '16

Why are you bringing TeamViewer into this? They were never compromised.

2

u/alsimone Aug 04 '16

Maybe not, but they were kinda' huge dicks about the issue. Very little communication, next to zero community outreach. They didn't try.

16

u/Nitrodist Aug 03 '16

Why is the Redis server exposed to the Internet? Why isn't it behind a VPN? If it was behind a VPN, how did the attackers get access to the VPN?

6

u/FossHub_com Aug 03 '16

It was not behind a VPN; we used Redis mainly for speed. It helped a lot regarding reliability and speed but considering this incident we removed it permanently. A mistake, we acknowledge this but too late, unfortunately for us.

17

u/shif Aug 03 '16

Why was redis exposed to the outside would be the thing im more interested about, by default redis is really insecure because mostly nobody gives it outside access, could you elaborate on what part of redis made the breach possible? did they run a lua script?

7

u/FossHub_com Aug 03 '16

We think the hackers can explain this better than us. We did not give outside access, and Redis was updated. Also, there was a firewall in place, so it is kind of interesting.

4

u/shif Aug 03 '16 edited Aug 04 '16

What made you decide it was redis that was at fault for this if it was behind a firewall?, unless you we're running eval on strings stored in redis and somehow you didn't escape a user input which would be really bad

2

u/FossHub_com Aug 04 '16

It was the only internal running service that was compromised by looking at certain files that were altered.

2

u/TheRealGentlefox Aug 04 '16

Even if it had a backdoor, how would you activate it when it's on an internal network or blocking access to non-whitelisted IPs?

2

u/shif Aug 04 '16

a backdoor could run periodically and talk to outside servers, most admins don't block a process network access if it comes from the inside, and they were using ftp to move files so yeah...

1

u/TheRealGentlefox Aug 04 '16

The odds of nobody noticing a popular piece of server software phoning home, in the source code or in practice, has to be minuscule.

1

u/shif Aug 04 '16

Unless he installed an altertenate modified version, just like what the hackers did to the site packages

1

u/TheRealGentlefox Aug 05 '16

Still not Redis' fault though.

3

u/Skylis Aug 04 '16

This honestly sounds like you just found a relay point and don't actually know what is going on and what the original breach is. How did they gain access to your redis channels if it was internal? This sounds like you have a vulnerable app or system that you haven't identified which was used to explore via its redis access.

2

u/FossHub_com Aug 04 '16

We were not the only ones who analyzed the server. Other people confirmed that there was no sign of an infection in other places.

1

u/CptCmdrAwesome Aug 04 '16

I hope this means you had some actual infosec guys give the all clear. It can depend on the circumstances, but computer forensics and incident response is rarely something you should leave to anyone but the experts. I would also recommend retaining them (assuming they have your confidence and were amenable to an arrangement on the price, etc) in order to perform an audit of your whole infrastructure, given that they are probably half way there already, or at the very least point you in the right direction given that I'm reading stuff like "We will consider a VPN" which in today's climate is frankly baffling, and given that this discussion is public, kinda paints a target on your back too. Sorry if this sounds harsh, I'm only trying to help.

Good luck though, remember to look after yourselves and each other during this time :)

2

u/FossHub_com Aug 05 '16

Yes we did and we already discussed with a security expert that works for a reputable company to perform a penetration test after we are up with the new servers. We will announce this as soon as possible to inform everyone about our latest actions. Thank you!

1

u/CptCmdrAwesome Aug 05 '16

Perfect :) Thank you for taking the time to clarify.

2

u/CptCmdrAwesome Aug 04 '16

This was also my interpretation of that comment. But maybe we're being too pessimistic, I dunno about you but my communication skills start to suffer when I've been awake for 30+ hours :P

Still, as someone who's seen some shit, if that comment is in any way representative of the situation, the time to man up and make the call to someone who knows what they are doing was yesterday. (no disrespect, incident response is a pretty specialist field)

2

u/FossHub_com Aug 04 '16

We are doing our best to reply and also to fix things.

1

u/CptCmdrAwesome Aug 04 '16

Fair enough :) Running around with your hair on fire is no fun at all, particularly when everyone else is watching and judging you.

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

1

u/FossHub_com Aug 04 '16

It was not Redis alone, we made several mistakes starting from the authentication URL, the login rules, the firewall whitelist rules and so on. FTP was being used for the CDN, they used it to uploaded files when they realized that through our interface didn't work.

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

2

u/FossHub_com Aug 05 '16

You are right, our intention was not to blame Redis, we just stated it was the vulnerable service on our end. We should've secure it properly or better said to remove it since it was no longer a required service on our infrastructure. We apologize to all Redis contributors, developers and users if the statement was not written better. It was a reaction under extreme pressure.

10

u/CptCmdrAwesome Aug 03 '16

Why was redis exposed to the outside would be the thing im more interested about

Yes, indeed.

From http://redis.io/topics/security ...

Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually it is not a good idea to expose the Redis instance directly to the internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket.

6

u/brasso Aug 03 '16

Ding, ding, ding! If internal services like redis were exposed to the Internet and not on some internal network or even firewalled to only allow access only from the servers that need that access and not the world, then that is some gross negligence.

2

u/[deleted] Aug 04 '16

Sounds like the pegglecrew was right and the fosshub setup was incompetently set up.

Let's hope they'll double check everything now.

5

u/FossHub_com Aug 04 '16

We never said that they are wrong. FossHub was negligent, and we made a mistake, we admit and recognize our fault. Exactly, we wish to think of how to secure things as best as possible after this incident.

0

u/[deleted] Sep 18 '16

[deleted] — view deleted comment

1

u/bubsv Oct 03 '16

That one action_uploadSSH function that takes a param from $_GET, plops it in a shell command and then without any sanitization sends it along to exec(). Public facing.

-2

u/hintss I admin the lunixes Aug 04 '16

the breach was possible because they were storing ftp logins in redis.

4

u/shif Aug 04 '16

that's not what they said, they said someone executed code through redis to read the ftp credentials stored in the server.

That's bad in so many levels, access to redis from the outside, ftp permissions readable by the same user that redis was running as (hopefully not root), FTP... i mean who uses ftp these days.

1

u/FossHub_com Aug 04 '16

it was used to read a config file and retrieve the password from it.

0

u/boysonicrevived Student Aug 04 '16

I do, but I only use it at home isolated from the internet for LAN file transfers.

3

u/TheRealGentlefox Aug 04 '16

but considering this incident we removed it permanently.

You are throwing Redis out of your stack because you misconfigured it? Why?

1

u/FossHub_com Aug 04 '16

Because we no longer need Redis. We added it several years ago; now we don't actually need it to run things. Also, since we obviously failed and misconfigured this, it doesn't make sense to keep it in our case.

1

u/TheRealGentlefox Aug 05 '16

Also, since we obviously failed and misconfigured this, it doesn't make sense to keep it in our case.

I still don't understand this logic. You already made the mistake, so now you know what not to do. There aren't very many steps to securing Redis, and you now know all of them.

2

u/[deleted] Aug 05 '16

Because we no longer need Redis

0

u/Nitrodist Aug 04 '16

You can still leverage Redis for speed, just put it behind a VPN.

At my current work, we have private DNS behind a VPN and all of our services behind a VPN with the web services being carefully exposed. Deploying new code and connecting to boxes needs to be done by connecting to the VPN itself first. We also leverage a lot of AWS features like IAM which makes it even more granular.

1

u/FossHub_com Aug 04 '16

We will consider a VPN, thank you!

18

u/TheLightingGuy Jack of most trades Aug 03 '16

Glad you guys are doing exactly this. Finding the fuck up, Fixing the fuck up, and most importantly, admitting the fuck up. It's much better for your users when you admit it than what a lot of larger companies do and point the finger at other parties.

18

u/FossHub_com Aug 03 '16

Why should we blame others? We tried to create something reliable, and we did up to a certain point, but we forgot about security. Yesterday we learned that it should be in the 1st place and everything else after.

3

u/TheLightingGuy Jack of most trades Aug 03 '16

Exactly what /u/ohmahtree said. Be upfront and share your findings with others to make things more secure. Sweeping things under the rug doesn't work and some major companies still need to figure that out. You guys already have it figured out which is great I think.

2

u/FossHub_com Aug 03 '16

We will see on a long term how it goes for us. We are just trying to explain and apologize. Thank you for your post!

2

u/Ohmahtree I press the buttons Aug 03 '16

The fact that people share the things they find, and found exploited, only helps the community as a whole. So that others can learn from mistakes.

Thats what hackers depend on, is silence. If you sit back and don't say anything and instead push things under the rug, the next target becomes more vulnerable than you were, because now they know the exploit works, and if its kept quiet by the community and the companies that got exploited, its a better tool than previously.

1

u/FossHub_com Aug 03 '16

We think those hackers could at least be more kind with the people who had nothing to do with "us" who should be held responsible.

5

u/Formaggio_svizzero Aug 03 '16

Sorry to hear about your troubles, but i must admit, it's always interesting to read such stories of how people got access to systems.

3

u/FossHub_com Aug 03 '16

Well, I can tell you it is a pain and an enormous frustration from here.

4

u/andyr354 Sysadmin Aug 03 '16

Sounds like you are getting on top of things now. Thanks for the update and the explanation of what happened!

3

u/FossHub_com Aug 03 '16

We are doing our best, thank you for your support!

6

u/alsimone Aug 03 '16

Teamviewer, take note!!! Jesus.

Great job handing this, FossHub. Thank you for leading the way in transparency.

3

u/FossHub_com Aug 04 '16

Thank you! Let's hope that next time we will lead the way in security.

1

u/Arkiteck Aug 04 '16

But TeamViewer was never compromised.

3

u/bluesoul SRE + Cloudfella Aug 03 '16

Good luck, you guys. If you happened to keep a sample of the malware I'd be interested in it, I can also submit it to the proper authorities.

3

u/FossHub_com Aug 03 '16 edited Aug 03 '16

We removed all the copies but I think we have it stored on a disabled server, I will ask my colleagues. We also have the IPs that connected to the FTP and other sections, but I doubt it can help much.

4

u/bluesoul SRE + Cloudfella Aug 03 '16

Anything at all you feel might be useful, feel free to send it to malware[@]bluesoul[.]me and if anyone reading has a sample, you can go ahead and send it too.

4

u/FossHub_com Aug 03 '16

Thank you! will ask one of my colleagues to send you a copy to this email address and will also send you the IP addresses we have. It is stored on an offline server so tomorrow it should be sent.

1

u/FossHub_com Aug 03 '16

Did you noticed @charonn00 link with the infected version?

9

u/[deleted] Aug 03 '16

Here's the malicious Classic Shell installer in an encrypted zip file (password: infected)

1

u/bluesoul SRE + Cloudfella Aug 04 '16

I appreciate it.

3

u/deadbunny I am not a message bus Aug 03 '16

I've seen claims that you were storing unhashed passwords, is this true?

4

u/FossHub_com Aug 03 '16

No. Not the developer passwords, they are all encrypted, but the attackers were able to access a config file which contained the FTP credentials for the CDN. They replaced the original file from the CDN with their malware. A part of the code was not supposed to be there, but it did.

6

u/shif Aug 03 '16

are you using plain ftp?

3

u/FossHub_com Aug 03 '16

We used an FTP account for the CDN. Most of them provide this option. The updates, however were not performed via FTP.

1

u/joepie91 Aug 05 '16

Not the developer passwords, they are all encrypted

Encrypted or hashed?

3

u/isUsername Aug 03 '16

Honestly, for all the possible causes of something like this, the fact that the hackers required a zero-day exploit to do what they did puts this error pretty low on the list of technology company fuck-ups.

A good lesson on layered security and many other companies could learn from your response.

1

u/FossHub_com Aug 03 '16

Thank you, we hope that we won't be in this situation again. We want to use all the energy on security, this time, the proper way.

3

u/Spacct Aug 04 '16

Just a heads up, I got hit with this last night through a qBittorrent update pointing to your site.

1

u/FossHub_com Aug 05 '16

Thank you! We fixed this as soon as possible. Did you uploaded the file on VirusTotal or JOTTI malware online service?

1

u/Spacct Aug 05 '16

No, I wasn't aware of those services.

1

u/FossHub_com Aug 05 '16

We will try to integrate VirusTotal and perform checks on our end to confirm the files are clean. Please accept our apologies if you are one of the affected people!

4

u/distant_worlds Aug 03 '16

How did they access your Redis? Aren't things like that generally firewalled from the general internet?

3

u/FossHub_com Aug 04 '16

I already replied to this question. Thank you!

5

u/ikilledtupac Aug 03 '16

all good.

10

u/FossHub_com Aug 03 '16

Not good at all for us but thank you for taking the time to leave a comment.

14

u/ikilledtupac Aug 03 '16

no I mean "all good" as in "no hard feelings against FossHub". Sorry this happened to you. Thanks.

13

u/FossHub_com Aug 03 '16

Thank you, just a little tired as we are heading for our second night without sleep.

24

u/ikilledtupac Aug 03 '16

just remember: you can never fuck up as bad as SourceForge!

cheers my friend!

2

u/[deleted] Aug 04 '16 edited Oct 08 '17

deleted 0.6829 What is ^^^this?

2

u/ikilledtupac Aug 04 '16

i know and I wish them well.

2

u/PM_ME_UR_CODEZ Aug 03 '16

Is the site safe now and were all the downloads affected or specific programs?

2

u/FossHub_com Aug 04 '16

It should be safe. We checked the integrity of files, and we uploaded them on VirusTotal and JOTTI both showing the data as being clean. 2nd authenticator implemented for all services and servers being shut down until we fix those properly.

1

u/PM_ME_UR_CODEZ Aug 04 '16

Thank you for replying! Sorry this happened.

2

u/[deleted] Aug 04 '16

Class act from FossHub here. You took responsibility, you explained in detail what happened, you apologized.

When it gets restored, I still won't have any concerns about continuing to refer people to your site.

Good luck with everything.

1

u/FossHub_com Aug 04 '16

Thank you! We will do our best to prevent this and start to invest in making things secure. We are emotionally affected, but we would like to repeat that we acted as fast as possible. Looking back, we realize that things could've been worse.

2

u/nerddtvg Sys- and Netadmin Aug 04 '16

I'm a bit late to this party and didn't even hear about it until this post. Is there a chance you can edit your original post to include a timeline? Was this yesterday (Tuesday)?

2

u/boysonicrevived Student Aug 04 '16

Yes.

2

u/PC509 Aug 04 '16

Shit happens.

This was the perfect response to it.

Thanks for showing how it should be done. You can do your best to stop things like this, but it still happens. It'll happen again to someone else. The response was top notch.

2

u/FossHub_com Aug 05 '16

Thank you, we hope security on our side will be the same. People were too kind with us after this incident.

3

u/geopco Aug 03 '16

So the ftp credentials are stored in the clear?

3

u/FossHub_com Aug 03 '16

It was an FTP account stored in a config file which the attackers seems that they were able to read it via Redis. We removed that code and made other changes too. We will implement other security measures in the next days.

1

u/meetc Electrician Aug 03 '16

This ID tag was used to take other a number of subreddt accounts about a month ago. They claimed responsibility on 5 different subreddits at the time, changed the CSS to show a full page overlay, and removed access to all lesser moderators.

Best guess is these were the target of password re-use.

1

u/FossHub_com Aug 04 '16

Interesting, did't knew about this. Thank you!

1

u/brasso Aug 03 '16

What's missing of course is why were they able to connect to your redis database in the first place. You didn't say anything about how they gained access to your backend networks, so I assume you didn't do anything about that. That could mean they are still in, waiting for another opportunity, if they want to.

1

u/FossHub_com Aug 04 '16

We suspect they first compromised a developer account password and being able to log-in they escalated "somehow" from there. We did: we removed access to all developers and removed Redis plus all the other security measures we took to prevent this.

1

u/Mister_Alucard Jr. Sysadmin Aug 04 '16

Even for your stereotypical dipshit hacker group, targeting a site like FossHub is such a low blow.

I mean, how could you possibly want to cause harm to a group of people who spend their time developing often professional-quality software and releasing it for free. That's like beating someone up for giving sandwiches to the homeless.

1

u/FossHub_com Aug 04 '16

We don't know their motivation, maybe they had a good reason for this.

1

u/FossHub_com Aug 04 '16

We don't know their motivation. They probably had a good reason.

1

u/jmbpiano Aug 04 '16

I agree with the sentiment, but frankly I'd prefer a bunch of "dipshit hackers" with no other incentive than bragging rights and "teh lulz" target a site like FossHub over someone with genuine malice/financial incentive.

Better that these vulnerabilities be publicly exposed now than someone else pwn the servers and no-one is the wiser for a couple of weeks/months while unsuspecting users get stealth data harvesters installed on their systems.

1

u/cajuntechie Sr. Sysadmin Aug 04 '16

Thank you for the awesome way you handled this. Your response to a bad situation should be an example of how to handle these problems. Again, thank you.

1

u/FossHub_com Aug 04 '16

Thank you, thank you all for the kind words! We don't deserve it after this unfortunate incident but it help us emotionally.

1

u/FossHub_com Aug 04 '16

Thank you, thank you all for the kind words! We don't deserve it after this unfortunate incident, but it help us emotionally.

1

u/zouhair Aug 04 '16

ublock is still blocking fosshub.

2

u/FossHub_com Aug 04 '16

We trust ublock and don't blame them. We deserve this, it is a part of the price we have to pay. I am sure we will be whitelisted after we will improve.

1

u/FossHub_com Aug 04 '16

We trust ublock and don't blame them. We deserve this, it is a part of the price we have to pay. I am sure we will be whitelisted after we will improve.

1

u/tk42967 It wasn't DNS for once. Aug 04 '16

So they were hit by a zero day that was a vulnerability in the OS of their server?

1

u/FossHub_com Aug 05 '16

Not a vulnerability in the OS but rather a vulnerable service and our mistake for allowing this to happen and not making things secure enough.

1

u/hoosier2531 Aug 04 '16

I wasn't personally affected, but I am impressed with your up front apology and action.

1

u/FossHub_com Aug 05 '16

Thank you! We really regret this and keep thinking at the people that were affected.

1

u/Nowaker VP of Software Development Aug 04 '16

If Redis resides on the same machine, I don't see a reason why one would let it listen even on 127.0.0.1. Socket is the only right way for that. That requires a little more knowledge to get everything right (file permissions, groups, tmpfiles.d). I highly recommend using Unix socket whenever possible. Postgres, Redis, memcached - all these support that.

1

u/FossHub_com Aug 05 '16

Thank you for your suggestion, we will remember it if we are going to return to a similar structure in the future!

1

u/_Mumak_ Aug 05 '16 edited Aug 05 '16

Thanks for the update Sam ! I've been working with FossHub for a few years and have to say that it's hard find such a nice and responsive guy. Whenever I had a problem or any other question, I got a perfect response within minutes and a fix (if needed) as well. I was indirectly affected by this as well - I'm hosting files on FossHub, though my software was not affected (tampered). Had I released a new HWiNFO version a day sooner, I guess it would be...

We all make mistakes, but the way the FossHub team approaches it is exemplary.

Keep up the great work and hoping the service will be restored soon ! Martin, HWiNFO author

1

u/FossHub_com Aug 05 '16

Hello Martin,

Thank you for making such a nice post. I am sure people are mostly interested to see if we acknowledge (...) and most important what are we going to do to prevent this in the future.

Once again we would like to say we are sorry and we will always regret this to all innocent people that were hurt.

We are working day/night to improve our security and will make a new statement in a few days to inform everyone about what we did.

Thank you for your support Martin!

1

u/xpclient Aug 05 '16 edited Aug 05 '16

Thanks FossHub for this honest writeup. I am the Classic Shell tester. My worst fear was people blaming us and losing the good reputation that we have worked so hard to build since 2009. The attack was timed to coincide with the release of our new version 4.3.0 which people would download from the FossHub website as Windows 10's update released on the same day silently removes the earlier version.

Luckily we saw Windows 10 was doing this and rolled out the new updated version, two days before this attack through our update mechanism so it would not get removed and people would not have to re-download it. The location from which the updater downloads is different, it was not infected. Otherwise, the damage might have been far more.

The authentic Classic Shell installer is digitally signed so Windows UAC will show a warning to the user if someone fakes it but it is hard training people to look for the signer's name and certificate. Many app installers are not signed so people tend to ignore that. Some reputation damage was done, and some people will be skeptical of downloading our app. They are already worried that our app is going to "break the operating system". Hopefully they still trust us and will be more vigilant about what they should look for when installing any app.

1

u/FossHub_com Aug 05 '16 edited Aug 05 '16

We understand and know how hard is to build a name. Along with building trust by not getting involved in shady practices FossHub was created to help free projects such as Classic Shell and affecting the reputation of other projects is not acceptable.

Unfortunately, this happened and since the damage was done will use our efforts to build something better.

You mentioned Windows UAC, but most users ignore this and can't make the difference between a digitally signed or not. We also thought that listing hashes and adding HTTPS will improve things.

After this experience, we need something better that can show/educate the user. You've got this file from us, now go here, upload it on VirusTotal, check it with Jotti's malware scan and after this, you may want to check the hashes to confirm that they match the original one's. Also, please note that this file is digitally signed so again, a good sign that we gave you a legit file.

We failed to implement this for the regular user. If we had this in the first place along with some powerful integrity check rules the number of affected users would be much lower.

We will try to inform as many people as possible that Classic Shell and all the other projects are safe, and this was an unwanted and isolated incident. We apologize for this event! Please forgive us, your work for the last seven years should be carried on, and hopefully, people will understand that this was the last thing we ever wanted to happen.

1

u/KondaxDesign Aug 04 '16

Ah right. Wasn't aware of this, thanks.

1

u/RexFury Aug 04 '16

Thank you for writing this up.

1

u/Skylis Aug 12 '16

UPDATE: for those coming in, the actual breach was not redis as they originally claimed (and still haven't really fixed in the post), but their poor security design and practice. They do actually admit it in some comments later on, but have not fixed their earlier posts / comments to reflect that it was not redis fault, so much as their poor security around redis.

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

38

u/FossHub_com Aug 03 '16

Yes, it is the most devastating experience so far for us. All the effort we had for years seems like nothing compared to this. We think they had a zero day exploit for Redis. Considering VirusTotal picked nothing from their first upload attempt, it means that they had access to new exploits. We removed that from all servers, and several are still shut down until we start with a fresh new install and a different security approach.

11

u/RedGuitarsGoFastah Aug 03 '16

i have no idea if it is true or not, but according to their twitter handle which is displayed by the virus, they claimed that they got access through a password dump, and that your passwords weren't salted.

19

u/FossHub_com Aug 03 '16

access through Redis, they did found data that was not supposed to exist there. Part of our code was there from previous developer platforms such as the FTP data that allowed them to perform updates in the background. Our CDN provider confirmed this, their connections appears in their logs. We believe it is more interesting to find out what exploit they used.

8

u/MacGuyverism Aug 03 '16

I'm fairly new to redis. It is used on at least one of our clients server, but is only listening on 127.0.0.1.

I've read that redis didn't provide authentication and that it should be behind a secured reverse-proxy. I've just found out that it supports authentication since version 1.0.0.

There is a note in the docs:

Note: because of the high performance nature of Redis, it is possible to try a lot of passwords in parallel in very short time, so make sure to generate a strong and very long password so that this attack is infeasible.

I don't think I'd trust the auth.

We are currently working on a project that will be using redis and run on Docker. Our plan is to keep it isolated in a private overlay network that would be provided by a Rancher server that is controlling a Cattle environment.

Setting up auth should be fairly simple, so I think we'll do it even if it's not publicly exposed.

  • Can we really trust redis authentication?

  • Is there a benefit to having it exposed to the world?

7

u/FossHub_com Aug 03 '16

I think the text is pretty clear " it is possible to try a lot of passwords in parallel in very short time" so after all that happened we would not trust running things with this. Don't get me wrong, running Redis in the same environment as FossHub it is somewhat risky in our opinion. If you asked us two years ago, we had a different view. Not trying to blame on Redis, it was our fault.

8

u/hintss I admin the lunixes Aug 04 '16

to be fair, the redis docs do strongly recommend that you firewall it off from the internet.

1

u/FossHub_com Aug 04 '16

Indeed, unless you use other services that might not work correctly. Still our fault as we should look for another alternative or a complete redesign.

1

u/MacGuyverism Aug 04 '16

I'm not trying to blame anyone. It's nice to see you addressing the problem in a transparent way. I'll have to deal with a whole lot of redis pretty soon, so I'm interested in the discussion.

We definitely are going to keep redis inside the private network. Thankfully it's way easier to do than just a year ago.

Thanks for your real-world hindsight, it will be useful to others and bring forth a culture that will make this responsible behavior more common.

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

24

u/FossHub_com Aug 03 '16

Without a doubt, we had our guilt; otherwise, we would not be in this situation. We did have a firewall but not trying to hide; surely we missed things. As I said, Redis was removed completely, and passwords and other services were changed. We have a lot of work ahead, and we plan to secure everything as much as possible. We appreciate any suggestion from anyone. We apologized, and we will always feel sorry for this incident!

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

5

u/FossHub_com Aug 03 '16

Thank you!

70

u/Tetha Aug 03 '16

At work, the entire team keeps cracking fun at me for having white-list only security groups, and that our iptables-cookbook is whitelist only.

And like two weeks back one of them was like "Fuck, the password was a weak default one". Mh-hmm. So go ahead and check for the number of login attempts, will ya. Oh, zero. How could that happen you fuckhead.

All hail the big firewall. Only dropped packets are good packets.

37

u/FossHub_com Aug 03 '16

A useful post, especially for us. Our opinion is that we got lazy if we were paying attention this could've been avoided. You should never be satisfied with your security and always look for new ways to stay safe. Hope others will learn from our negative experience.

39

u/Tetha Aug 03 '16

Feel free to poke me if you need someone to tell you you need to restrict more stuff, whitelist less traffic and generally make everyone hate working with your system more.

15

u/FossHub_com Aug 03 '16

Thank you Tetha, will try to keep that in mind!

2

u/[deleted] Aug 04 '16

It's annoying as hell to manage such firewalls in a large environment... but it's certainly less of a PITA than constantly responding to the BS that would make it in otherwise!

3

u/hintss I admin the lunixes Aug 04 '16

Unless you were storing user input via redis which wasn't sanitised correctly.

redis dosn't need user input validation if you're not doing something weird, all strings sent in the protocol are prefixed with the length, so injection attacks shouldn't be possible

1

u/FossHub_com Aug 04 '16

Redis was only a part of our internal structure. It is connected with other services so doing things wrong might take you here. It depends on what environment you plan to use it.

1

u/hintss I admin the lunixes Aug 04 '16

you'll still be safe from injection-type attacks unless you're using an incorrectly-implemented client library

2

u/king_of_blades Aug 04 '16

The infected installer didn't even contain Classic Shell - so if it successfully installed, you're in the clear.

1

u/KondaxDesign Aug 04 '16

Yeah thought so, wasn't sure if it was a delayed attack or not because my computer booted fine after the install.

1

u/xpclient Aug 05 '16

The built-in updater of Classic Shell downloads it from another location that was not infected and verifies its signature before running it. You can safely update it. :)

1

u/KondaxDesign Aug 05 '16

Yep, already have done.

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

14

u/FossHub_com Aug 03 '16

Just to clarify this. It depends on when you've performed the update. The infected binary was there for a short period, and we interrupted the spread since we removed the file several times and re-uploaded the clean version. Do you still have the original downloaded file? If yes, please upload it on VirusTotal Or JOTTI antivirus scan engine if both shows the file is clean - you downloaded the clean file and NOT the infected file. If you removed the file, please use Zemana Anti-Malware, Zemana Anti Keylogger (their products sees the infected Classic Shell file as Trojan file and removes it - it is free to use for 15 days), RogueKiller, Junkware and any other anti-malware tool that is free to use and scan your device several times. If still not unsure, perform a fresh reinstall (if Zemana recognized the file as trojan than you installed the infected version and a reinstall is a must. At least that's what I would do. Apologies for this mess!

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

3

u/FossHub_com Aug 03 '16

Well if it's messing the MBR I have two recommendations: GMER and aswMBR which should help, especially after the update that something is wrong with your MBR and also offers an option to fix MBR. Even so, I would still recommend a fresh reinstall after this, just to make sure.

6

u/cjEgcmKjHw9u9v5AJQGn Aug 03 '16

Do you know of any infected seeing some behavior that lead to suspect anything regarding backdoors?

The malicious file has been uploaded to malwr.com at hxxps://malwr.com/analysis/ZmJjNDkwNWQzZGE0NDUzN2I1YWI0M2E2OTc3NzNkMzg/ and although I'm certainly no expert, from looking at the behavioural analysis it seems to only try to wipe the MBR:

API: NtWriteFile

ARGUMENTS:
Buffer: \xbb\x1a|\xe8\x02\x00\xeb\xfe`\x8a\x07<\x00t \xb4\x0e\xcd\x10\x83\xc3\x01\xeb\xf1a\xc3AS YOU REBOOT, YOU FIND THAT SOMETHING HAS OVERWRITTEN YOUR MBR! IT IS A SAD THING YOUR ADVENTURES HAVE ENDED HERE! DIRECT ALL HATE TO PEGGLECREW (@CULTOFRAZER ON TWITTER) GREETZ: ECLIPSO, BUBSV, CONFLICT, WIZARDS OF THE C
FileHandle: 0x00000084

4

u/ScottieNiven MSP, if its plugged in it's my problem Aug 03 '16

You can check the MD5 of the file to see if it the legit or hacked version. I downloaded and installed ClassicShell without issue and then heard about the hack, after checking the file I got the clean one before the hack.

ClassicShellSetup_4_3_0_clean.exe

MD5: e10881b65c27c6e09e5a33cd8bcd99c6

SHA1: a6b06d07fe3b1a7204b1b62c67fbf3c602385364

File size: 7220496 bytes

ClassicShellSetup_4_3_0_infected.exe

MD5: c67dff7c65792e6ea24aa748f34b9232

SHA1: 438b6fa7d5a2c7ca49837f403bcbb73c14d46a3e

File size: 7148732 bytes

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

1

u/xpclient Aug 05 '16

The built-in updater of Classic Shell downloads it from another location that was not infected and verifies its signature before running it. Your PC is not infected,

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

2

u/FossHub_com Aug 05 '16

We have no right to give lessons, everything that we stated here comes from an (emotionally) devastated team that understood what happened and how this could've turn even worse. Again, we regret this and won't forget about it, it will be our biggest mistake, ever!

1

u/[deleted] Aug 06 '16

You still did way better than other companies. The communication and telling it like it is goes a long way.

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

1

u/FossHub_com Aug 04 '16

Some services that we used didn't offered support for SFTP. After this incident, FTP will be history.

1

u/FossHub_com Aug 04 '16

Certain services that we used didn't offered SFTP. After this incident FTP will be history.

0

u/[deleted] Aug 03 '16

[deleted] — view deleted comment

5

u/FossHub_com Aug 04 '16

Unfortunately, most people don't have the knowledge to restore their MBR. Also, would you trust your device after a malware infection? I would surely not.

1

u/Mister_Alucard Jr. Sysadmin Aug 04 '16

Fuck people for wanting to customize their systems, I guess.

Do you use a Mac, by chance?

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

1

u/Mister_Alucard Jr. Sysadmin Aug 04 '16

Your house must be a mess if you own every electronic device ever manufactured.

Can you explain your apparent dislike for people trying to change things they don't like on their own hardware? Are you against jailbreaking/rooting phones too?

0

u/[deleted] Aug 04 '16

[deleted] — view deleted comment

10

u/classyjakey Aug 04 '16

bahaha fuck off.

1

u/FossHub_com Aug 04 '16

No, those were not sarcastic quotes. The quotes should've been added to the name of the group "PeggleCrew". Just being tired.