Snowflake, a cloud storage platform used by some of the largest companies in the world, is investigating a targeted attack on its users who lack multifactor authentication. Join Matt Radolec and David Gibson for an episode of State of Cybercrime in which we will discuss the increased attacks on Snowflake customers and share our five-point checklist for ensuring your cloud databases are properly configured and monitored.
Snowflake, a cloud storage platform used by some of the largest companies in the world, is investigating a targeted attack on its users who lack multifactor authentication.
Join Matt Radolec and David Gibson for an episode of State of Cybercrime in which we discuss the increased attacks on Snowflake customers and share our five-point checklist for ensuring your cloud databases are properly configured and monitored.
WE’LL ALSO COVER:
...and more! More from Varonis
⬇️ Visit our website: https://www.varonis.com
LinkedIn: / varonis
X/Twitter: / varonis
Instagram: / varonislife
[00:00:00]
Matt Radolec: hello, everyone. It is so great to be back for another episode of State of Cybercrime. Hi, my name's Matt Radolec, one of our hosts. I'm joined by one of my esteemed colleagues.
David Gibson: Hey, Matt. How's it going? I'm David Gibson.
Matt Radolec: It's great to have everybody here today.
We have a jam packed show for everybody today. We are going to go through some of our normal segments, and if you haven't been here before, we'll definitely tell you about them. One of the things we're absolutely going to talk about is [00:01:00] good news. There's always some good news to talk about in a state of cybercrime, and we always like to give credit when law enforcement, government, or even other cyber companies get things right.
We'll cover everything and all the happenings that happen in AI with our section on AI Vey, and we'll cover a couple, I think one pretty vulnerable vulnerability, and then jump on the highway to the danger zone and talk a little bit about some threat actors and probably the thing that a lot of you want to hear about, which is Snowflake.
Our first thing is always on good news. We find that in cybersecurity, there's always this doom and gloom outlook. There's only more threat actors and zero days and people coming after your data, and while that's true, there often is a lot of progress by what people would refer to as the good guys on getting stuff done and one of those I wanted to shout out really, the Department of Justice is I'd say in 2024, and maybe even late 2023, has really stepped up their game.
They've recently arrested the alleged creator and operator of a nearly decade old [00:02:00] anonymizing proxy service, botnet, in an operation that they called TunnelRat. And this brought this 9 11 S 5 proxy, enabled cyber criminals to rent zombie machines in the locations of their would be victims, which was something that they did in order to avoid geo based alerting and blocking.
And also, when you think about fraud detection often used by credit card companies and banks, where they're looking for you to like, Making a weird transaction in a new place. If that transaction comes from a place where you're usually, making purchases, it's going to be a lot harder for them to catch that.
And that was really the genius behind this botnet authorities claim that the operator who is based in Signapore was responsible for more than six Billion dollars or right around $6 billion in losses between 2015 and 2022.
David Gibson: Crazy. And the delivery mechanism for a lot of the botnets, I, if I read it correctly, was a free VPN service that you could download right.
And [00:03:00] install. Did I get that right?
Matt Radolec: It's It's kind of a guise of being like a proxy service, where you can get around web broing, but in reality, you're participating as a zombie and a botnet.
David Gibson: Yeah. And I guess the alerts that people get that, hey, some of your traffic is encrypted, that they would go unnoticed there.
But I guess the bigger thing is anytime something's free, it's chances are you're the product.
Matt Radolec: Yeah, that's a good way to look at it. Now, that's not the only good news though. It seems like Department of Justice has been hard at work and they've even made a few arrests, David.
You want to tell us about that?
David Gibson: Yeah, Operation Endgame. I guess good guys assembled. And they took down 100 servers, 2, 000 domains, they arrested 4 people, made these other 8 people infamous. They're in countries like Russia, I think one's in the UAE. But they were able to, kinda, not arrest everybody, but for the people and took down a lot of the infrastructure.
And this network was responsible or used a lot of the malware that we've seen over the years, right? TrickBot they used like ICED ID yeah.
Matt Radolec: And I think they were a part of QuackBot or [00:04:00] QBot as well, which is something that we covered. Dave, when the show first started, I feel like we were talking about that on nearly every show.
First, it was 100, 000, then it was a million, then it was something like 3 million people had gotten infected by that, and, had fallen bit into the TrickBot malware, and the QBot, and the QuackBot malware. Just to zoom out on this for a second, though, we're covering a lot more arrests than we did in the past.
I remember early last year, we spoke a lot about coalitions, U. S. federal government, It says executive order that the U. S. is going to collaborate with law enforcement in foreign countries. Then we saw Interpol and U. K. Office of Cybercrime say the same thing. Are we starting to realize the gains from that now?
Like, Are we starting to see the actual operations, the takedowns the interception by law enforcement? What do you think about that?
David Gibson: It doesn't come from nothing, right? So there's got to be this looks like evidence of some effort and hopefully the scales can balance a little bit more than they have been, right?
Matt Radolec: Yeah, and we keep, reinforcing to cyber criminals that if you do the crime, eventually you're going to do the time. In our next segment, [00:05:00] and our newest one, AI Vey, everybody loves hearing about AI. Nobody is tired of it, and I'm sure you're all saying AI Vey to yourselves right now. But we'll cover the good, the bad, and the ugly of the happenings as it's related to AI and our eventual demise to robot overlords.
First up, though, let's talk about Hugging Faces. What is this and why should people be worried about vulnerabilities or exploits from actors in Hugging Faces, David?
David Gibson: This was pretty amazing. I learned a lot. I learned a lot about malicious pickles. Which I hadn't been aware of before.
I'd never liked pickles, but I didn't realize that some of them were malicious. But it, what I learned from this more than just the stolen authentication tokens which they are addressing, get more granular. But if you have an authentication token, you can get in to whatever service you have the token for.
In this case, it's an AI platform. And it the weakness in this AI platform where basically you can build your own LLMs. [00:06:00] And, if you think of an LLM as a passive thing it's actually a lot more complex than that. They have to be complex because in doing so, In addition to referencing some of these inference APIs or some of these kind of a, like you might say, GPT 4 as a language model.
In addition to referencing those, you can build your own. And when you do that, you're uploading a an LLM package, and some of these AL model formats some of the popular ones, there's SafeTensors that's popular, and there's PICL, or PyTorch is another one, and what the researchers have recreated is you can upload a malicious PICL package and this is where it gets scary, You basically can do a remote code execution from an LLM.
And because this infrastructure is shared, you can then move laterally. And in this case, there was a lot of Amazon infrastructure underneath that was potentially accessible like [00:07:00] made other people's, other companies LLMs accessible. I the good news is a lot of this was discovered, I think, early.
Before some of these vulnerabilities were really exploited more than the tokens, but I think it's certainly safe to say that some of the end user data has been, it's, you potentially have to look at that, but I think the bigger story and something that we might see more of is this remote code execution on these LLMs.
Matt Radolec: Yeah, it's almost as if we need to think not only about securing our LLMs and protecting our own organization's data from attack inside of a, from a co pilot or inside of a large language model, but we also need to think as an industry about securing AI as a service. And this wasn't the only story about this, David, there was another one, right?
David Gibson: In fact, there was and it's clearly happening more by exploiting vulnerabilities in AI LLM containers. This one was a container named Cog, not Pickle. And once they were able to exploit that remote [00:08:00] code execution they were able to see infrastructure in GCP. It's very similar.
Matt Radolec: Yeah, I keep having flashbacks here to when the App Store came out and the big debate was between did you get an iPhone because Apple supposedly, and I think they do, runs like tests and validates the apps that get uploaded to the App Store to make sure they're not stealing data off your phone, whereas like an unlocked phone or the Google Play Store does not have quite as many restrictions on developers that upload applications.
Yes. Do we need to jump headfirst into that from an AI as a developer service standpoint and make sure that AI vendors are validating the, things that get uploaded to their services?
David Gibson: That, that's actually a great question. And apparently Pickle is it's a Python library and by its own documentation says it's not secure.
It's vulnerable to potential serialization or deserialization attacks and things like this. And so I think that's a good analogy because if you were Apple, you probably wouldn't allow pickle based apps in your app store. If you'll allow me, Matt, I really did want to ask the audience [00:09:00] a very important question.
Which band name do you like better? Malicious Pickle or Pickle Malicious? If you could just chat in, I I'm struggling with that, and and if you could help me out, that would be great. That would be uh,
Matt Radolec: be helpful. Megan says, oh, a lot of malicious pickles.
A lot of malicious pickles. A lot of malicious pickles. That's very helpful. Thank you. But maybe what we're getting at is that this style of attack, we'll see if this catches on as a trend. is this AI as a Service injecting bad code into an AI as a Service is a malicious pickle attack. How about that?
Let's see if it sticks. Let's see if it sticks, okay? But speaking of things that stick or don't stick, I feel like I gotta come out and say it. We need to recall recall. This is another happening in AI that is from a just a complete dystopian nightmare. I'm sure some of our audience members have watched Black Mirror, and Recall brings me back to that episode where you can replay things that have happened in your life.
And I think Microsoft has actually already recalled it already. by not enforcing users to have to opt into it as opposed to being on by [00:10:00] default in the new co pilot laptops. And Kevin Beaumont, a former Microsoft Threat Intel Analyst, calls it a disaster. He says that with just two lines of code, you can see anything that your computer has.
How Recall works is it captures what you're working on. And it allows like the co pilot AI to search through it and find it faster. So on the surface wow, great, another thing that's going to help me be more productive at work. It's incredibly dangerous. Anything that you've ever opened or worked on your computer, even files that aren't stored on your computer, but stored in the cloud, could be seen, at least in part,by recall. And so I thought to myself man, that's pretty dangerous in the hands of an attacker, but also in the hands of law enforcement. Like they now can see like snippets and parts of files or even like government snooping on a laptop that they get control of, even if that original document sits in a secure container somewhere else.
I even thought a little bit about how, working on investigations a lot, David, when you're doing insider threats, you have to prove motive. And I feel like you recall would also be really useful at that point. Trying to show the motive of someone and building that story towards [00:11:00] why it is they were doing something on a particular computer.
But the good news for everyone here is Microsoft has already made it so that you have to opt into it, but many security pros, myself included, are really urging people to stay away. So I'll say it again, I really think we should recall, recall.
David Gibson: I think you're onto something there it's a very scary thing all your passwords are going to be in there because it logs every key, I mean you might as well print all the, all your, every, all your keyboard history on post it notes and paste it all around your house, it's and the other thing is it's we've seen what happens when you take stuff out of context, like recordings or snippets, it's just imagine the use of this if you wanted to, convince somebody that, yeah, oh, it's just it just, it's a really scary thing.
Matt Radolec: Yeah. Now I, I know probably for some of the most vulnerable vulnerabilities that are out there, and the one that caught our attention, David, other than all this stuff that's going on with AI, is some routers, some massive number of routers came under attack. What the heck happened there?
David Gibson: Yes. And, I think there's [00:12:00] a trend for CD Flora that starts with a P because apparently the ISP was codenamed Pumpkin Eclipse, but all of these routers in this ISP got taken down all at once.
It was about 600, 000 small office home routers, like Zoho routers, right? And It basically installed Chalubo, which is another cool name, but it's a remote access trojan, it's actually a pretty common one, but then it bricked the SOHO routers. So this is a pretty scary thing this happened over a three day period single ISP, taken out.
And I think we've seen this before as a precursor to war. If this is like a proof of concept thing, that's really scary.
Matt Radolec: Yeah.
David Gibson: Yeah. So I, it's just that the scale of this and just one flaw, when everybody has One common component that they rely on. That's a massive single point of failure.
Matt Radolec: And just before we we, we jump on to the danger zone and talk about probably why a lot of people are here, was talking about Snowflake, the chat has really lightened up about this recall thing. And I just want to go over a couple of things that people are [00:13:00] saying here. One from Bradley, only a threat actor could gain control.
opt in to recall and then basically reduce their footprint to just utilize recall to do all of the hard work on, the recon and the data theft and you wouldn't even necessarily know what's happening. The other thing it really like shines for me is, and I'm curious, a couple people want to put in the chat, anyone use like a password manager or password vault where you have to log into a system and then you can see the password for just a second and then you have to copy and paste it?
Like I now can just. Go back in the recall to see what those passwords were. And I just feel like maybe that this isn't the right, there's got to be some way that we can limit it or censor what goes into it. It seems like it was something that got, that just got rushed out and is way too dangerous versus being beneficial from a, at least a corporate, enterprise, use case.
David Gibson: Yeah, it's a nuts single point of failure there, too. It's even if you're really, paranoid and, have a password manager and put a pin in after everyone, or before everyone, or in the middle of everyone,
Matt Radolec: you're going to get the pin! [00:14:00] You're going to get old! Yeah
now in our final segment, The Danger Zone, we usually cover threat actors and happenings in the cybersecurity space that people should know about.
And first up is this North Korean threat actor, formerly known as Storm 1789 now called Moonstone Sleet, who is really known for, stealing money, financial type things. Fraud and Cyber Espionage. And why the rebrand is they've really stepped up their game. They're known for setting up things like fake companies and fake job opportunities in order to engage targets, but they also employ trojanized versions of legitimate tools.
I'll give you an example with one related to PuTTY in a second. And they've even created a fully functional video game in order to get users to put the malicious code onto their computers. When you look at. A cybercrime, but APT grade cybercrime. There are some similarities to your everyday ransomware as a service operator, but their bag of tricks is just way more sophisticated and harder to avoid.
They have custom exploits, custom loaders, tools that aren't [00:15:00] going to be detected. You got to find the, them attacking your The outcomes and their targets, you're not going to find them from catching the exploits or the CNC infrastructure that they drop on your computers. I think one of the most novel things that they did was they repackaged their own version of Putty to create malicious terminal sessions to LinkedIn and Telegram, to send people malicious information.
Zip files that otherwise appeared as completely legitimate. I think another thing that kind of stuck out to me about this was that there is a little bit of a targeted nature of it as well. So they'll create a custom loader that looks like it's from software that's been packaged by your company. Once they figure out how your company packages software.
So they're going to extra lengths to try to disguise themselves as an otherwise legitimate service and kind of copying legitimate tools.
David Gibson: Yeah I, they're, they actually, I think they, they have websites that look like legitimate businesses, and this malicious game, I have to, I when my son gets home from school, I'm going to ask him, did you ever play D Tank, [00:16:00] D E Tank?
Because then I got it. Yeah, there's one, and there's another one, like D PHI Tank, right? Which I'm hoping is more crypto related, but and he wouldn't have been interested, but D Tank sounds right up his alley. Yeah, that's a good thing, yeah. There's
Matt Radolec: another popular D Tank. and TKM. Now that's not the only actor though, David that people should be worried about, and especially if you're a critical infrastructure or a healthcare company Black Boston should be on your radar.
They've been touted by both the FBI and CISA as a major threat to the global healthcare sector. their MO is the pretty typical ransomware actor, disrupt your operations, steal data, unleash ransomware. And they've had some success and seen success in the healthcare industry, so I'd like to say that they're refocusing their metaphorical eye of Sauron on healthcare companies across North America, Europe, and Australia, and it's something, just to keep in mind about this, The, the TTPs tend to be pretty similar across these different ransomware actors.
They're either going to use an initial access broker to land, try to find the [00:17:00] data that they're looking for, something that's going to give them a meaningful bit to exfiltrate and ransom. But they're also heavily focused on disrupting operations, as that's what, is going to lead someone to pay the ransom, right?
Ultimately, to gain control of their network, get their billing system back into, back into works. Definitely something to keep an eye on and be concerned about if you're critical infrastructure, but especially if you're a healthcare company operating in, North America, Europe or Australia.
David Gibson: Yep. These actors keep going keep using similar techniques. They keep making use of spiders. That's I say that because they're, apparently they're related to WizardSpider as well, or they have some resemblance to them as well. Yeah, maybe
Matt Radolec: a rebrand, maybe we got like a Black Cat, Alpha Bee, one person and a couple people from WizardSpider and they do a rebrand.
Now, that's probably not the biggest story though, right? The campaign, both David and I have been really excited to talk with everybody about. And I think if we weren't at, Snowflake, or at InfoSec London, or at the Salesforce London event this week, I probably would have had the show earlier in the week.
It's Snowflake. Now, as I look at this campaign against Snowflake [00:18:00] customers, I'm having flashbacks to 23andMe. And we'll talk more about that in a couple minutes, but, David, something happened early on when the Snowflake breach hit the news. It got people really worked up, and it appears like it might have been some fake news.
Can you tell us a little bit more about what we're talking about?
David Gibson: Yeah, despite the early report for the contrary And uh, which has been taken down. There is no evidence, this is a quote from I think Snowflake and Mandiant. There's no evidence to suggest vulnerabilities in Snowflake's product or did any data exfiltration caused by compromised credentials of a current or former Snowflake employee.
That was the initial report. But what it looks like is that a lot of Snowflake accounts don't have MFA enforced. And this is, obviously when you don't have MFA enforced on the Snowflake account that's a pretty huge risk factor, as I'm sure everybody on this webinar knows. But I think that really, enforcing MFA has traditionally fallen in the realm of the customer's [00:19:00] responsibility.
in the shared responsibility model and I think, and this probably is what you're saying with the 23andMe comparison, maybe such critical data should not be accessible without multi factor.
Matt Radolec: I'll call it out now. I think we should be pressing regulators to say that, if there's some, if there's sensitive regulated data behind whatever thing you're running and authentication in front of it, you're like opening the door for what happened here.
And I don't, why I want to take it back to 23andMe is, and Snowflake didn't do this, in the initial handling of the 23andMe incident of a similar type, that they blamed the users for reusing passwords and using weak passwords. I can't sit here and say it's the Snowflake users fault for not turning on MFA.
We could say, hey, the industry responsibility model, the companies that bought and operate Snowflake should be the ones that are turning it on, and we'll actually show you how to do that in just a second. Yeah. It just begs the question are we at a point where, I know some people are saying, oh, let's move past passwords, let's move past multi factor, but are we at a point where it should be [00:20:00] mandated, potentially by a regulatory body, and I'm usually never one to suggest regulation is the answer so that if there's sensitive data in there, we have to have multi factor authentication in front of it.
David Gibson: It certainly is a, valid question. I don't know if the, yeah. I don't know what the argument is against it, except it's inconvenient. It's interesting, like with 23 and me, I think it's that's used by everybody with so many different degrees of technical sophistication, so I think it's a very.
It's a big argument, right? You got to have multiple, you got to enforce it because there are a lot of people that don't know any better. I think you could argue that, corporate users should potentially know better, but then again, we see a lot of non technical business users or, non technically oriented, very, data scientists that, you know, but they wouldn't necessarily think of it.
And the other point on this is, I think, since it's a corporate application, and maybe correct me if I'm wrong here, but you could. potentially forget you didn't have MFA on. If you have MFA on with something like Okta, Okta, and use that to authenticate to Snowflake, but you didn't do the actual Snowflake [00:21:00] account, should we unpack that a little
Matt Radolec: bit?
Yeah, so what David's getting at, and Varonis, we call this like identity versus logical identity. David and me, we have accounts in the Varonis domain, but that account exists inside of SaaS applications as well. And if that account has some password that is replicated to Snowflake, and it's brute forceable because the multi factor is not on Snowflake, it's on the identity provider that we put in front of it, and we didn't disable direct login on the other side, There is a way for an attacker to get through that, and this happens quite often, where organizations think that everything is routed through their VPN, so it's never going to happen, but they don't realize that they can also log in directly to the application.
All this said, we do have a little bit of a get well plan if you want to start to unpack that, David.
David Gibson: Yeah, make it so. Oh, so I guess that's for me. I was going to
Matt Radolec: give you a chance to tell people what the first thing they should do is.
David Gibson: Yeah we are able to very quickly install and take a look at Snowflake, if you're not already looking at Snowflake with Varonis, and isolate, identify those users that don't have MFA enabled.
So that's a [00:22:00] very simple thing for us to do right out of the gate. And and after that, what do you recommend there, Matt?
Matt Radolec: Yeah, I was going to say one, look, step one, you got MFA on no, turn it on. Step two, whether you do or not, you Probably have sensitive content inside of Snowflake, so it's important to find out where it is and where it's exposed.
Because another way that sensitive data could leave your Snowflake is every user, and we see this quite often, has the ability to share outside the organization or, collaborate on, particular data inside of Snowflake, and you're allowing them to share sensitive content. So where this got, you know, kind of like a Daisy chain effect or a Russian doll effect for this attack is initial person gets brute force.
They have high levels of sharing privileges they share externally so that now the attacker, even if you overcome the brute force in the MFA, they now have persistent access through sharing and collaboration. And so step two is figure out like where the exposure is to be able to figure out where, again, where that sensitive content is, and make sure that it's not shared externally or publicly, and if it is to do something about it.
What happens next? Dave? What else?
David Gibson: Yeah, then definitely ought to [00:23:00] take a look at what's been happening with the sensitive data, right? Whether that has been shared has been accessed, usually if there's any evidence after an account was compromised, right? What they've been doing.
Or you can potentially see if somebody's looking like they've been accessing a lot more stuff than usual as well.
Matt Radolec: And I was on one of our healthcare clients earlier this week, and they swore that this attack, quote, couldn't have happened to them, quote, because they have MFA enabled everywhere, quote.
And I, really tried to dig into it with them, and what we did was we looked at the audit events. of the actual authentications to Snowflake. So regardless of how they thought they had checked the permission settings already and whether or not the config was set, as it turns out, there were still dozens and dozens of users that were authenticating without MFA due to like third party agreements, APIs, and other methods.
And so even though they thought that they had it set up like through a, identity provider like Okta, As it turns out, there's still many users that could get in with just a username and password. All of these would have been vulnerable to that, password spray, account enumeration, brute force [00:24:00] style attack that led to that massive kind of widespread compromise.
So audit, see if people outside your organization are accessing that sensitive content, but two, if they're authenticating without MFA.
David Gibson: And that's a good thing to capture if any account has MFA turned off, and, monitor for that, as well as what should we be looking for in terms of the activity there.
Matt Radolec: Yeah, and what your network access policies are. To go back to that use case around, can someone get to Snowflake without being on your corporate network, on a VPN, or have through single sign on, if you allow direct access to Snowflake, you might want to consider sitting on policies that would prevent that.
And then the last thing David I'll, If, again, you should be doing this all the time, but definitely think about monitoring your snowflake.
David Gibson: Absolutely. And legitimate people will end up needing access to sensitive data in order to do their jobs and making sure they're not compromised.
Those accounts, MFA apparently can be bypassed too. It's not a catch all for everything. Definitely want to be monitoring what people are doing after they authenticate. I think Dan said very well, [00:25:00] folks, post authentication attacks do exist.
Matt Radolec: Yeah, and we had a good question from Moira, maybe we could cover David, which is, was Snowflake data for different customers leaked?
Were they all accessed using different credentials? At first, it looks like it was one employee that had tenant access.
David Gibson: So that was the first report that I mentioned that has since been taken down and has been challenged by Snowflake and Mandiant. I know of at least two companies that have had data, there's been a report that data, Snowflake data was out there, but those it sounds like may have been a result of compromised credentials.
Is that your read on it too?
Matt Radolec: Yeah. And again if you guys hadn't heard of 23andMe and maybe our partners. production team can post a link to our state of cybercrime episode about 23andMe. If you guys could do that'd be awesome. It's very similar where there's just a massive, think of it like a password spraying attack where there were either known usernames and passwords or maybe a combination of known and unknown, and some were guessed, and they were able to simultaneously gain access to many snowflakes because of password reuse.
I didn't see any other [00:26:00] questions come in. Let me just do a quick read through. Otherwise, as always, I want to thank you guys, our audience. This show is made possible by our audience, and thanks for your participation and being here with us. We hope that you enjoyed the show today.
Anything you want to add, David?
David Gibson: Just thanks so much, everybody, and love the comments in the chat. And I'll let you know how my band Malicious Pickle does. Yeah, how
Matt Radolec: Malicious Pickle does. Or whether or not we can get the news outlets to pick that up as this, AI as a service, remote code execution and injection attack.
Well, Thanks so much. If you guys wouldn't mind just filling out a quick feedback survey otherwise, have a great rest of your day.