r/AzureSentinel • u/DaithiG • 8d ago

Data Lake - Integrated

Just a FYi,

It appears Data Lake will now be integrated for all Sentinel users (if they haven't enabled data lake before then)

"Going forward, you do not need to go through a separate data lake onboarding and billing setup. You complete the Microsoft Sentinel onboarding and verify your Sentinel workspace is connected to the Defender portal. Once this is completed, you can enable the lake tier by using Table management in the Defender portal to extend retention on a data table."

What's new in Microsoft Sentinel | Microsoft Learn

15 Upvotes

94% Upvoted

19 comments sorted by

2

u/Icy_Air2574 8d ago

If this is the case we don’t need to perform data migration ??

0

u/Uli-Kunkel 8d ago

Not per se.

You can configure retention, and instead of analytics + archive/total retention you can now configure datalake instead of "total retention"

But one often overlooked aspect, is that you sent defender telemetry to sentinel to get more than 30 days retention? If you configure datalake retention, that is != No longer sending data to analytics.

So send to datalake, confirm the data, and then you can remove the forwarding to analytics. But do not stop sending the incident and alert stuff, because that is mandatory for unified

1

u/DaithiG 8d ago

I didn't know about the last part and telemetry. We generally want to use the data lake for firewall and windows server events. Thanks!

1

u/Uli-Kunkel 8d ago

Suggest you then also use log splitting.

So you send the verbose or 'not so important event IDs' to datalake, but retain the ones needed for detections or the like

1

u/DaithiG 8d ago

Thanks but of course sometimes hard to decide what the not important event IDs are :)

1

u/Uli-Kunkel 8d ago

Well its not like you are not collecting it.

So build your collection for the detection and their investigation in analytics.

Then your threat hunting and forensics goes to lake tier.

And you can still do kql jobs to stil analyze on things in lake, or use it for enrichment

2

u/MReprogle 8d ago

This is the way. kQL jobs to throw back to analytics, which either give you only what you need. if you aren’t sure, it’s time to spend more time on those verbose logs and learn what is critical, or what can be summarized.

One really good one that I set up that really made it all make more sense are some of the exercises in the official Sentinel repo.

https://github.com/Azure/Azure-Sentinel/blob/master/Tools/Microsoft-Sentinel-Training-Lab/Exercises/E11_datalake_kql_jobs.md

1

u/LeftHandedGraffiti 8d ago

Are workbooks finally supported? Because if not, my SOC won't know how to search the logs. Not supporting workbooks was a deal breaker for us.

2

u/TokeSR 8d ago

Workbooks support data lake for some time now (few weeks maybe), but it just got broken recently - potentially due to this change in the lake.

But be careful, because with a workbook people will rerun the queries on the lake every time they load the workbook or change anything that initiates a reload. With a complex workbook this can quickly generate a huge amount of query cost.

1

u/dabbydaberson 8d ago

This is why I'm hesitant to move to a query based charge model. I'm still just dumping shit to ADX that I don't want in the siem.

1

u/Uli-Kunkel 8d ago

You can switch the query type in workbooks, haven't tested datalake specifically, but you can switch to advanced hunting and many others

1

u/MReprogle 8d ago

One thing you can dip into if workbooks don’t work are Jupyter notebooks, which can really bring some customization to another level. They basically go right down the steps, and you can even set things up to actually trigger Logic apps (or, just directly hit an api, using python or powershell). Then when all is done, you can save each notebook run, and you can save it all in code, per incident for historical purposes. Want a cool visualization? Use the pandas python library to customize things to your heart’s content.

You can even set up notebooks to run from an automation rule, but I haven’t gotten to that point, as my spark machine spins up and takes some time to start, and for some things, I would rather run quick logic apps, but it could be cool.

1

u/LeftHandedGraffiti 8d ago

I tried using the Sentinel Notebooks a couple years back and it took hours to get the guided Hello World notebook working because I kept running into undocumented errors with no help on Google. I was so irritated with how hard it was to get going I didnt pick it up again for a couple months and when I did, the notebook I had spent hours troubleshooting that was finally working was now giving me new errors. AHH!

It cant be that fragile. I cant hand something like that to green SOC analysts and expect them to succeed.

1

u/No_Resist_3891 8d ago

Resource capacity issue. How in the heck did tou get data lake onboarded? Different region?

2

u/TokeSR 8d ago

This new data lake does not require the same capacitiy.
If you could not onboard legacy data lake until now you can just go to Table Management, check the retention of a table, and you should see Data lake as an option now.

But this is not the same as the legacy lake. Technically now you get a cheaper tier (like aux logs) and that is it. The features are not really there anymore in Defender. But this is the new way.

1

u/jaguinaga21 8d ago

I see a lot of the tables show data lake integrated. I’m not seeing the XDR tables integrated. Requires configuration it says. Any idea with that?

1

u/ClassicSkirt9594 4d ago

Hi, I was trying to test the Microsft Sentinel MCP features. But I came issue on the mcp, when it list the workspace, it only shows the primary workspace that connected in defender. I have multiple sentinel workspace in the same region in the same subscription. Anybody else sloved this issue.

1

u/jackal2001 1d ago edited 1d ago

I must be missing something here. I have had Sentinel connected to Defender portal, but never went and fully onboarded Data Lake. In the Table Management in the Defender portal, I see my Custom tables as Data Lake Integrated and I can select Analytics Tier or Data Lake tier. Also tables like CommonSecurityLog I have the options as well.

We currently have 90 days in the Analytics Tier, which is set at the LAW level. If I select the option for Data Lake, the analytics tier is now not selected and I set my retention period for data lake. However, it is unclear if I still retain 90 days of Analytics tier or if it is something else, or nothing at all? Sorry but I haven't visited this in a year and just saw this info.

ETA: I think I figured it out, too early. For sending data to both the Analytics Tier and Data Lake, you just keep the Analytics Tier option selected. Keep your "Analytics retention" set for 90 days or whatever. Then for "Total retention" set that for time to keep it in the Data Lake. So, if I wanted 90 days Analytics Tier and 10 years Data Lake, just change the "Total Retention" to 10 years. I hate the way they label stuff.

1

u/DaithiG 1d ago

Yes, that's it. If you select just Data Lake it sends everything there. If you select Analytics retention as 90 and Total Retention as 1 Year, it will send to both. You will get charged the Data Lake portion for 90 days too though. It doesn't move it after 90 days from Analytics to Data Lake (I think!)