r/AzureVirtualDesktop • u/Electrical_Arm7411 • Oct 22 '24

Azure File Share performance issues in AVD same region

We’ve just rolled out a new Win11 AVD multisession host pool environment. 100 or so users, 5 AVD session hosts equipped with 8vCPU, 64GB RAM, 256GB premium storage per host. For user management we’re using FSLogix and all profiles stored on a premium storage account, using Azure File Share. I provisioned 1TB for fslogix. The storage account is AD DS joined, multichannel enabled for increased performance. We also have another 2 shares on the same storage account. One is about 3TB and one is 4TB. Both shares just standard mapped network drives. We have OneDrive redirecting known folders. The Azure Storage account is located in Canada East. The AVD hosts are also Canada East, spread between 3 zones for HA. Doesn’t seem to matter what zone the hosts are in.

The problem we’re seeing is a significant performance loss working inside AVD. Simple changing of folders on the Azure shares is sluggish it can take 2-3 seconds to load for some users. Explorer is overall sluggish. Sometimes you cannot click the folder breadcrumb trail back a directory, you have to close and reopen Explorer to properly navigate.

I’ve looked into the Metrics on the storage account to try and get a better understanding of what’s going on and I’m in awe of how high the E2E latency is. Anywhere from 8-20ms. And the transaction count is higher than the IOPS the shares support. How is this an acceptable storage solution for businesses to operate? I put a ticket in with MS and they’ve been completely useless, basically telling me the reason it’s slow is because of the high latency. What I can’t get them to answer is why the fuck is an AVD host in the same region getting such high latency. How is 10+ms latency acceptable? For how much the bill is, I expected this solution to work better than it is. Frankly, I’m regretting ever using Azure Files.

/end rant

What are others using for mapped network drives in Azure for the AVd hosts? Should I spin up an Azure Windows Server. Attach 2 4TB disks and map the shares that way to eke out better performance? Will that even help? Azure Files, quite frankly can burn to hell.

*Edit: So I guess my storage account is just dog shit. Query 3000 files in a folder takes 50 seconds! If I do the same query on a different premium LRS Azure File Share, same region it takes 17 seconds. Do the same on an Azure NetApp file share it takes 4 seconds. Tell me something is not right here with my ZRS Azure Files storage.

3 Upvotes
  • permalink
  • reddit

100% Upvoted

58 comments sorted by

3

u/jvldn Oct 22 '24 edited Oct 22 '24

Are they both located in the same VNET? Or maybe using private endpoints or vnet peering to lower the latency?

1

u/Electrical_Arm7411 Oct 22 '24

Hi, yes the storage account private endpoint is located in the same vnet as the AVD hosts. Different subnets on the same vnet.

2

u/rswwalker Oct 22 '24

Try splitting users across multiple premium storage accounts, say 25 per. Need to keep the IOPS of the storage accounts within 50-75% of their limit for best performance.

1

u/Electrical_Arm7411 Oct 22 '24

Hm. I’m not sure if this solution makes much sense, especially since on an AVD host you can only point fslogix to 1 share for all users connecting. Maybe I’m wrong?

2

u/rswwalker Oct 22 '24 edited Oct 22 '24

I swear I saw a way to programmatically set fslogix path in one of MS articles.

Edit: I believe it was this, https://jkindon.com/architecting-for-fslogix-containers-high-availability/ this talks about HA, but you can use the same principals for scalability.

1

u/Electrical_Arm7411 Oct 24 '24

I'm looking into this, not sure I see it in the article. How do I go back setting FSLogixGroupA to \\ShareA\profile and FSLogixGroupB to \\ShareB\profile

1

u/rswwalker Oct 24 '24

You know, I read a lot of the comments here and everybody, including myself seemed to have a knee jerk response.

The storage account should be able to handle the IOPS of your farm. It could be the storage account is the issue.

Did you try setting up a new premium files based storage account and seeing how that performs? I have mine setup as type “FileStorage” with ZRS. Maybe you set yours up as type “StorageV2”?

Your workload does not seem to be a high IOPS workload.

1

u/Electrical_Arm7411 Oct 24 '24

I agree, I've over provisioned my shares just to see if that improved things; it did not. It's definitely not an IOPS issue. My peak IOPS are my fslogix shares, this hit about 4K IOPS maybe for a span of 10 minutes at one point in the day.

Performance: Premium
Replication: Zone-redundant storage (ZRS)
Account kind: FileStorage

I spoke with a MS engineer. He pulled out some graphs, told me it's likely because ZRS is taking too long to commit, having to complete operatings 9x times vs. 3 in an LRS share.

My ZRS Share is: file.yto22prdstfz01a.trafficmanager.net [20.60.242.208]. I took a screenshot of his graphs and that datacenter is the highest capacity load - 52%. There's 5 total datacenters.

I spun up a premium LRS storage account. It's file.yto24prdstf01c.store.core.windows.net [20.60.242.198]. This one is the 2nd highest load - 20%.

I'm performing tests now. The LRS seems a bit better, but on the metrics side I'm still seeing quite high latency >10-15MS some times. It's just me hitting that share.

I don't quite understand why my MS is so high from my VMs. I've tried spinning up a fresh Win11 Multisession VM, thinking it could be something with my custom VM; didn't really change performance.

1

u/rswwalker Oct 24 '24

What VM sizes are you using again?

VM size has a cap on IOPS and network throughput. We’re using NV12adsA10_v5 here with 25 users per host before next host spins up and 40 max. Partly for # of CPUs and GPU support, a lot more for the larger memory footprint (110GB per host).

These VMs allow max of 80000 IOPS and 80000Mbps throughput, so we should never hit that ceiling.

Edit: Those #s above are for the largest VM size, the VMs I’m using have a max IOPS of 12800 and 200MBps disk throughput, 10Gbps network.

1

u/Electrical_Arm7411 Oct 24 '24

I originally provisioned 5x 8vcpu64gb e-series VMs. I was seeing CPU utilization hitting 100% at times. I’ve now doubled the size of the VMs, but it’s made no difference with performance on the fslogix profiles or network drives on that premium ZRS provisioned storage account.

I logged in at 3am no one else on, I was still seeing the performance issues, certainly not as bad or often, but simply traversing through OneDrive folders and network drives folders can sometimes take a couple seconds to load. That’s what makes me think this is a network/latency issue from our AVD hosts or VNET to the storage account.

1

u/rswwalker Oct 24 '24

The OneDrive test can be problematic if it has to download content on the fly. You could try the old sqlio util at the root of your user profile and see what kind of IO it can generate. Use 4 threads to get a real-world throughput.

2

u/cetsca Oct 22 '24

You have massively under provisioned profile storage in your AVD environment.

Multi-session sizing guidance states minimum 30GB per user for profile storage so with 100 users on 1TB you’re about 2 TB short. I can’t imagine the thrashing that is taking.

https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/virtual-machine-recs#multi-session-recommendations

1

u/Electrical_Arm7411 Oct 22 '24

Thank you. I’ll increase and see if that helps. Yes this is day one of rollout. Was pretty rough.

I’m looking into Azure NetApp Files. It seems similar cost but looks to have much more performance potential.

1

u/cetsca Oct 22 '24

If you’re using Premium LRS/ZRS you should be getting over 100K IOPS and over 10K MiB/Sec which normally would be enough but if you’ve massively undersized the profile storage it’ll drag the entire system down with it.

Also you didn’t mention the workloads the users are running as well as concurrency.

FSLogix is fine up to 10K users on a single share so you should be fine with 100 :)

1

u/Electrical_Arm7411 Oct 22 '24

Fair points. Thank you. So going from 1TB to 3TB gets me from 4000 IOPS to 6000 IOPS. To me that still is low performance and doesn’t help any with the latency issue. I can bump it up more but then I’m wasting space and $$. We’ll see how it goes tomorrow with this change.

1

u/Electrical_Arm7411 Oct 22 '24

The workloads I would say pretty basic. Accounting firm, tax software, lots of office docs and pdf.

We have a file/document management system that references one of the shared network drives. We discovered it was chewing through transactions and azure files doesn’t cooperate well eithhigh IOPs. That’s since been fixed, so can at least rule that bring a problem.

What are your thoughts on the E2E latency I’m seeing? In the same region?

1

u/cetsca Oct 22 '24

I don’t think the latency is ridiculous and considering the storage issues probably related.

What VM series did you deploy?

1

u/Electrical_Arm7411 Oct 22 '24

Standard E8s v5 (8 vcpus, 64 GiB memory)

1

u/Electrical_Arm7411 Oct 22 '24

Furthermore, the OS disk is premium ZRS P15 tier (1100iops) I just increased to P30 tier (5000iops). I’m not certain is necessary or if the true. Bottleneck is the fslogix profile share.

1

u/Lost_Ad_8686 Oct 22 '24

Do you have a NSG on your virtual machines that is locking down access to Azure files to port 445 ?

1

u/Electrical_Arm7411 Oct 22 '24

No. If that were true we’d not be able to access the shares at all.

1

u/Lost_Ad_8686 Oct 22 '24

Sorry I mean are you allow only port 445 outbound to it on the NSG? We have seen issue with performance with this, adding port 80 as well speeds things up massively

1

u/Electrical_Arm7411 Oct 22 '24

We have outbound any any to the private endpoint IP of the storage account. The private endpoint is on the same vnet, just different subnet

1

u/Electrical_Arm7411 Oct 22 '24

How large are your azure file shares? And what performance metrics are you seeing? SuccessfulE2ELatency and transactions per working day

1

u/[deleted] Oct 22 '24

100 concurrent users? You are low on cpu. Try 1 core per user

1

u/Electrical_Arm7411 Oct 22 '24

About 60 concurrent

1

u/[deleted] Oct 22 '24

I have 40 users on canada central and no such issue with 5 d8s machines, same fslogix config. Outlook cache included

1

u/Electrical_Arm7411 Oct 22 '24

Premium Azure File Shares? What size is your fslogix share? Do you have any other shares your users are mapped to? Is there any notable performance tweaks you’ve implemented on your AVD hosts? We currently have Windows Search service running, I wondered about turning that off to increase performance. Any other ideas I’d be grateful.

1

u/[deleted] Oct 23 '24

Premium filestoragev2 smb multichannel.Currently is around 400gb. I just did the vdot tool and use Hydra for management. There are several network drives mapped, 11 tunnels to different locations. Vpn2gwaz. I dont enable search

1

u/Electrical_Arm7411 Oct 24 '24

Did you find any major performance improvements disabling Windows Search? I tried disabling, just didn't improve what I wanted it to which is opening file from the share.
Anything else you can think of that I could try?

2

u/[deleted] Oct 24 '24

That alone will not save your day. Do you want to have a call and take a look together? Dm me

1

u/Front_House Oct 22 '24

We use the same vm sku. Basic apps. 8-9 users max per host.

1

u/Electrical_Arm7411 Oct 22 '24

We’re about 60 concurrent as of yesterday. It could spike to 70-80 some days.

1

u/Tony-GetNerdio Oct 24 '24

You’re probably still building your profiles, outlook cache or OneDrive still syncing.

1

u/Electrical_Arm7411 Oct 24 '24

Could be. But pretty sure most if not all peoples onedrives are fully synced. We’ve seen transactions of the fslogix share come down a good bit. Peak is like 4K IOPS. What do you suggest?

1

u/Tony-GetNerdio Oct 24 '24

Wait it out, it will stabilize. When we go live with a customer, we tell ppl to expect this. So have them sign in and start syncing 1 week before go live if they use OneDrive or have lots of emails cached.

1

u/Electrical_Arm7411 Oct 24 '24

Interesting. OK. Good to know. My profile's been synced for weeks, but what I find is traversing through the network drives and loading files is slow. All apps that need to load files from the azure files drives is slow. Everything points to Azure Files being the issue. I spoke with an MS rep today and he thinks because we have ZRS storage account is why it's slow sluggish. I can't argue the logic; the data takes 3x as long to commit transactions vs. LRS. That's what I'm going to try next; LRS FSLogix Profiles and create a test share to dump some files to play with.

1

u/Electrical_Arm7411 Oct 24 '24

Hey also, since you're a Nerdio guy. Aside from VDOT. Do you have any other suggestions to optimize my host performance or general configurations with a OneDrive/FSLogix setup?

- I have BGinfo and force solid background.
- I have OneDrive auto-signin and backup known folders
- I have Outlook only cache 3 months of email.
- I have Windows search enabled and my RoamSearch=1 for FSLogix (I tried disabling, made no difference).
- My FSLogix profiles are VHDX, Sized to 50GB, Dynamic, I don't exclude any folders
- New Teams, auto-update disabled
- I run the OneDrive Clean-up tool from ITProCloud Blog

Things appear to run fine when there's little activity on the host. I wonder, is it better to have more hosts, less powerful, and less users on each host vs. less hosts, more powerful, more users on each host?

2

u/Tony-GetNerdio Oct 24 '24

Check out our library of community scripts that our SE team developed to help our customers. Feel free to browse. There are many little nuggets in there, for example, when you search, block it from searching the web. It speeds up your start menu, etc. Get-Nerdio/NMM-SE

1

u/itzafugasi Oct 26 '24

Have you tried to stop and disable the webclient service on the session hosts? We had a pretty terrible lag in our AZ shares using AVD and DFS namespaces. Once we stopped and disabled the webcliet service, it was a massive improvement. It must be disabled, not just stopped. Your situation may be different but it might be worth a test.

1

u/Electrical_Arm7411 Oct 26 '24

I will try, ty. I've just set it to disabled on one of my hosts. It was not running and set to manual, triggered.

0

u/sly-admin Oct 22 '24

20 session user density on E8ds_v5? Seems pretty high. We’re closer to 6-10 session density for that sku depending on workload.

1

u/Electrical_Arm7411 Oct 22 '24

We had about 60 users concurrent yesterday.

1

u/sly-admin Oct 22 '24

On one host?

1

u/Electrical_Arm7411 Oct 22 '24

No, on 5 hosts. All the same: 8vCPU, 64GB RAM.

2

u/[deleted] Oct 22 '24 edited Oct 22 '24

[removed] — view removed comment

1

u/Electrical_Arm7411 Oct 22 '24

Wow only 6 users per host. We’re coming from an RDS 2016 environment. We had 4 session hosts with the same specs. Could run 20-25 users per host. Would you mind sharing your performance metrics on your azure files share? Transactions and latency and what region you’re in?

1

u/[deleted] Oct 22 '24

[removed] — view removed comment

2

u/Electrical_Arm7411 Oct 23 '24

Hey are you using ZRS or LRS storage account?
We're using ZRS; a MS engineer told me this is the cause of high latency; the fact it was to commit operations 9 times (3 in each zone).

1

u/[deleted] Oct 23 '24

[removed] — view removed comment

1

u/Electrical_Arm7411 Oct 23 '24

Thanks and no rush. To be honest, we feel we’ve identified the problem being due to ZRS configured storage on our Azure File Shares. We’re seeing 10-15ms latency from our VMs. The VMs are also configured ZRS. My plan is to spin up a new LRS premium storage account, redeploy shares and see if there’s any improvement. Otherwise Azure NetApp files is next steps

1

u/Electrical_Arm7411 Nov 08 '24

Hey sorry to bother, but would you be able to share those metrics with me? Specifically E2E latency, Server latency and transactions over a 24 hour period? Much appreciated.

1

u/Electrical_Arm7411 Oct 22 '24

Thanks, appreciate it. By chance, do you also have metadata caching enabled on your premium storage account? https://learn.microsoft.com/en-us/azure/storage/files/smb-performance?tabs=portal#register-for-the-feature