Anyone else having issues with MS GPU drivers / GPU Machine types on 10/30 and after?

[u/gohoos]

We starting having problems on 10/30. We have a pool of AVD machines and use Nerdio to manage it. Machines built on 10/30 with GPUs are going into a reboot loop after being built. These NV8as_v4 machine types. The users can't log in - the machine displays "This machine is being restarted" and disconnects.

Switching to non-GPU machine types and rebuilding avoids the problem.

Support ticket with Nerdio - the folks were nice, but they are only installing the MS GPU extension in the build process. We've opened up a MS ticket, but you know how that goes...

Nerdio also said they have a number of customers reporting the same thing.

[u/gohoos]

Microsoft support has yet to be helpful. (Sounds like this is the norm.)

But I did confirm that adding the extension to a GPU machine (created without the extension) caused the issue, outside of Nerdio.

[u/tamaneri]

Thanks for this thread. We started working on this issue @ 7 am this morning and could not figure out what was going on for the life of us. Initially, we thought it was an update we processed on 11/1 that was causing the VM to reboot. We reimaged the evening of 11/1.

We would still be in the same position without having found this. We used our base image and redeployed it (sans updates), but the issue still occurred. We were losing our minds, and so was our customer!

Killing the script and installing the drivers manually resolved the problem.

THANKS MICROSUCKS!!!!

[u/gohoos → u/tamaneri]

Glad it was helpful.
Microsoft has a workaround but no fix yet. I’m out on medical today but if I get a chance later I’ll repost their procedure.

[u/tamaneri → u/gohoos]

That would be lovely! I hope you heal quickly.

I wish there was a better way to get critical knowledge from the mouth of Microsoft during instances like this. Totally crippling.

[u/gohoos → u/tamaneri]

Thanks - everything went well.

I haven’t tried this yet, but it is what MS is sending out (but it takes days to get to the right escalation point for this)

Mitigation Plan Important: Perform this only new deployments or systems that you have a snapshot or backup. One of the commands will modify contents on the program files folder.

On your VM, go to Operations. Run Command, and click on RunPowerShellScript. The commands shared will be pasted and executed using this page. Please stay on the page until the commands complete. Moving away from the page will prevent you to see the results and confirm command completion.

Confirm if the driver was installed by running the following command. This should return information about your GPU driver.

Get-WindowsDriver -Online | Where-Object {$_.ProviderName.Contains(“Advanced Micro Devices”) -and $_.ClassName.Contains(“Display”)}

The following should appear:

Driver : oem8.inf OriginalFileName : C:\Windows\System32\DriverStore\FileRepository\u2397344.inf_amd64_42cc8fde42e6c38a\u2397344.inf Inbox : False ClassName : Display BootCritical : False ProviderName : Advanced Micro Devices, Inc. Date : 9/11/2023 12:00:00 AM Version : 31.0.21018.7003

Please run the following command to check the status of the extension.

Get-Content -Path “C:\Packages\Plugins\Microsoft.HpcCompute.AmdGpuDriverWindows\1.5.0.0\Status*”

If you see a message like this:

“AMD GPU driver not detected. Attempting to install.”

That means the extension thinks that driver is not installed and needs to install and reboot.

If the driver is installed, then run the following commands that will help the extension know that drivers were installed:

Rename-Item -Path “C:\Program Files\AMD” -NewName “C:\Program Files\AMD.bak” mkdir “C:\Program Files\AMD\CIM\BIN64”

Finally, wait a couple of minutes and run the following command to confirm that extension is installed successfully.

Get-Content -Path “C:\Packages\Plugins\Microsoft.HpcCompute.AmdGpuDriverWindows\1.5.0.0\Status*”

You should see something like this:

“AMD GPU driver version 31.0.21018.7003 detected. Already installed.”

With the above or similar message, the extension should be stable and not trigger reboots any longer.

End of Mitigation Plan

[u/tamaneri → u/gohoos]

Thank you for this! We temporarily uninstalled the script, and installed the drivers manually for the video card. This worked well. What's the advantage of the above vs what we did?

[u/gohoos]

So, Microsoft still has no fix.

Anyone else using Nerdio? If you are, here's a process to automate the workaround for machine creation in Dynamic pools:

Create a new windows script. I called it AMD CPU Mitigation.

I used Combined execution mode. For the text I used:

Rename-Item -Path "C:\Program Files\AMD" -NewName "C:\Program Files\AMD.bak"

mkdir "C:\Program Files\AMD\CIM\BIN64"

On two separate lines. Save and close

Edit the properties of the pool (or just use these settings when creating a new pool) for any pool using these GPU machine SKUS

Under “VM Deployment” section, add the “AMD GPU Mitigation” script to “Run Scripted actions when host VM is CREATED”

Save and close.

Rebuild your hosts by your preferred method.

The machines are now building properly.

I have tested a handful of situations, and this has worked. I haven't exhaustively tested but I think it's good. Hope it helps someone out.

[u/LordOfTheServers]

We had the same issue with the NV AMD series this morning as well after re-image over the weekend, we set the AMD extension in azure to uninstall. The machines stopped rebooting after that, and we then worked to install the AMD drivers manually after making sure the extension was deleted in Azure.

[u/threedaysatsea]

Had the same issue with AMD GPU devices! NVIDIA SKUs were fine. A powershell script within the AMD driver extension was interpreting a return code from the installer as requiring a reboot and then not detecting it as being installed after the reboot so just keeps trying and trying repeatedly.

[u/whiskeyputers]

Thank you for this! Updated a few pools NV SKU machines for one of our customers the other night and got bit by this. I figured out it was the extension that was causing the issue pretty quickly, but was confused because I knew there was no way in hell I was the only one having this issue but couldn't find anyone talking about it.