NVIDIA GRID and Microsoft Windows Server OSs and Hyper-V

Just a quick blog to clear up some FAQs on Microsoft Hyper-V support and NVIDIA GRID.

The GRID 4.1 release support matrix details support for Windows Server OSs. Windows Server OSs also include the Hyper-V role and support for the Hyper-V hypervisor.

Baremetal Support for Windows Server OSs

Bare metal (when a single version of an OS is installed directly on a server) is a popular choice for many customers especially those using RDSH remoting solutions such as Citrix XenApp. Some GRID cards e.g. M60, M10 carry multiple GPUs and Microsoft OSs are limited in their support for multiple GPUs on bare metal. GRID options do exist:

• M6 support for Windows Server OSs – as the M6 (blade) GRID card has a single GPU this card is fully supported for baremetal use with the documented versions of Microsoft Server
• Cards with multiple GPUs can be used with Windows Server OS but due to the current limitations
  • M60 / M10 are not supported as boot/primary device on Windows. NVIDIA support it as a secondary device, where some other GPU must be available as the primary. On OS earlier than Server 2012 / Windows 8, the primary GPU must have a vendor WDDM driver, otherwise the NVIDIA driver will not load. Any software / apps being run must be able to handle using non-primary GPUs; not all software does (AutoCAD < 2017, for example).
• Older K1 / K2 cards are support as the primary device
  • In this scenario on a K1 card, one GPU will be the primary and the other GPUs will be available as secondaries to applications designed to leverage secondary GPUs e.g. RemoteFX is one such application.

Full details are given in the GRID support matrix: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

DDA Support and Hyper-V

Deploying a hypervisor allows multiple copies (or even multiple OS varieties) to be installed on a server, each within its own VM. As such deploying a hypervisor allows GRID cards with multiple GPUs to be utilized whereby each VM can access a one of the GPUs available. NVIDIA and Hyper-V currently support only one physical GPU passed through per VM. So on a M60 card with 2 GPUs, two copies of Windows Server should be run to fully utilize the card with DDA.

DDA for Hyper-V is the equivalent of GPU pass-through/vDGA available for Citrix/VMware. Hyper-V support was introduced to Tesla passthough drivers in the R375 driver branch. DDA can be used to associate a VM running Windows Server OS with a GPU on a multi-GPU board.

With DDA (and similar PCIe passthrough technologies) the VM leverages the GPU vendors driver to get access to the native GPU drivers and capabilities such as support for the latest versions of OpenGL, DirectX, OpenCL, CUDA etc. The latest level of support for such technologies for the GRID supported Tesla GPU products such as M60, M10, M6 etc can be found in the vGPU user guide, see http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf.

Support for the Hyper-V role is included in the Windows Server support detailed in: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

Direct Device Assignment (DDA) was new Windows Server 2016 Hyper-V feature enabling PCI-Express devices to be passed directly to a VM, specifically GPUs and NVMe storage devices. With DDA passing a GPU to a VM it enables the native GPU driver to be used within the VM and all capabilities of that driver such as DirectX 12 and CUDA (at the time of writing RemoteFX supports Direct11 but not 12). With DDA you cannot share the GPU between VMs, it is assigned directly to a specific VM; it is directly equivalent to GPU pass-through and vDGA on Citrix XenServer and VMware ESXi/vSphere. Additionally the guest must be Windows 10, Windows Server 2012 R2, Windows Server 2016 or a supported Linux distribution.

NVIDIA GRID Drivers for Hyper-V DDA

These can be obtained for the M6, M60 and M10 GRID cards via the customer login portal, see: http://nvidia.custhelp.com/app/answers/detail/a_id/4120/

NVIDIA GRID licensing for Hyper-V DDA

This is the same as licensing for pass-through/vDGA on Citrix XenServer/VMware ESXi respectively (see: http://images.nvidia.com/content/pdf/grid/guides/GRID-Packaging-and-Licensing-Guide.pdf). For passthrough VDI VMs on Citrix XenDesktop a vWs license is required, whilst when used with XenApp/RDSH solutions a per user vWS or vApp license is required depending upon whether Quadro features are needed (see my blog on the equivalent licensing for XenApp upon passthrough, see https://virtuallyvisual.wordpress.com/2016/09/05/nvidia-grid-rdsh-licensing-including-xenapp/

Microsoft RemoteFX Support

The RemoteFX protocol is able to leverage GRID GPUs and can be used as an alternative to Citrix/VMware VDI access to VMs. Microsoft RemoteFX is currently licensed as a vPC feature vPC license per CCU, details can be found in NVIDIA GRID PACKAGING AND LICENSING GUIDE, when used without DDA.

Customers are also able to leverage RemoteFX in conjunction with Citrix VDI technologies although the Citrix HDX/ICA protocols are often more appropriat ein a Citrix environments. Customers looking to do this should consult Citrix Support for advice, see https://support.citrix.com/article/CTX129509.

RemoteFX’s use of GPUs is confusingly often called vGPU, this is completely different to NVIDIA GRID vGPU (which pertains to sharing a GPU between multiple VMs). The system requirements, DirectX support and limitations are detailed by Microsoft, see: http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

RemoteFX has been available for a long while on many legacy versions of Windows. Improvements in releases during 2016 included support for DirectX11, OpenGL 4.4, OpenCL 1.1, 4k resolution, and Windows Server virtual machines (see https://technet.microsoft.com/en-us/windows-server-docs/compute/remote-desktop-services/rds-whats-new)

RemoteFX vs DDA

There are a lot of user blogs and experience that might help you choose if RemoteFX or DDA is the most appropriate way to use a GPU for your licensing and application needs. E.g.:

• http://windowsitpro.com/windows-server-2016/remotefx-vs-dda-windows-server-2016
• https://www.starwindsoftware.com/blog/windows-2016-makes-a-100-in-box-high-performance-vdi-solution-a-realistic-option

As with all hypervisors pass-through/PCIe technologies such as DDA have limited hypervisor support for some features:

• Monitoring from the hypervisor of the GPU is not possible, only in-guest
• Features like High-availability, live snapshots, VM migration are not possible

Hardware Compatibility List

Microsoft publish a list of GPUs supported for use with Windows Server OS, customers should verify that the version of Windows Server OS they are using supports the use of the specific NVIDIA GPU they are using in the Windows Server Catalogue.

Customers should verify that their Server is certified for use with NVIDIA GRID to ensure support. The list of GRID supported servers can be found on the NVIDIA GRID web pages, here: http://www.nvidia.com/object/grid-certified-servers.html

Citrix and Hyper-V

Citrix XenApp and XenDesktop both provide versions that support the use of Hyper-V as an alternative to other hypervisor options supported (such as VMware ESXi, Citrix XenServer, Nutanix Acropolis AHV) and Citrix XenApp supports the use of Windows Server OSs in bare metal scenarios. See the Citrix Support Matrix for details: https://www.citrix.co.uk/products/xenapp-xendesktop/feature-matrix.html.

Microsoft Azure N-series

These VM’s from Azure leverage the Hyper-V DDA functionality. The demand has been significant and even in “available” regions there has been over-demand so availability may be limited. I’ve previously blogged about options to leverage this and some details for those wishing to including Citrix XenApp, fra.me and Teradici Cloud Access Software.

https://virtuallyvisual.wordpress.com/2016/12/08/a-few-faqs-on-azure-n-series-inc-do-azure-n-series-vms-include-nvidia-grid-software-license/

When using Azure N-series, customers should source the supported drivers via Azure and the process as documented by Microsoft for Azure.

Feedback

Within minutes I started getting questions on twitter… so as it comes in I’ll add my best effort answers… ideally though post any questions in the comments below… twitter is not the easiest way to respond!

Q: So does #HyperV2016 support up to 64 sessions (M10)) on a XenApp VM with #WindowsServ2016 ?

A: There isn’t a session limit with XenApp GPU-sharing (this also applies to all pass-through technologies doing GPU-sharing). With 4 GPUs on an M10 you would use 4 Server OS VMs using DDA. The number of sessions allowed on each GPU is not limited by the technology but the raw GPU capacity and performance demands of the applications. Many XenApp users may find better densities using an M60 card, if you have lots of CAD/video/media users but for a boost to windows aero and 2D-apps I suspect the M10 could reach the densities you want. GPU-sharing densities were covered when this technology was first launched for XenApp on XenServer with pass-through (see https://www.citrix.com/blogs/2013/02/22/how-many-users-can-share-a-gpu/).