NVIDIA GRID and Microsoft Windows Server OSs and Hyper-V

Just a quick blog to clear up some FAQs on Microsoft Hyper-V support and NVIDIA GRID.

The GRID 4.1 release support matrix details support for Windows Server OSs. Windows Server OSs also include the Hyper-V role and support for the Hyper-V hypervisor.

Baremetal Support for Windows Server OSs

Bare metal (when a single version of an OS is installed directly on a server) is a popular choice for many customers especially those using RDSH remoting solutions such as Citrix XenApp. Some GRID cards e.g. M60, M10 carry multiple GPUs and Microsoft OSs are limited in their support for multiple GPUs on bare metal. GRID options do exist:

  • M6 support for Windows Server OSs – as the M6 (blade) GRID card has a single GPU this card is fully supported for baremetal use with the documented versions of Microsoft Server
  • Cards with multiple GPUs can be used with Windows Server OS but due to the current limitations
    • M60 / M10 are not supported as boot/primary device on Windows. NVIDIA support it as a secondary device, where some other GPU must be available as the primary. On OS earlier than Server 2012 / Windows 8, the primary GPU must have a vendor WDDM driver, otherwise the NVIDIA driver will not load. Any software / apps being run must be able to handle using non-primary GPUs; not all software does (AutoCAD < 2017, for example).
  • Older K1 / K2 cards are support as the primary device
    • In this scenario on a K1 card, one GPU will be the primary and the other GPUs will be available as secondaries to applications designed to leverage secondary GPUs e.g. RemoteFX is one such application.

Full details are given in the GRID support matrix: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

DDA Support and Hyper-V

Deploying a hypervisor allows multiple copies (or even multiple OS varieties) to be installed on a server, each within its own VM. As such deploying a hypervisor allows GRID cards with multiple GPUs to be utilized whereby each VM can access a one of the GPUs available. NVIDIA and Hyper-V currently support only one physical GPU passed through per VM. So on a M60 card with 2 GPUs, two copies of Windows Server should be run to fully utilize the card with DDA.

DDA for Hyper-V is the equivalent of GPU pass-through/vDGA available for Citrix/VMware. Hyper-V support was introduced to Tesla passthough drivers in the R375 driver branch. DDA can be used to associate a VM running Windows Server OS with a GPU on a multi-GPU board.

With DDA (and similar PCIe passthrough technologies) the VM leverages the GPU vendors driver to get access to the native GPU drivers and capabilities such as support for the latest versions of OpenGL, DirectX, OpenCL, CUDA etc. The latest level of support for such technologies for the GRID supported Tesla GPU products such as M60, M10, M6 etc can be found in the vGPU user guide, see http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf.

Support for the Hyper-V role is included in the Windows Server support detailed in: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

Direct Device Assignment (DDA) was new Windows Server 2016 Hyper-V feature enabling PCI-Express devices to be passed directly to a VM, specifically GPUs and NVMe storage devices. With DDA passing a GPU to a VM it enables the native GPU driver to be used within the VM and all capabilities of that driver such as DirectX 12 and CUDA (at the time of writing RemoteFX supports Direct11 but not 12). With DDA you cannot share the GPU between VMs, it is assigned directly to a specific VM; it is directly equivalent to GPU pass-through and vDGA on Citrix XenServer and VMware ESXi/vSphere. Additionally the guest must be Windows 10, Windows Server 2012 R2, Windows Server 2016 or a supported Linux distribution.

NVIDIA GRID Drivers for Hyper-V DDA

These can be obtained for the M6, M60 and M10 GRID cards via the customer login portal, see: http://nvidia.custhelp.com/app/answers/detail/a_id/4120/


NVIDIA GRID licensing for Hyper-V DDA

This is the same as licensing for pass-through/vDGA on Citrix XenServer/VMware ESXi respectively (see: http://images.nvidia.com/content/pdf/grid/guides/GRID-Packaging-and-Licensing-Guide.pdf). For passthrough VDI VMs on Citrix XenDesktop a vWs license is required, whilst when used with XenApp/RDSH solutions a per user vWS or vApp license is required depending upon whether Quadro features are needed (see my blog on the equivalent licensing for XenApp upon passthrough, see https://virtuallyvisual.wordpress.com/2016/09/05/nvidia-grid-rdsh-licensing-including-xenapp/

Microsoft RemoteFX Support

The RemoteFX protocol is able to leverage GRID GPUs and can be used as an alternative to Citrix/VMware VDI access to VMs. Microsoft RemoteFX is currently licensed as a vPC feature vPC license per CCU, details can be found in NVIDIA GRID PACKAGING AND LICENSING GUIDE, when used without DDA.

Customers are also able to leverage RemoteFX in conjunction with Citrix VDI technologies although the Citrix HDX/ICA protocols are often more appropriat ein a Citrix environments. Customers looking to do this should consult Citrix Support for advice, see https://support.citrix.com/article/CTX129509.

RemoteFX’s use of GPUs is confusingly often called vGPU, this is completely different to NVIDIA GRID vGPU (which pertains to sharing a GPU between multiple VMs). The system requirements, DirectX support and limitations are detailed by Microsoft, see: http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

RemoteFX has been available for a long while on many legacy versions of Windows. Improvements in releases during 2016 included support for DirectX11, OpenGL 4.4, OpenCL 1.1, 4k resolution, and Windows Server virtual machines (see https://technet.microsoft.com/en-us/windows-server-docs/compute/remote-desktop-services/rds-whats-new)

RemoteFX vs DDA

There are a lot of user blogs and experience that might help you choose if RemoteFX or DDA is the most appropriate way to use a GPU for your licensing and application needs. E.g.:

As with all hypervisors pass-through/PCIe technologies such as DDA have limited hypervisor support for some features:

  • Monitoring from the hypervisor of the GPU is not possible, only in-guest
  • Features like High-availability, live snapshots, VM migration are not possible

Hardware Compatibility List

Microsoft publish a list of GPUs supported for use with Windows Server OS, customers should verify that the version of Windows Server OS they are using supports the use of the specific NVIDIA GPU they are using in the Windows Server Catalogue.

Customers should verify that their Server is certified for use with NVIDIA GRID to ensure support. The list of GRID supported servers can be found on the NVIDIA GRID web pages, here: http://www.nvidia.com/object/grid-certified-servers.html

Citrix and Hyper-V

Citrix XenApp and XenDesktop both provide versions that support the use of Hyper-V as an alternative to other hypervisor options supported (such as VMware ESXi, Citrix XenServer, Nutanix Acropolis AHV) and Citrix XenApp supports the use of Windows Server OSs in bare metal scenarios. See the Citrix Support Matrix for details: https://www.citrix.co.uk/products/xenapp-xendesktop/feature-matrix.html.

Microsoft Azure N-series

These VM’s from Azure leverage the Hyper-V DDA functionality. The demand has been significant and even in “available” regions there has been over-demand so availability may be limited. I’ve previously blogged about options to leverage this and some details for those wishing to including Citrix XenApp, fra.me and Teradici Cloud Access Software.


When using Azure N-series, customers should source the supported drivers via Azure and the process as documented by Microsoft for Azure.


Within minutes I started getting questions on twitter… so as it comes in I’ll add my best effort answers… ideally though post any questions in the comments below… twitter is not the easiest way to respond!

Q: So does support up to 64 sessions (M10)) on a XenApp VM with ?

A: There isn’t a session limit with XenApp GPU-sharing (this also applies to all pass-through technologies doing GPU-sharing). With 4 GPUs on an M10 you would use 4 Server OS VMs using DDA. The number of sessions allowed on each GPU is not limited by the technology but the raw GPU capacity and performance demands of the applications. Many XenApp users may find better densities using an M60 card, if you have lots of CAD/video/media users but for a boost to windows aero and 2D-apps I suspect the M10 could reach the densities you want. GPU-sharing densities were covered when this technology was first launched for XenApp on XenServer with pass-through (see https://www.citrix.com/blogs/2013/02/22/how-many-users-can-share-a-gpu/).

NVIDIA GRID: Citrix HDX adds support for Relative and Absolute Mouse Modes to Linux Receiver

Update: 14th September 2016 – Receiver For Windows 4.5 released today now adds support for Windows in addition to Linux!

Just a quick blog to highlight the availability on the Citrix HDX/ICA protocol of a feature enabling Relative Mouse mode. This is a particularly interesting for many NVIDIA GRID vGPU and graphical users as it enables better behavior of certain gaming like applications, particularly those favoured in federal simulations (battle and flight-simulators) e.g Bohemia Simulations VBS 2 & 3. Before on certain application without using an addition gamepad device the mouse could behave strangely ending up with the user pointing their barrel at the ground or sky. Continue reading NVIDIA GRID: Citrix HDX adds support for Relative and Absolute Mouse Modes to Linux Receiver

Optimising TCP for Citrix HDX/ICA including Netscaler

Marius Sandbu – NGCA (NVIDIA GRID Community Advisor)  aka Clever Viking!

The TCP implementation within Citrix HDX/ICA protocol used by XenDesktop and XenApp and also Citrix Netscaler is pretty Vanilla to the original TCP/IP standards and definition and the out-of-the-box configuration usually does a good job on LAN. However, for WAN scenarios particularly with higher latencies and certain kinds of data (file transfers), Citrix deployments can benefit greatly from some tuning.


One of our new NGCAs (NVIDIA GRID Community Advisors) Marius Sandbu has written a must-read blog on how to optimize TCP with a Citrix Netscaler in the equation: http://msandbu.org/tag/netscaler-tcp-profile/Marius highlights some of the configuration optimisations hidden away in the Netscaler documentation and you’ll probably want to refer to that  documentation too (https://docs.citrix.com/en-us/netscaler/11-1/system/TCP_Congestion_Control_and_Optimization_General.html).

Citrix HDX TCP is not optimized for many WAN scenarios but at the moment it can also be tuned manually following this advice: CTX125027 – How to Optimize HDX Bandwidth Over High Latency Connections. This is one configuration I’d love to see Citrix automate as having to tune and configure the receiver is fiddly and also not possible in organisations/scenarios where the end-points and server/network infrastructure might be provided by different teams or even companies (e.g. IaaS).


For Citrix NVIDIA GRID vGPU customers with looking at high network latency scenarios – it really is worth investigating the potential and benefits of TCP window tuning. I’d be really interested to hear feedback if you have tried this and what your experience / thoughts are too!


Norwegian, Marius Sandbu was recently awarded NGCA status by NVIDIA for his work with our community through his Netscaler, remoting protocols and experience with technologies such as UDP and TCP/IP. You can follow him on twitter @msandbu and of course do follow his excellent blog on http://msandbu.org/ !!!

NVIDIA GRID – A Guide on GPU Metric Integration for Citrix XenServer

Just a quick blog aimed at those looking to develop GPU hypervisor monitoring products by integrating the NVIDIA GPU metrics exposed by XenServer via their APIs. Really it’s a bit of a guide as to where to find the information provided by Citrix.



Two NVIDIA GPU technologies are available on Citrix XenServer:

  • GPU (PCIe) pass-through (including GPU-sharing for XenApp and VDI passthrough)
  • vGPU (shared GPU technologies)

Owing to the nature of PCIe passthrough whereby the hypervisor is bypassed and the VM itself obtains complete control and sole access to the GPU, host and hypervisor level metrics are not available to the NVIDIA SDK and APIs on host nor to the hypervisor.

Developing a supported solution

Many Citrix customers insist on a monitoring solution being certified by the vendor via the Citrix Ready program. ISVs are advised to join the Citrix Ready program (access level is free) to obtain advise on developing a supported product and to eventually certify and market their product. In particular ISVs are recommended to evaluate the conditions of the vendor self-certification “kit” for supported products.

Whilst monitoring can be performed by inserting a kernel module or supplemental pack into XenServer’s dom0 this is an unsupported mechanism that Citrix generally will not support and customers are rarely willing to compromise their support agreements to use such products. ISVs are strongly advised to consider using the XenServer APIs and SDK to access metrics in a supported manner. See: https://www.citrix.com/partner-programs/citrix-ready/test.html (under XenServer-> Citrix XenServer (6.x) Integrated ISV Self-Certification Kit).

XenServer SDK / API

The XenServer API provides bindings for five languages: C, C#, Java, Python and Powershell.

XenServer maintains a landing page for ISV developers: http://xenserver.org/partners/developing-products-for-xenserver.html

Additionally there is developer (SDK) support forum where many XenServer staff answer questions: http://discussions.citrix.com/forum/1276-xenserver-sdk/

XenServer Metrics

XenServer captures metrics in RRDs. Details of the RRDs, code examples and information on how the XenServer SDK can be used to access the metrics are given on this landing page: http://xenserver.org/partners/developing-products-for-xenserver/18-sdk-development/96-xs-dev-rrds.html

XenServer have integrated many of the metrics available from NVIDIAs NVML interface into their RRDs. This means customers can access the metrics via the XenServer APIs in a supported manner rather than inserting unsupported kernel modules to call NVML in the hypervisor’s host operating system (dom0).  See https://www.citrix.com/blogs/2014/01/22/xenserverxendesktop-vgpu-new-metrics-available-to-monitor-nvidia-grid-gpus/

XenServer APIs – querying GPU infrastructure:

For information on which VMs have vGPUs, the type of vGPU profile etc. see http://nvidia.custhelp.com/app/answers/detail/a_id/4117/kw/citrix%20monitoring under “Checking your GPU configuration” for links to appropriate XenServer documentation.


Useful links:


NVIDIA GRID: Linux Guest OS support for Linux distributions on Citrix and VMware

I was recently involved in a support inquiry where a user wanted to know if NVIDIA GRID vGPU was available on Linux VDAs with the Linux guest OS, OpenSUSE LEAP (the answer at the time of writing is that it’s NOT!). Finding the answer was a lot harder than I expected as both VMware and Citrix documentation took a bit of hunting around.

Much of the marketing around Linux VDA’s mentions support for “SUSE”, “CentOS” or other genres of Linux, such as this blog. It is important that customers check both their hypervisor and VDI solutions official support matrix as both Citrix and VMware only certify, QA and support specific versions of Linux Guest OSs (usually only enterprise supported versions). Customers may find themselves unsupported by the virtualization vendors if they fail to check that the OS and specific version is supported by both their hypervisor and VDI solution (especially if mixing vendors such as Citrix XenDesktop on VMware ESXi).

Both vendors are evolving their Linux support rapidly and customers must check the documentation associated with the relevant versions of VMware/Citrix products they intend to use.

NVIDIA cannot provide support for guest OSs unsupported by the relevant virtualization vendor and as such customers are recommended to contact VMware/Citrix if they wish to use alternative versions/distributions. It is very likely many other varieties of Linux will “work” but customers should be aware that they will be unable to obtain hypervisor or VDI support in the event of an issue.

At the time of writing Horizon 7 on ESXi supports:

  • Ubuntu 12.04 and 14.04
  • Red Hat Enterprise Linux (RHEL) 6.6 and 7.1
  • CentOS 6.6
  • NeoKylin 6 Update 1 (Chinese)
  • SUSE Linux Enterprise Desktop 11 SP3


At the time of writing Citrix XenDesktop 7.9 on XenServer supports:

  • SUSE Linux Enterprise:
    • Desktop 11 Service Pack 4
    • Desktop 12 Service Pack 1
    • Server 11 Service Pack 4
    • Server 12 Service Pack 1
  • Red Hat Enterprise Linux
    • Workstation 6.7
    • Workstation 7.2
    • Server 6.7
    • Server 7.2
  • CentOS Linux
    • CentOS 6.7
    • CentOS 7.2

Ongoing if you want to check the OSs available for a Linux VDA you should follow the advice below.


XenServer Support for Linux Guest OSs

This is documented in the “Citrix XenServer® Virtual Machine User’s Guide” for the relevant version of XenServer e.g. for 7.0, here: http://docs.citrix.com/content/dam/docs/en-us/xenserver/xenserver-7-0/downloads/xenserver-7-0-vm-users-guide.pdf

XenDesktop Guest OSs Supported by the Linux VDA

This can be found in the Linux VDA product documentation for the relevant version of XenDesktop under the section “System Requirements” e.g. for XenDesktop 7.9 Please see http://docs.citrix.com/en-us/xenapp-and-xendesktop/7-9/install-configure/suse-linux-vda.html (This is where I had to hunt around as bizarrely Citrix detail the genres and versions of Linux supported under each supported OS rather than in a master list, so the SUSE documentation is where you can find RHEL and other supported versions listed)


ESXi/vSphere Support for Linux Guest OSs

Supported Linux OSs are listed in the “VMware Compatibility Guide”: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=software

Horizon Support for Linux Guest OSs

The versions and distributions supported by Horizon are listed in the FAQ for the appropriate release e.g. for Horizon 7, here: http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/horizon/vmware-horizon-for-linux-faq.pdf

More Lenovo Servers Support NVIDIA GPUs Including the M60

Lenovo have recently qualified and announced support for more NVIDIA GPUs for several servers including the x3650 M5 (E5-2600 v4), details can be found on Lenovo’s site, here:

Also recently listed is the x3500 M5:

This means Lenovo have worked with NVIDIA to test and certify that both parties hardware, firmware and software is fully-compatible, thermally and electrically stable.

Lenovo and vGPU/GPU-passthrough

Lenovo’s “redbook” site with server specifications and support also carries a wealth of information about Lenovo’s investment and joint development to support GPU technologies and virtualization including NVIDIA GRID vGPU. In particular their reference architecture designs including considerations for GPU usage are excellent and available for both VMware and Citrix infrastructures. You can read them here:

I’ve found the best place to start a search on Lenovo’s site is here: https://lenovopress.com/redpxref-system-x-reference and here:


Hypervisor Support

The GRID M60 card is now supported on more bare-metal/physical servers. Customers looking to use the M60 card with GRID vGPU in conjunction with a hypervisor such as Citrix XenServer or VMware ESXi should verify that the server OEM has also certified with the hypervisor by checking the VMware/Citrix HCL (Hardware compatibility list), details of how to do this can be found in these NVIDIA Support articles:

New Cisco Validated Design featuring UCS B200 M4 with NVIDIA GRID M6 vGPU – available now!

It’s great to see a new validated design released by Cisco in recent weeks. Particularly as this features the NVIDIA GRID M6 options for blade servers to enable virtualized GPU-accelerations (vGPU). This reference architecture joins other available for UCS but in particular features a reference blueprint for Citrix XenDesktop/XenApp 7.7 and VMware vSphere 6.0 for 5000 Seats. Key features include

  • Citrix XenDesktop/XenApp 7.7.
  • Built on Cisco UCS (including Cisco B200 M4 Blade Server) and Cisco Nexus 9000 Series
  • with NetApp AFF 8080EX
  • VMware vSphere ESXi 6.0 Update 1 Hypervisor Platform

Cisco have done a great job providing a comprehensive guide and reference for a full VDI/XenApp deployment that includes networking, storage and graphics acceleration considerations.


Cisco-NVIDIA Relationship

There are plenty of case studies, whitepapers and webinar recording covering Cisco long-investment in NVIDIA GRID and vGPU too: