NVIDIA GRID and Microsoft Windows Server OSs and Hyper-V

Just a quick blog to clear up some FAQs on Microsoft Hyper-V support and NVIDIA GRID.

The GRID 4.1 release support matrix details support for Windows Server OSs. Windows Server OSs also include the Hyper-V role and support for the Hyper-V hypervisor.

Baremetal Support for Windows Server OSs

Bare metal (when a single version of an OS is installed directly on a server) is a popular choice for many customers especially those using RDSH remoting solutions such as Citrix XenApp. Some GRID cards e.g. M60, M10 carry multiple GPUs and Microsoft OSs are limited in their support for multiple GPUs on bare metal. GRID options do exist:

  • M6 support for Windows Server OSs – as the M6 (blade) GRID card has a single GPU this card is fully supported for baremetal use with the documented versions of Microsoft Server
  • Cards with multiple GPUs can be used with Windows Server OS but due to the current limitations
    • M60 / M10 are not supported as boot/primary device on Windows. NVIDIA support it as a secondary device, where some other GPU must be available as the primary. On OS earlier than Server 2012 / Windows 8, the primary GPU must have a vendor WDDM driver, otherwise the NVIDIA driver will not load. Any software / apps being run must be able to handle using non-primary GPUs; not all software does (AutoCAD < 2017, for example).
  • Older K1 / K2 cards are support as the primary device
    • In this scenario on a K1 card, one GPU will be the primary and the other GPUs will be available as secondaries to applications designed to leverage secondary GPUs e.g. RemoteFX is one such application.

Full details are given in the GRID support matrix: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

DDA Support and Hyper-V

Deploying a hypervisor allows multiple copies (or even multiple OS varieties) to be installed on a server, each within its own VM. As such deploying a hypervisor allows GRID cards with multiple GPUs to be utilized whereby each VM can access a one of the GPUs available. NVIDIA and Hyper-V currently support only one physical GPU passed through per VM. So on a M60 card with 2 GPUs, two copies of Windows Server should be run to fully utilize the card with DDA.

DDA for Hyper-V is the equivalent of GPU pass-through/vDGA available for Citrix/VMware. Hyper-V support was introduced to Tesla passthough drivers in the R375 driver branch. DDA can be used to associate a VM running Windows Server OS with a GPU on a multi-GPU board.

With DDA (and similar PCIe passthrough technologies) the VM leverages the GPU vendors driver to get access to the native GPU drivers and capabilities such as support for the latest versions of OpenGL, DirectX, OpenCL, CUDA etc. The latest level of support for such technologies for the GRID supported Tesla GPU products such as M60, M10, M6 etc can be found in the vGPU user guide, see http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf.

Support for the Hyper-V role is included in the Windows Server support detailed in: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

Direct Device Assignment (DDA) was new Windows Server 2016 Hyper-V feature enabling PCI-Express devices to be passed directly to a VM, specifically GPUs and NVMe storage devices. With DDA passing a GPU to a VM it enables the native GPU driver to be used within the VM and all capabilities of that driver such as DirectX 12 and CUDA (at the time of writing RemoteFX supports Direct11 but not 12). With DDA you cannot share the GPU between VMs, it is assigned directly to a specific VM; it is directly equivalent to GPU pass-through and vDGA on Citrix XenServer and VMware ESXi/vSphere. Additionally the guest must be Windows 10, Windows Server 2012 R2, Windows Server 2016 or a supported Linux distribution.

NVIDIA GRID Drivers for Hyper-V DDA

These can be obtained for the M6, M60 and M10 GRID cards via the customer login portal, see: http://nvidia.custhelp.com/app/answers/detail/a_id/4120/

ddadriverlocation.png

NVIDIA GRID licensing for Hyper-V DDA

This is the same as licensing for pass-through/vDGA on Citrix XenServer/VMware ESXi respectively (see: http://images.nvidia.com/content/pdf/grid/guides/GRID-Packaging-and-Licensing-Guide.pdf). For passthrough VDI VMs on Citrix XenDesktop a vWs license is required, whilst when used with XenApp/RDSH solutions a per user vWS or vApp license is required depending upon whether Quadro features are needed (see my blog on the equivalent licensing for XenApp upon passthrough, see https://virtuallyvisual.wordpress.com/2016/09/05/nvidia-grid-rdsh-licensing-including-xenapp/

Microsoft RemoteFX Support

The RemoteFX protocol is able to leverage GRID GPUs and can be used as an alternative to Citrix/VMware VDI access to VMs. Microsoft RemoteFX is currently licensed as a vPC feature vPC license per CCU, details can be found in NVIDIA GRID PACKAGING AND LICENSING GUIDE, when used without DDA.

Customers are also able to leverage RemoteFX in conjunction with Citrix VDI technologies although the Citrix HDX/ICA protocols are often more appropriat ein a Citrix environments. Customers looking to do this should consult Citrix Support for advice, see https://support.citrix.com/article/CTX129509.

RemoteFX’s use of GPUs is confusingly often called vGPU, this is completely different to NVIDIA GRID vGPU (which pertains to sharing a GPU between multiple VMs). The system requirements, DirectX support and limitations are detailed by Microsoft, see: http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

RemoteFX has been available for a long while on many legacy versions of Windows. Improvements in releases during 2016 included support for DirectX11, OpenGL 4.4, OpenCL 1.1, 4k resolution, and Windows Server virtual machines (see https://technet.microsoft.com/en-us/windows-server-docs/compute/remote-desktop-services/rds-whats-new)

RemoteFX vs DDA

There are a lot of user blogs and experience that might help you choose if RemoteFX or DDA is the most appropriate way to use a GPU for your licensing and application needs. E.g.:

As with all hypervisors pass-through/PCIe technologies such as DDA have limited hypervisor support for some features:

  • Monitoring from the hypervisor of the GPU is not possible, only in-guest
  • Features like High-availability, live snapshots, VM migration are not possible

Hardware Compatibility List

Microsoft publish a list of GPUs supported for use with Windows Server OS, customers should verify that the version of Windows Server OS they are using supports the use of the specific NVIDIA GPU they are using in the Windows Server Catalogue.

Customers should verify that their Server is certified for use with NVIDIA GRID to ensure support. The list of GRID supported servers can be found on the NVIDIA GRID web pages, here: http://www.nvidia.com/object/grid-certified-servers.html

Citrix and Hyper-V

Citrix XenApp and XenDesktop both provide versions that support the use of Hyper-V as an alternative to other hypervisor options supported (such as VMware ESXi, Citrix XenServer, Nutanix Acropolis AHV) and Citrix XenApp supports the use of Windows Server OSs in bare metal scenarios. See the Citrix Support Matrix for details: https://www.citrix.co.uk/products/xenapp-xendesktop/feature-matrix.html.

Microsoft Azure N-series

These VM’s from Azure leverage the Hyper-V DDA functionality. The demand has been significant and even in “available” regions there has been over-demand so availability may be limited. I’ve previously blogged about options to leverage this and some details for those wishing to including Citrix XenApp, fra.me and Teradici Cloud Access Software.

https://virtuallyvisual.wordpress.com/2016/12/08/a-few-faqs-on-azure-n-series-inc-do-azure-n-series-vms-include-nvidia-grid-software-license/

When using Azure N-series, customers should source the supported drivers via Azure and the process as documented by Microsoft for Azure.

Feedback

Within minutes I started getting questions on twitter… so as it comes in I’ll add my best effort answers… ideally though post any questions in the comments below… twitter is not the easiest way to respond!

Q: So does support up to 64 sessions (M10)) on a XenApp VM with ?

A: There isn’t a session limit with XenApp GPU-sharing (this also applies to all pass-through technologies doing GPU-sharing). With 4 GPUs on an M10 you would use 4 Server OS VMs using DDA. The number of sessions allowed on each GPU is not limited by the technology but the raw GPU capacity and performance demands of the applications. Many XenApp users may find better densities using an M60 card, if you have lots of CAD/video/media users but for a boost to windows aero and 2D-apps I suspect the M10 could reach the densities you want. GPU-sharing densities were covered when this technology was first launched for XenApp on XenServer with pass-through (see https://www.citrix.com/blogs/2013/02/22/how-many-users-can-share-a-gpu/).

NVIDIA GRID: More info on vApps and VPC/vWS Licensing

lukeblog
Check out Luke Wignall’s blog on NVIDIA GRID licensing and other GRID topics!

I wrote a blog on RDSH (including XenApp) licensing and the options available with NVIDIA GRID vGPU and GPU-passthrough a few weeks ago, which you can read – here (including support for multi-monitor and resolutions). Since then my colleague Luke has added some more information in a blog where he outlines various case studies including many on vApps, which is worth a read here:

Luke answers how many licenses and what type you will need for various use cases, answering questions such as:

  • Q: I am deploying Citrix XenDesktop for 5000 global users, using two data centers, to meet a follow the sun productivity goal.  The data centers are also backup sites to each other.  I expect at most 1200 users at each of our three regional areas to be on during their workday, connecting to their closest data center, but there is some overlap (people working late or starting early) so I am architecting with a buffer for a total of 1500 virtual desktops.  I need to be able to run all users from either data center of one should go down. My users are all engineers and their apps require Quadro.
  • Q:  I am deploying virtual desktops but using XenApp to do so, and am looking for improved end user experience, for 1000 users.  At any given time I expect no more than 850 users to be connected.  I have no other desktop delivery method.
  • Q:  I chose to run XenApp on a bare metal host, so no hypervisor (I would question the decision to forgo the flexibility and manageability of virtualization), delivering three Microsoft Office applications so .  I have 500 users but expect no more than 350 of them to be connected at any given time.  I have no Virtual desktops for these users.
  • Q:  I have 250 engineers using CATIA and similar apps, they must have Quadro drivers, but usually only 200 of them are working at any given time.  I also have 1000 knowledge workers that range from sales to support, their apps do not need Quadro but perform much better with GPU (=happy users), of those I typically see 800 actively on their desktops.  I am deploying VMware Horizon.  We have a set of web apps that all 1250 employees use for time keeping, expenses, and safety training, these I am delivering with XenApp.

 

There is a lot of information on GRID licensing in our knowledge base – just search on “GRID licensing” on our KB home page here:

Highlights include:

Licensing Documentation:

Of course one of the best references is the official licensing guides on the GRID resources page (under deployment guides) here: http://www.nvidia.com/object/grid-enterprise-resources.html. In particular these two are useful:

 

Questions

Any questions – ask below or on the support NVIDIA GRID forums at https://gridforums.nvidia.com

NVIDIA GRID – RDSH licensing (including XenApp)

I’ve had a few questions about what licensing is needed under the GRID 2.0 and up software licensing for the M60/M10/M6 GPUs for RDSH solutions such as XenApp. I think the confusion arises because it’s possible to use a number of GPU/vGPU different profiles for a server OS VM. The key point is to remember that the licensing is always per user.

Continue reading NVIDIA GRID – RDSH licensing (including XenApp)

Optimising TCP for Citrix HDX/ICA including Netscaler

MArius
Marius Sandbu – NGCA (NVIDIA GRID Community Advisor)  aka Clever Viking!

The TCP implementation within Citrix HDX/ICA protocol used by XenDesktop and XenApp and also Citrix Netscaler is pretty Vanilla to the original TCP/IP standards and definition and the out-of-the-box configuration usually does a good job on LAN. However, for WAN scenarios particularly with higher latencies and certain kinds of data (file transfers), Citrix deployments can benefit greatly from some tuning.

 

One of our new NGCAs (NVIDIA GRID Community Advisors) Marius Sandbu has written a must-read blog on how to optimize TCP with a Citrix Netscaler in the equation: http://msandbu.org/tag/netscaler-tcp-profile/Marius highlights some of the configuration optimisations hidden away in the Netscaler documentation and you’ll probably want to refer to that  documentation too (https://docs.citrix.com/en-us/netscaler/11-1/system/TCP_Congestion_Control_and_Optimization_General.html).

Citrix HDX TCP is not optimized for many WAN scenarios but at the moment it can also be tuned manually following this advice: CTX125027 – How to Optimize HDX Bandwidth Over High Latency Connections. This is one configuration I’d love to see Citrix automate as having to tune and configure the receiver is fiddly and also not possible in organisations/scenarios where the end-points and server/network infrastructure might be provided by different teams or even companies (e.g. IaaS).

 

For Citrix NVIDIA GRID vGPU customers with looking at high network latency scenarios – it really is worth investigating the potential and benefits of TCP window tuning. I’d be really interested to hear feedback if you have tried this and what your experience / thoughts are too!

 

Norwegian, Marius Sandbu was recently awarded NGCA status by NVIDIA for his work with our community through his Netscaler, remoting protocols and experience with technologies such as UDP and TCP/IP. You can follow him on twitter @msandbu and of course do follow his excellent blog on http://msandbu.org/ !!!

NVIDIA GRID – A Guide on GPU Metric Integration for Citrix XenServer

Just a quick blog aimed at those looking to develop GPU hypervisor monitoring products by integrating the NVIDIA GPU metrics exposed by XenServer via their APIs. Really it’s a bit of a guide as to where to find the information provided by Citrix.

GPU-Graph

Background

Two NVIDIA GPU technologies are available on Citrix XenServer:

  • GPU (PCIe) pass-through (including GPU-sharing for XenApp and VDI passthrough)
  • vGPU (shared GPU technologies)

Owing to the nature of PCIe passthrough whereby the hypervisor is bypassed and the VM itself obtains complete control and sole access to the GPU, host and hypervisor level metrics are not available to the NVIDIA SDK and APIs on host nor to the hypervisor.

Developing a supported solution

Many Citrix customers insist on a monitoring solution being certified by the vendor via the Citrix Ready program. ISVs are advised to join the Citrix Ready program (access level is free) to obtain advise on developing a supported product and to eventually certify and market their product. In particular ISVs are recommended to evaluate the conditions of the vendor self-certification “kit” for supported products.

Whilst monitoring can be performed by inserting a kernel module or supplemental pack into XenServer’s dom0 this is an unsupported mechanism that Citrix generally will not support and customers are rarely willing to compromise their support agreements to use such products. ISVs are strongly advised to consider using the XenServer APIs and SDK to access metrics in a supported manner. See: https://www.citrix.com/partner-programs/citrix-ready/test.html (under XenServer-> Citrix XenServer (6.x) Integrated ISV Self-Certification Kit).

XenServer SDK / API

The XenServer API provides bindings for five languages: C, C#, Java, Python and Powershell.

XenServer maintains a landing page for ISV developers: http://xenserver.org/partners/developing-products-for-xenserver.html

Additionally there is developer (SDK) support forum where many XenServer staff answer questions: http://discussions.citrix.com/forum/1276-xenserver-sdk/

XenServer Metrics

XenServer captures metrics in RRDs. Details of the RRDs, code examples and information on how the XenServer SDK can be used to access the metrics are given on this landing page: http://xenserver.org/partners/developing-products-for-xenserver/18-sdk-development/96-xs-dev-rrds.html

XenServer have integrated many of the metrics available from NVIDIAs NVML interface into their RRDs. This means customers can access the metrics via the XenServer APIs in a supported manner rather than inserting unsupported kernel modules to call NVML in the hypervisor’s host operating system (dom0).  See https://www.citrix.com/blogs/2014/01/22/xenserverxendesktop-vgpu-new-metrics-available-to-monitor-nvidia-grid-gpus/

XenServer APIs – querying GPU infrastructure:

For information on which VMs have vGPUs, the type of vGPU profile etc. see http://nvidia.custhelp.com/app/answers/detail/a_id/4117/kw/citrix%20monitoring under “Checking your GPU configuration” for links to appropriate XenServer documentation.

 

Useful links:

 

More Lenovo Servers Support NVIDIA GPUs Including the M60

Lenovo have recently qualified and announced support for more NVIDIA GPUs for several servers including the x3650 M5 (E5-2600 v4), details can be found on Lenovo’s site, here:

Also recently listed is the x3500 M5:

This means Lenovo have worked with NVIDIA to test and certify that both parties hardware, firmware and software is fully-compatible, thermally and electrically stable.

Lenovo and vGPU/GPU-passthrough

Lenovo’s “redbook” site with server specifications and support also carries a wealth of information about Lenovo’s investment and joint development to support GPU technologies and virtualization including NVIDIA GRID vGPU. In particular their reference architecture designs including considerations for GPU usage are excellent and available for both VMware and Citrix infrastructures. You can read them here:

I’ve found the best place to start a search on Lenovo’s site is here: https://lenovopress.com/redpxref-system-x-reference and here:

 

Hypervisor Support

The GRID M60 card is now supported on more bare-metal/physical servers. Customers looking to use the M60 card with GRID vGPU in conjunction with a hypervisor such as Citrix XenServer or VMware ESXi should verify that the server OEM has also certified with the hypervisor by checking the VMware/Citrix HCL (Hardware compatibility list), details of how to do this can be found in these NVIDIA Support articles:

New Cisco Validated Design featuring UCS B200 M4 with NVIDIA GRID M6 vGPU – available now!

It’s great to see a new validated design released by Cisco in recent weeks. Particularly as this features the NVIDIA GRID M6 options for blade servers to enable virtualized GPU-accelerations (vGPU). This reference architecture joins other available for UCS but in particular features a reference blueprint for Citrix XenDesktop/XenApp 7.7 and VMware vSphere 6.0 for 5000 Seats. Key features include

  • Citrix XenDesktop/XenApp 7.7.
  • Built on Cisco UCS (including Cisco B200 M4 Blade Server) and Cisco Nexus 9000 Series
  • with NetApp AFF 8080EX
  • VMware vSphere ESXi 6.0 Update 1 Hypervisor Platform

Cisco have done a great job providing a comprehensive guide and reference for a full VDI/XenApp deployment that includes networking, storage and graphics acceleration considerations.

 

Cisco-NVIDIA Relationship

There are plenty of case studies, whitepapers and webinar recording covering Cisco long-investment in NVIDIA GRID and vGPU too: