NVIDIA GRID and Microsoft Windows Server OSs and Hyper-V

Just a quick blog to clear up some FAQs on Microsoft Hyper-V support and NVIDIA GRID.

The GRID 4.1 release support matrix details support for Windows Server OSs. Windows Server OSs also include the Hyper-V role and support for the Hyper-V hypervisor.

Baremetal Support for Windows Server OSs

Bare metal (when a single version of an OS is installed directly on a server) is a popular choice for many customers especially those using RDSH remoting solutions such as Citrix XenApp. Some GRID cards e.g. M60, M10 carry multiple GPUs and Microsoft OSs are limited in their support for multiple GPUs on bare metal. GRID options do exist:

  • M6 support for Windows Server OSs – as the M6 (blade) GRID card has a single GPU this card is fully supported for baremetal use with the documented versions of Microsoft Server
  • Cards with multiple GPUs can be used with Windows Server OS but due to the current limitations
    • M60 / M10 are not supported as boot/primary device on Windows. NVIDIA support it as a secondary device, where some other GPU must be available as the primary. On OS earlier than Server 2012 / Windows 8, the primary GPU must have a vendor WDDM driver, otherwise the NVIDIA driver will not load. Any software / apps being run must be able to handle using non-primary GPUs; not all software does (AutoCAD < 2017, for example).
  • Older K1 / K2 cards are support as the primary device
    • In this scenario on a K1 card, one GPU will be the primary and the other GPUs will be available as secondaries to applications designed to leverage secondary GPUs e.g. RemoteFX is one such application.

Full details are given in the GRID support matrix: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

DDA Support and Hyper-V

Deploying a hypervisor allows multiple copies (or even multiple OS varieties) to be installed on a server, each within its own VM. As such deploying a hypervisor allows GRID cards with multiple GPUs to be utilized whereby each VM can access a one of the GPUs available. NVIDIA and Hyper-V currently support only one physical GPU passed through per VM. So on a M60 card with 2 GPUs, two copies of Windows Server should be run to fully utilize the card with DDA.

DDA for Hyper-V is the equivalent of GPU pass-through/vDGA available for Citrix/VMware. Hyper-V support was introduced to Tesla passthough drivers in the R375 driver branch. DDA can be used to associate a VM running Windows Server OS with a GPU on a multi-GPU board.

With DDA (and similar PCIe passthrough technologies) the VM leverages the GPU vendors driver to get access to the native GPU drivers and capabilities such as support for the latest versions of OpenGL, DirectX, OpenCL, CUDA etc. The latest level of support for such technologies for the GRID supported Tesla GPU products such as M60, M10, M6 etc can be found in the vGPU user guide, see http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf.

Support for the Hyper-V role is included in the Windows Server support detailed in: https://griddownloads.nvidia.com/flex/GRID_4_Product_Support_Matrix.pdf

Direct Device Assignment (DDA) was new Windows Server 2016 Hyper-V feature enabling PCI-Express devices to be passed directly to a VM, specifically GPUs and NVMe storage devices. With DDA passing a GPU to a VM it enables the native GPU driver to be used within the VM and all capabilities of that driver such as DirectX 12 and CUDA (at the time of writing RemoteFX supports Direct11 but not 12). With DDA you cannot share the GPU between VMs, it is assigned directly to a specific VM; it is directly equivalent to GPU pass-through and vDGA on Citrix XenServer and VMware ESXi/vSphere. Additionally the guest must be Windows 10, Windows Server 2012 R2, Windows Server 2016 or a supported Linux distribution.

NVIDIA GRID Drivers for Hyper-V DDA

These can be obtained for the M6, M60 and M10 GRID cards via the customer login portal, see: http://nvidia.custhelp.com/app/answers/detail/a_id/4120/

ddadriverlocation.png

NVIDIA GRID licensing for Hyper-V DDA

This is the same as licensing for pass-through/vDGA on Citrix XenServer/VMware ESXi respectively (see: http://images.nvidia.com/content/pdf/grid/guides/GRID-Packaging-and-Licensing-Guide.pdf). For passthrough VDI VMs on Citrix XenDesktop a vWs license is required, whilst when used with XenApp/RDSH solutions a per user vWS or vApp license is required depending upon whether Quadro features are needed (see my blog on the equivalent licensing for XenApp upon passthrough, see https://virtuallyvisual.wordpress.com/2016/09/05/nvidia-grid-rdsh-licensing-including-xenapp/

Microsoft RemoteFX Support

The RemoteFX protocol is able to leverage GRID GPUs and can be used as an alternative to Citrix/VMware VDI access to VMs. Microsoft RemoteFX is currently licensed as a vPC feature vPC license per CCU, details can be found in NVIDIA GRID PACKAGING AND LICENSING GUIDE, when used without DDA.

Customers are also able to leverage RemoteFX in conjunction with Citrix VDI technologies although the Citrix HDX/ICA protocols are often more appropriat ein a Citrix environments. Customers looking to do this should consult Citrix Support for advice, see https://support.citrix.com/article/CTX129509.

RemoteFX’s use of GPUs is confusingly often called vGPU, this is completely different to NVIDIA GRID vGPU (which pertains to sharing a GPU between multiple VMs). The system requirements, DirectX support and limitations are detailed by Microsoft, see: http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

RemoteFX has been available for a long while on many legacy versions of Windows. Improvements in releases during 2016 included support for DirectX11, OpenGL 4.4, OpenCL 1.1, 4k resolution, and Windows Server virtual machines (see https://technet.microsoft.com/en-us/windows-server-docs/compute/remote-desktop-services/rds-whats-new)

RemoteFX vs DDA

There are a lot of user blogs and experience that might help you choose if RemoteFX or DDA is the most appropriate way to use a GPU for your licensing and application needs. E.g.:

As with all hypervisors pass-through/PCIe technologies such as DDA have limited hypervisor support for some features:

  • Monitoring from the hypervisor of the GPU is not possible, only in-guest
  • Features like High-availability, live snapshots, VM migration are not possible

Hardware Compatibility List

Microsoft publish a list of GPUs supported for use with Windows Server OS, customers should verify that the version of Windows Server OS they are using supports the use of the specific NVIDIA GPU they are using in the Windows Server Catalogue.

Customers should verify that their Server is certified for use with NVIDIA GRID to ensure support. The list of GRID supported servers can be found on the NVIDIA GRID web pages, here: http://www.nvidia.com/object/grid-certified-servers.html

Citrix and Hyper-V

Citrix XenApp and XenDesktop both provide versions that support the use of Hyper-V as an alternative to other hypervisor options supported (such as VMware ESXi, Citrix XenServer, Nutanix Acropolis AHV) and Citrix XenApp supports the use of Windows Server OSs in bare metal scenarios. See the Citrix Support Matrix for details: https://www.citrix.co.uk/products/xenapp-xendesktop/feature-matrix.html.

Microsoft Azure N-series

These VM’s from Azure leverage the Hyper-V DDA functionality. The demand has been significant and even in “available” regions there has been over-demand so availability may be limited. I’ve previously blogged about options to leverage this and some details for those wishing to including Citrix XenApp, fra.me and Teradici Cloud Access Software.

https://virtuallyvisual.wordpress.com/2016/12/08/a-few-faqs-on-azure-n-series-inc-do-azure-n-series-vms-include-nvidia-grid-software-license/

When using Azure N-series, customers should source the supported drivers via Azure and the process as documented by Microsoft for Azure.

Feedback

Within minutes I started getting questions on twitter… so as it comes in I’ll add my best effort answers… ideally though post any questions in the comments below… twitter is not the easiest way to respond!

Q: So does support up to 64 sessions (M10)) on a XenApp VM with ?

A: There isn’t a session limit with XenApp GPU-sharing (this also applies to all pass-through technologies doing GPU-sharing). With 4 GPUs on an M10 you would use 4 Server OS VMs using DDA. The number of sessions allowed on each GPU is not limited by the technology but the raw GPU capacity and performance demands of the applications. Many XenApp users may find better densities using an M60 card, if you have lots of CAD/video/media users but for a boost to windows aero and 2D-apps I suspect the M10 could reach the densities you want. GPU-sharing densities were covered when this technology was first launched for XenApp on XenServer with pass-through (see https://www.citrix.com/blogs/2013/02/22/how-many-users-can-share-a-gpu/).

VMWare ESXi announce High Availability (HA) for NVIDIA GRID vGPU VMs with vSphere 6.5

I was very pleased yesterday to see Pat Lee from VMware’s PM team tweet about this yesterday…

patleetweet

It’s something we knew VMware had added to vSphere 2016, vSphere 2016 supported in the GRID 4.1 (Nov 2016) release. As a VMware implemented feature this was something we at NVIDIA had to wait for them to announce. I think there have been a few problems with the documentation update staging which is why this has been a rather quiet feature release. I’ll update this blog with links to the documentation when it becomes available which should be soon!

But since Pat has let the cat out of the bag…. Probably best to answer a few basic questions straing away.

What is High Availability (HA)?

Basic HA is a feature to ensure VMs are up and running as soon as possible in the event of host failure. The VM will automatically restart as soon as possible on another host if one is available with sufficient resources. So for vGPU enabled VMs that means on a host with an appropriate GPU etc. Although the user will experience some down-time where possible this is minimized without the need for manual intervention by a system administrator.

Guaranteed High Availability…

This can be provided by HA features by allowing resources to be resourced such as RAM/CPU on hosts e.g. maybe 15% of a hosts capacity, which allows a guarantee that resource will be available to restart VMs upto a certain number of host failures. I believe that VMware’s configuration does not extend to configuring GPU resource reservation and so the support announced today will not offer guaranteed HA. It is a feature VMware could add in the future though if they saw sufficient demand, it is not a feature engineered by NVIDIA.

Can HA provide continual up-time?

No, not alone. Many hypervisors though offer Fault Tolerance (FT) which can provide such support, this is a very expensive feature to use as it relies on running essentially a duplicate VM on mirrored hardware which is phase-locked to the original (i.e. milliseconds behind), in the event of failure the user is switched to the duplicate with only a momentary glitch in user experience. It’s a feature essentially only used in a few safety / mission critical use cases as it’s so costly to implement.

So is Fault Tolerance (FT) supported for vGPU?

No not today, the technology to continually essentially snapshot a live GPU is not available. This is also a pre-requisite for live migration/motion e.g. vMotion and also regular snapshots.

The Future

NVIDIA and all the partners such as Citrix and VMware appreciate that live motion and snapshotting are key enterprise datacenter needs so we continue to work towards making such technology happen (it’s very technically hard I’m told!). We all know what you want and what you want our priorities to be!!!

NVIDIA GRID is architected with a software model which gives us the ability to add additional support for new OSs for customers existing hardware allowing them to pick up new features.

Autodesk Applications – Anywhere, Anytime Mobility on Any Device! Powered by NVIDIA GRID technologies.

greendrafters1-300x225NVIDIA GRID vGPU was launched in 2013 to enable applications to virtualize GPU resource and share GPUs to enable super-responsive graphics at a cost-effective price. Autodesk applications benefit greatly from GPU acceleration and over the last 3 years we’ve seen a huge number of customers deploy Autodesk applications such as Maya, Inventor, Moldflow, Revit and Autocad with our technologies.

This week NVIDIA is yet again at the amazing Autodesk University and I thought it would be nice to highlight some of the benefits of these technologies via a review of 4 of my personal favourite real customer implementations and stories and what they achieved virtualizing Autodesk applications.

 

  • Allow university students doing course work with Autodesk applications to do their coursework on their own laptops (Macs, Chromebooks, Windows laptops) or even their iPad from wherever they want without compromising security.

For many Autodesk users a choice of end-point is vital. Students want to remotely log in to use software for course work on their own Chromebooks, Apple Macs, tablets, laptops etc. BYOD is a huge driver for virtualizing Autodesk software.  Istanbul Aydin University (IAU) have recently deployed an NVIDIA GRID deployment to 1000 users using Citrix XenDesktop and VMware vCenter on Dell R720 servers. You can read more about the BYOD driving factors that drove the project here in this Dell Case Study.

Professional AEC users and designers also often have their own preferences for iMacs or even Linux workstations and love the ability to run Windows applications on their workstation or laptop of choice. Many companies already have lots of hardware so the flexibility to repurpose it as an end-point has cost benefits too!

 

  • Architects (AEC) able to buy cloud services rather than run and maintain hardware so can quickly adapt to changing projects.

dwp|suters are an architectural firm headquartered in Australia who operate in 15 countries.  With one  option being a very costly hardware upgrade  this company decided instead to implement a Citrix XenApp DaaS solution from a service provider on HP servers. It’s worth reading the Citrix Case Study as it covers how as well as savings it allowed dwp|suters to work in a completely different way freeing them from IT management and allowing them to scale their business, take on new projects rapidly and build a larger distributed team with enhanced security and data backup benefits too.

 

  • How Autodesk with NVIDIA GRID is simply the power behind amazing world renowned AEC and BIM projects including Olympic Stadiums!

Populous has created some of the most recognisable sporting venues, including Wembley Stadium in London, the new Yankee Stadium in New York, Soccer City in South Africa, and the Fisht Stadium in Sochi, which was the focal point of the 2014 Winter Olympics. The firm used VMware Horizon View and vSphere together with NVIDIA GRID K1 and K2 to virtualize AutoDesk Revit alongside other AEC applications. You can read more here: NVIDIA Case Study or even watch this VMware Video;

 

 

  • Enables designers, engineers and users to work remotely, flexibly and differently.

At Citrix Synergy in the spring of 2014 (NVIDIA GRID’s been around a while!), I saw one of the most fascinating demos of NVIDIA GPUs running Citrix XenApp to deliver Autodesk REVIT. XenApp (or similar RDSH solutions) can be a really cost effective and user-satisfying way to deliver Autodesk applications on VMware vSphere, Citrix XenServer or physical servers using NVIDIA GPUs.

This demo wasn’t so much about the underlying cost of the technologies though but end-user experience and utility. NVIDIA commissioned a real CAD drafter to design a house over three days at Synergy using our GRID GPU technology; this meant that you could go back time-after-time and watch how it had progressed and the various stages of design from 2D plans to 3D rendering and ray-tracing. It was a fascinating insight into the design process and I was delighted to find a blog by the designer himself on the GreenDrafters blog, here.

Designers are by their nature demanding and sensitive to the user-experience and I was delighted to see his comments:

  • “The short answer is that I was blown away.”
  • “The bottom line is that I did not notice any performance degradation between my personal workstation and the NVIDIA GRID K2 VDI”

I was also excited to see how the designer himself started imagining working differently, better, with virtualisation technologies:

  • “What I did notice was the simple thin client on my desk that took up far less space, needed far fewer cables, and caused far less clutter than my workstation.”
  • “I could use my Macbook Air or possibly even an Ipad to continue the design work away from my office, maybe at a Starbucks.”
  • “I also realized the intriguing potential of client/designer collaboration at the job site. Tweaking the design with the client in Revit, on-site. Now that is cool.”

 

 

Interested? Come and talk to NVIDIA at Autodesk University this week

Learn how designers can create from any connected device and location with NVIDIA GRID, which will be running Autodesk Revit 2016, 123D Design, AutoCAD, Inventor and other applications on VMware Horizon with vSphere. See our side-by-side comparison and discover how GPU-accelerated cloud services such as Autodesk’s Fusion 360 can offer 3D CAD, CAM and CAE tools on a single cloud-based platform with twice the performance of the CPU.

We’re also helping lead some informative AU classes:

 

 

Visit the BOXX, Dell, HP and Lenovo booths to see how our partners use NVIDIA graphics technologies to deliver the best performance and visual experience for Autodesk users. And be sure to follow us on Twitter on @NVIDIAGRID and @NVIDIA_MFG for the latest on our activities at AU or follow #AU2016.

 

There will also be plenty of other NVIDIA technologies and exciting demos at #AU2016 including Photorealistic rendering, Virtual Reality as well as Quadro and workstation products.

 

Can’t make it? There’s plenty of other ways to learn more:

NVIDIA GRID: More info on vApps and VPC/vWS Licensing

lukeblog
Check out Luke Wignall’s blog on NVIDIA GRID licensing and other GRID topics!

I wrote a blog on RDSH (including XenApp) licensing and the options available with NVIDIA GRID vGPU and GPU-passthrough a few weeks ago, which you can read – here (including support for multi-monitor and resolutions). Since then my colleague Luke has added some more information in a blog where he outlines various case studies including many on vApps, which is worth a read here:

Luke answers how many licenses and what type you will need for various use cases, answering questions such as:

  • Q: I am deploying Citrix XenDesktop for 5000 global users, using two data centers, to meet a follow the sun productivity goal.  The data centers are also backup sites to each other.  I expect at most 1200 users at each of our three regional areas to be on during their workday, connecting to their closest data center, but there is some overlap (people working late or starting early) so I am architecting with a buffer for a total of 1500 virtual desktops.  I need to be able to run all users from either data center of one should go down. My users are all engineers and their apps require Quadro.
  • Q:  I am deploying virtual desktops but using XenApp to do so, and am looking for improved end user experience, for 1000 users.  At any given time I expect no more than 850 users to be connected.  I have no other desktop delivery method.
  • Q:  I chose to run XenApp on a bare metal host, so no hypervisor (I would question the decision to forgo the flexibility and manageability of virtualization), delivering three Microsoft Office applications so .  I have 500 users but expect no more than 350 of them to be connected at any given time.  I have no Virtual desktops for these users.
  • Q:  I have 250 engineers using CATIA and similar apps, they must have Quadro drivers, but usually only 200 of them are working at any given time.  I also have 1000 knowledge workers that range from sales to support, their apps do not need Quadro but perform much better with GPU (=happy users), of those I typically see 800 actively on their desktops.  I am deploying VMware Horizon.  We have a set of web apps that all 1250 employees use for time keeping, expenses, and safety training, these I am delivering with XenApp.

 

There is a lot of information on GRID licensing in our knowledge base – just search on “GRID licensing” on our KB home page here:

Highlights include:

Licensing Documentation:

Of course one of the best references is the official licensing guides on the GRID resources page (under deployment guides) here: http://www.nvidia.com/object/grid-enterprise-resources.html. In particular these two are useful:

 

Questions

Any questions – ask below or on the support NVIDIA GRID forums at https://gridforums.nvidia.com

NVIDIA GRID – RDSH licensing (including XenApp)

I’ve had a few questions about what licensing is needed under the GRID 2.0 and up software licensing for the M60/M10/M6 GPUs for RDSH solutions such as XenApp. I think the confusion arises because it’s possible to use a number of GPU/vGPU different profiles for a server OS VM. The key point is to remember that the licensing is always per user.

Continue reading NVIDIA GRID – RDSH licensing (including XenApp)

Significant leaps in virtualized NVIDIA vGPU monitoring

managesdk
Read the documentation – the User Guide provided alongside the managmeent SDK is really comprehensive!

Today NVIDIA announced a new monitoring SDK / API incorporated into its GRID vGPU products as part of their GRID August 2016 (4.0) release. This will be available from Friday 26th August 2016 as a software release for existing hardware, greatly enhancing the functionality for existing as well as new customers. (You can read the announcement here).

NVIDIA has broken ranks with traditional hardware-only GPU models and recognized enterprises needs software to manage and monitor GPUs as a component of the data centre. Software licensing has enabled existing customers to benefit from new features with fully supported software, directly supported by NVIDIA (you wouldn’t run your Microsoft OS or CAD software unsupported!). Continue reading Significant leaps in virtualized NVIDIA vGPU monitoring

Optimising TCP for Citrix HDX/ICA including Netscaler

MArius
Marius Sandbu – NGCA (NVIDIA GRID Community Advisor)  aka Clever Viking!

The TCP implementation within Citrix HDX/ICA protocol used by XenDesktop and XenApp and also Citrix Netscaler is pretty Vanilla to the original TCP/IP standards and definition and the out-of-the-box configuration usually does a good job on LAN. However, for WAN scenarios particularly with higher latencies and certain kinds of data (file transfers), Citrix deployments can benefit greatly from some tuning.

 

One of our new NGCAs (NVIDIA GRID Community Advisors) Marius Sandbu has written a must-read blog on how to optimize TCP with a Citrix Netscaler in the equation: http://msandbu.org/tag/netscaler-tcp-profile/Marius highlights some of the configuration optimisations hidden away in the Netscaler documentation and you’ll probably want to refer to that  documentation too (https://docs.citrix.com/en-us/netscaler/11-1/system/TCP_Congestion_Control_and_Optimization_General.html).

Citrix HDX TCP is not optimized for many WAN scenarios but at the moment it can also be tuned manually following this advice: CTX125027 – How to Optimize HDX Bandwidth Over High Latency Connections. This is one configuration I’d love to see Citrix automate as having to tune and configure the receiver is fiddly and also not possible in organisations/scenarios where the end-points and server/network infrastructure might be provided by different teams or even companies (e.g. IaaS).

 

For Citrix NVIDIA GRID vGPU customers with looking at high network latency scenarios – it really is worth investigating the potential and benefits of TCP window tuning. I’d be really interested to hear feedback if you have tried this and what your experience / thoughts are too!

 

Norwegian, Marius Sandbu was recently awarded NGCA status by NVIDIA for his work with our community through his Netscaler, remoting protocols and experience with technologies such as UDP and TCP/IP. You can follow him on twitter @msandbu and of course do follow his excellent blog on http://msandbu.org/ !!!