An Independent Remote End-User Experience Performance Benchmark
RDS Gurus decided to run performance tests on Outlook in non-persistent environments using FSLogix’s Office 365 Containers solution. We wanted to illustrate the user experience of remote desktop users working in an RDS environment configured to use FSLogix Office 365 Containers relative to the typical user experience when using native RDS UPD technology. Our primary focus was to measure how performance degrades when multiple users are simultaneously working with Outlook (“noisy neighbor effect”).
We did this by collecting performance metrics as well as video footage of a series of test runs where users in remote sessions “worked” in Outlook 2016. All the work was automated. Users opened Outlook and conducted a series of searches that queried the local windows search service (queries against All Mailboxes, All Outlook Items or Subfolders) and queried the Office 365 server search service (queries against current folder and mailbox).
We recorded the on-screen user experience of a primary user, and also collected performance counter data during each test run. In some runs only the primary user taxed the system. In other runs, we introduced a number of secondary users who also “worked” in Outlook. We conducted our tests in an on-premises environment, and a duplicate Azure environment to see if there were any notable differences in a cloud based environment.
Benchmarking Test Steps
These are the steps we performed to set up the environments, conduct the test runs and gather the results:
- Build both the on premises and the Azure environments (includes identical RDS deployments in Azure and on premises locations). Our lab setup is shown here.
- Create and compile the test sequences that will run the workloads – these are the executables that simulate user’s working in Outlook. Click here for more details about our compiled executables and how they work
- Conduct the test runs
- Collect metrics, performance counters and videos
- Build REX Analyzer 4-Up Video Panels that show the primary user experience along with certain performance monitor counters in real time
- Analyze the data
Benchmarking Setup
Our benchmarking setup had four parts: The On-Premise RDS deployment, the Azure RDS deployment, the Tracking Component and the Client Component (see Figure 1). The Tracking Environment included a Lab Controller PC connected to the primary client (so it can capture video data from a client PC’s RDS sessions and store those videos for later analysis), and an RD Session Host server to kick off secondary user sessions to the on-prem/azure RDS environments.
Figure 1 – The On-Premise RDS deployment, the Azure RDS deployment, the Tracking Component and the Client Component.
REX Analysis and Our Outlook Search Benchmarking Conclusions
In June 2017, we conducted 20 Outlook Search test runs. For each test run we gathered and analyzed the screen videos from the primary client and the Performance Monitor footage.
We compared runs that used User Profile Disks to like runs that used FSLogix container technology. In each comparison, the number of secondary users “working” in the background was the same (0, 5, 10 and 20 users). UPD secondary users were not indexed – this simulated the users having to re-index if they move to a different RD Session Host server. FSLogix secondary users started off indexed because their index roams with them (no need to rebuild if they end up on a new RD Session Host server).
We looked for differences in how Outlook search performs in the primary user session when using the two technologies. We watched for any noticeable search result slowdown for the primary user when the secondary user’s OST files were being indexed in the background. We looked for differences in resource consumption per technology when the user load was low, or increased.
We used our own REX Analyzer software to create 4 quadrant video comparisons of each test run match-up. In the top two quadrants, we ran videos of the two primary user sessions we were comparing. In the bottom two quadrants we ran performance counter data from those two sessions.
These are our conclusions from watching all Rex Analyzer 4 video panel match ups.
1. User Experience Is Similar If Outlook Is Indexed
While there are some differences in the resource consumption between UPD and FSLogix technologies (this is more evident through performance counter logs and graphing), the user experience for users using UPD and using FSLogix is very similar if the user’s Outlook is indexed.
Here is a REX Analyzer file comparing test runs using UPD and FSLogix with 5 noisy neighbors. When both primary users are indexed, and the server is under a relatively small user load, user experience is good. UPD users and FSLogix users both get search results returned in good time.
If resources are scarce due to more secondary load placed on the system the primary user’s experience degrades; In the next REX Analyzer panel, we show UPD and FSLogix test runs with 20 noisy neighbors. You will see the stop watch timer stutter and search results are still returned but with slight lag – but this is true for both UPD and FSLogix users. And even though you can see the CPU in the FSLogix test run is pegged more than in the UPD test, the user experience is still very similar.
2. User Experience with FSLogix Wins When Users Roam
This is not surprising as this is a main selling point for FSLogix, but it still rings very true and is worth stating: When users roam from RD session host to another RD Session host, their search experience is far better with FSLogix. This is because it roams the search index on per user basis inside the user’s profile container; there is no need to re-index a user’s Outlook OST no matter which RD Session Host server they end up on.
During our test runs we could observe time and time again that secondary FSLogix users that had Outlook indexed had a far better user experience in than secondary UPD users who had to have their OSTs indexed. The FSLogix secondary users got consistent returned results for searches that used the Windows Local Search service while the UPD secondary users did not. To show this we videoed the Noisy Neighbor sessions in two on premises test runs. Note in the following video where search results are not returned for searches that use the local windows search service (these are non-indexed UPD users):
Now you see next FSLogix users, with Outlook OST files indexed get consistent search results returned in Outlook:
3. The Leveler: Windows Search Service Throttling
Throughout the test runs, resource usage on the RD Session Host servers was similar, even when secondary UPD user sessions were busy indexing Outlook OSTs in the background. The mechanism that keeps the UPD and FSLogix sessions seeming so comparable is the Windows Search Service being throttled back. When the index service needs to index a user’s Outlook OST, this happens in the background and Windows Search service is not given priority on the machine. If many users are all being indexed at the same time, then we see this happen very slowly.
Windows Search Throttling Details
The indexer runs off a basic backoff system where it will do work for X ms then sleep for Y ms, where Y>>X. It doesn’t matter how much work you give it, the CPU usage won’t spike. Indexing just takes longer.
So, it makes sense that if, for example, 10 users all need indexing and the primary user does not, then the primary user will not feel the effects of the other users’ indexing needs. Therefore, it does not seem to matter performance wise if a user is fully indexed and uses either UPD or FSLogix.
There is a GPO that affects this throttling:
Computer Configuration / Policies / Administrative Templates / Windows Components / Search / Disable Indexer Backoff
If we enable this GPO, we would see more difference in resource consumption for the primary UPD user because the secondary users on that server would be indexing and would be allowed more resources to do so.
The GPO makes the basic backoff system sleep for Y ms, where Y = 0ms, so it never sleeps. The indexer will very happily use all the resources on your machine at normal priority, starving out anything else that is trying to run.
Important!: The GP point was added for people running file servers where they really only care about having search available all the time and will never run anything else on it. Don’t enable this in production unless you really like pain!
4. Windows Search Service Is Fragile – FSLogix Helps
Relying on Windows Search service in a multi-user environment is fragile. During several test runs, we had to quit the run and redo it because certain buttons (local search oriented buttons) were greyed out on the Outlook Search ribbon. Restarting the Windows Search service fixed this issue, but what the specific problem was with the Windows Search service remains a mystery. And because the issue happened more than once it does not instill great confidence in the Windows Search service.
It’s also worth mentioning that if anything goes wrong with the index on an RD Session Host server, and it decides it needs to rebuild, search experience for all UPD users on that machine is bad. We had this happen a few times during testing; “something bad” would happen to the windows server search index database and it would rebuild (which made us have to stop, wait and re-conduct the test runs).
The users will get little or no results returned for Outlook searches that use the Local Windows Search service (searches for All Outlook Items and Searches in Subfolders) while the index is being rebuilt. With FSLogix, this is not the case – the effects of one user’s search index does not affect the other users, as each user has their own search database file.
Note: The search service is not supported on multi-user installations, so it being fragile is true, but its at least partly due to using it in an unsupported manner.
Next: RDS User Experience Using OneDrive for Business and FSLogix Office 365 Containers
Based on the Microsoft knowledge base article (https://support.microsoft.com/enus/kb/2965687), the OneDrive for Business sync agent is not supported on a Terminal Services based implementation. In order to gain access to OneDrive for Business files, users will have to utilize a web browser. FSLogix Office 365 Containers however, does support roaming OneDrive for Business cache. Stay tuned for our next project where we will run performance tests on OneDrive for Business in non-persistent environments using FSLogix’s Office 365 Containers solution.
