About the Outlook Search Test Runs

In order to capture any effect of “noisy neighbors” (other users working on the RD Session Host server at the same time as the primary user) we conducted multiple test runs per scenario:

  • Baseline Test Run: We run the test sequences with only the primary client only logged in (no noisy neighbors)
  • Noisy Neighbor Test Runs: We performed the same test run again but this time we logged in a set number of secondary users and launched the automated secondary user workload in each secondary user session. We ran three test runs like this with
    1. 5 noisy neighbors
    2. 10 noisy neighbors
    3. 20 noisy neighbors

Each Outlook test run we ran was:

  • Executed on-premises or in Azure
  • Used UPD or FSLogix technology
  • Launched a certain number of secondary users (noisy neighbors)

Outlook Test Sequences Query the Local Windows Search and Office 365 Server Search

Our test runs include test sequences that searched against both the local windows search service and against Office 365 Server Search. (Searching “all outlook items” and “subfolders” works against the local windows search service while searching “current folder” and “current mailbox” defaults to using Office 365 server search service.

The Primary User Screen

The screen capture of the primary user session shows latency at the client and multi-media performance. A frame grabber records the client screen without impacting the overall performance of a test sequence. The screenshot below shows an example what the captured video looks like.

Figure 1 – For each test run the primary user desktop shows 5 applications.

Collecting Test Data

The screen capture of the primary user session shows latency at the client as well as multimedia performance. A frame grabber records the client screen without impacting the overall performance of a test sequence. We measure logon and reconnect times by recording videos of the primary user sessions, while 0, 5, 10 and 20 secondary users are logged in at the same time.

We use the individual raw videos as the input material to create pre-defined REX Analyzer configurations in a 4-up split screen arrangement. Four different video sequences play at the same time for visual comparison. Two screen recordings showing the same sequences in two different test runs in the two upper quadrants and associated Performance Monitor counter videos in the two lower quadrants.

Test Sequences

The test sequences are compiled executables that open Outlook, do predefined work, log metrics and performance counter data, and after a set time period, exit the application and stop logging. They are stored on RD Session Host servers and launched from the primary and secondary workforce.

Launching Secondary User Sessions

The secondary users are launched automatically from a controller server (control-01) using a PowerShell script. A corresponding Group Policy Object allows us to automatically launch Outlook and start a secondary load test sequence within each of those sessions. These secondary users run a continuous workload and while these workloads are running we initiate and capture the primary session. Upon completion of the test we launch a logoff script that properly logs of all secondary users.

Outlook Test Run Uber-Exes

All test sequences are combined into one long test run executable (uber-exe). In each test run, the uber-exe launches the test sequences in the primary user session and also in the secondary sessions. The way the test sequences run in the primary and secondary sessions differ in that:

  • Primary user session sequences run in a predefined order, each sequence running one time and then the test ends.
  • Secondary user session sequences are chosen at random. The test loops through the 16 sequence options at random for 20 minutes (or until we log those users out programmatically.)

The diagram below shows how the primary and secondary uber-exes both runs the same sequences.

Figure 2 – Primary and secondary workloads