Our ThreeWill Client Success Practice focuses on maximizing the value returned to our clients based on their investments in technology initiatives. We have a comprehensive set of tools and techniques that can be leveraged for this purpose both from a “technology” and a “people” perspective. However, clients aren’t interested in tools, techniques or even our past successes. They want to see quantified/tangible benefits and outcomes from their current technology investments. From a technology/project perspective, quantifying outcomes/benefits is usually pretty straightforward, e.g. using project scope/time/cost metrics, solution reliability, defect rates, etc. However, from a people perspective, how do we know if benefits are being achieved? We measure them! This often involves reviewing user analytics information to gauge levels of user adoption and proficiency with new solutions.
Our ThreeWill Client Success Practice focuses on maximizing the value returned to our clients based on their investments in technology initiatives.
This is certainly true for Microsoft 365 rollout projects. Regardless of whether they relate to totally new deployments, corporate acquisitions/divestitures, or other scenarios, user adoption is a key consideration in assessing value returned by the initiative. Microsoft provides a rich set of user analytics that not only provide insights for rollout but also for ongoing health and management of Microsoft 365 subscriptions and related workloads/applications. Of course, like most things in Microsoft 365, Microsoft provides multiple ways to accomplish similar things (in this case getting user analytics).
With this blog, I’m going to try to provide a birds-eye view of analytics information options available in Microsoft 365 so that you can make an informed decision on options that best fit your needs. By the way…tackling the breadth of Microsoft 365 user analytics-related services is a bit ambitious for a blog post. So, I’ve sprinkled a few Star Wars references in what follows to “lighten up” the coverage of a fairly broad set of capabilities. Be patient you must!
Microsoft 365 Admin Center Usage Reports
There is a progression in levels of sophistication with the different analytics tools available in Microsoft 365. The usage reports available in the Reports section of the Microsoft 365 Admin Center are a fairly full-featured set of analytics that are available to admins as well as non-admins with a “reports or global reader” role. These reports are accessed via the “Reports – Usage” menu selection in the main admin center menu.
For Star Wars fans, think of this as the “Padawan” level of adoption reports. No special configuration or special knowledge is required to consume these reports. The main dashboard covers 7-180 day reporting periods with a main focus on the number of “active users” for the various workloads centered at the top of the page. There are also “cards” further down the page that cover the main types of workloads with specific metrics for each particular workload.
All of the cards are clickable to provide more information and most of the detailed information can be exported to CSV files. These reports are provided as part of your Microsoft 365 subscription fee. For more information see Microsoft 365 Reports in the admin center. Since the number of “active users” is a centerpiece of these reports, it is helpful to understand exactly what that means (as it means something different for the different workloads, e.g., OneDrive, SharePoint, Teams, etc.). For more information see Active user in Microsoft 365 usage reports.
Microsoft 365 Usage Analytics – Microsoft’s Power BI App
If you look closer at the information available in the admin center usage reports, you’ll notice that they might fall short for some uses. For example, they can show information down to a specific individual level (although the reports can be anonymized through configuration settings) and are not related to Active Directory-type information like location, department, and organization. So, sharing of this information would likely not be suitable for non-admins. This information does not help people in the business that are trying to get insights at a higher level than individuals. The information lacks patterns that might reveal more information about groups of people that may have specific challenges in adoption of some workloads.
To address these types of needs, Microsoft provides a Power BI-based app called “Microsoft 365 Usage Analytics”. Sticking with the Star Wars theme, think of this as the “Jedi Knight” level of analytics/dashboards available in Microsoft 365 as these reports provide a more extensive set of metrics, visualizations, ability to extend and customize the reports, and distribute to various audiences.
Before using these reports, you have to “opt-in” to Microsoft 365 Usage Analytics by clicking a “Get Started” button on the “Microsoft 365 Usage Analytics” card on the usage reports page in the admin center. Opting in means that you agree to make Microsoft 365 usage analytics data available to Power BI. After opting in and allowing a bit of time for data population, the “Get Started” button will change to “Go to Power BI”. You can access the Microsoft 365 Usage Analytics Power BI app via this button or directly within the Power BI web app. In addition to the “opt-in” step in the admin center, you also have to enable the Power BI template app by installing the app within Power BI and then connecting it to your tenant by entering your tenant ID. See Enable Microsoft 365 usage analytics for more details.
The Microsoft 365 Usage Analytics app contains a report and a data set within Power BI. The report consists of 5 main sections (Executive Summary, Overview, Activation/Licensing, Product Usage, and User Activity) that can be navigated via selections on the left nav panel. In general, the report shows a cross-product view of information for the past 12 months. All values for “month” shown are for the latest complete month.
The main report sections include metrics and insights as indicated in the table below (from the Microsoft Web site Microsoft 365 usage analytics page).
|Executive Summary|| |
Contains six main sections:
Reports on Microsoft 365 activation (how many users have downloaded and activated Office apps) and how many licenses have been assigned by your organization.
|Product Usage|| |
Contains a separate report for each Microsoft 365 service, including Exchange, Microsoft 365 groups, OneDrive, SharePoint, Skype, Teams, and Yammer. Each report contains total enabled vs. total active user reports, counts of entities such as mailboxes, sites, groups, and accounts, as well as activity type reports where appropriate.
|User Activity|| |
Provides user-level detail usage data joined with Active Directory attributes. In addition, the Department Adoption report lets you slice by Active Directory attributes so that you can see active users across all individual services. All metrics are aggregated for the latest complete month
The Microsoft 365 Usage Analytics report is really just an example template of how to interact with usage data (data set). The report can be customized for further insights of interest. As you can see above, the report is fairly extensive. As a result, a typical approach I have seen in many videos and blogs is to actually trim the report back for a simpler report with more personalized insights. These customized reports can then be securely shared with other users via SharePoint sites, Teams or within Power BI itself. With this approach, specific fit-for-purpose reports can be provided to executives and other decision-makers without them needing to access the admin center or Power BI directly. You can also use the Power BI desktop to further customize your reports by connecting them to other data sources to gain additional insights about your business.
All in all, the above capabilities provide a very powerful option for producing/consuming user analytics…but this is not for free! A common misconception is that using Microsoft 365 Usage Analytics is a free proposition (probably because the Power BI app is freely distributed). However, for users to be able to consume/view the report, they all need Power BI Professional Licenses (currently $9.99/month/user).
Taking It to the Next Level – AI-based Insights
Microsoft has been making a series of analytics services available that leverage artificial intelligence approaches to the Microsoft Graph (essentially this is the unified data and programmability model for the tremendous amount of data in Microsoft 365).
You guessed it – this is truly the Jedi Master-level of user analytics. One of the services in this category is Microsoft Workplace Analytics.
Microsoft’s description of Workplace Analytics states “Workplace Analytics reveals insights about the way work gets done, empowering organizations to explain and predict the impact of day-to-day actions, then drive change.” It is provided as three main functions: 1) Employee Experience, 2) Organizational Agility, 3) Customer Focus. Another catchphrase from Microsoft states “Discover opportunities to optimize business processes and identify patterns that lead to success. Measure the impact on business outcomes throughout a continuous cycle of discovery and change.” For more on these statements and additional screen snaps/videos see Workplace Analytics on the Microsoft product site. You must purchase licenses for Workplace Analytics for each user covered by the analysis services. Note – This is not a “mom and pop” level service (not cheap). Microsoft’s web site states “Contact your Microsoft account team for pricing.”.
One of my favorite phrases from the Microsoft product site on Workplace Analytics is “digital exhaust.” Digital exhaust is described as “data that comes naturally off of our everyday work that we can then use to understand what is going on in an organization.” Love that.
Microsoft also makes a similar service available at an individual (person) level called “MyAnalytics”. The target here is not management level reporting but rather individual self-guided improvement. It also provides AI-based insights…but at a personal level. It mainly focuses on “how people spend their time” and “who they spend it with”. The figure below provides a small sample of the insights available with MyAnalytics. MyAnalytics is included with your Microsoft 365 subscription for the following subscription levels – Microsoft 365 E5, E3, and Business plans as well as Office 365 E5, E3, E1, Business Premium, and Business Essentials plans. Admins can turn this on at tenant level and individual users can opt-in or opt-out.
And finally…last November, Microsoft also announced a new service called “Productivity Score” and made it available as a “preview” only. Microsoft states that “Productivity Score helps organizations transform how work gets done with insights about how people use Microsoft 365 and the technology experiences that support them. The score reflects your organization’s performance against employee and technology experience measures and compares your score with organizations like yours”. They also state that the “score” includes metrics (how people are using Microsoft 365 to collaborate/communicate), insights (identifies opportunities to improve productivity and satisfaction), and recommended actions (recommendations on using Microsoft 365 products more efficiently). A lot of this seems to overlap a bit with the other AI-based services, although Productivity Score seems to be more focused on comparing your organization to benchmarks based on similar organizations.
Is That It?
If looking through all of the above analytics services has your head spinning, just consider the fact that I’ve not even included other application-specific analytics information sources. For a couple of examples, see Infographic: 7 Analytics Dashboards for SharePoint and Teams and Office 365 in Microsoft 365 and The Many Ways to Report Teams Usage Data. The second link details 5 different ways to report Teams usage analytics (only two of which I’ve mentioned above!).
So, what are my conclusions on the current state of play for user analytics? For now, “Microsoft 365 Usage Analytics” seems to be the most popular method for surfacing pure analytics information in a customizable/extensible way to business users outside of I/T. Surely, the AI-based services will eventually become popular as a way of “leading people to insights they would have otherwise missed”. However, that may be further in the future as these services mature.
For now, “Microsoft 365 Usage Analytics” seems to be the most popular method for surfacing pure analytics information in a customizable/extensible way to business users outside of I/T.
I hope the information in this blog has been helpful. Having multiple options for reporting on user analytics is actually a good thing that can be tailored as needed for each organization. As Tony Redmond points out in his article mentioned above (The Many Ways to Report Teams Usage Data), “just like personal fitness trackers, the important thing is to choose one method to track usage data and stick with it to achieve a consistent view of the world”.