Kirk Liemohn is a Principal Software Engineer at ThreeWill. He has over 20 years of software development experience with most of that time spent in software consulting.
The ThreeWill Transformation Practice enables clients to move to the Microsoft Cloud by transforming their existing on-premises business applications, processes, and content to a suitable remote computing architecture. Kirk Liemohn is the Transformation Practice Principal Consultant and has many years of experience enabling client’s move to the cloud, using custom-developed tools and scripts, 3rd party tools, and a combination of both. The following is Kirk’s deep dive into Microsoft’s SharePoint Import Migration API.
After years of transforming and migrating applications, I figured it was about time that I dug in a little deeper on the Migration API that Microsoft provides for tool vendors and system integrators. I already knew many of the benefits and the high-level details. However, I had never actually written code to use the API to understand it at that level.
What is a Migration API and why does it matter?
The SharePoint Import Migration API is an API for migrating certain types of content to SharePoint Online or OneDrive for Business. The primary reason for using the API would be to increase the speed of the content migration. Another reason is to avoid the destination tenant from throttling the migration.
The primary reason for using the API would be to increase the speed of the content migration and avoid the destination tenant from throttling the migration.
Traditional content migration approaches (CSOM, REST, PnP, or even the Microsoft Graph) are not intended for mass changes, and, over the years, Microsoft has determined that these traditional approaches could cause problems by overloading the destination tenant. Microsoft’s response to this was to implement throttling to safeguard the tenant, at the expense of causing failures in these approaches. As discussed in the above link, you can use techniques with these traditional approaches to avoid or manage throttling. The Migration API should help even more!
What can the API migrate?
Unfortunately, the Migration API cannot migrate all types of content. Some traditional content migration approaches will likely still be needed to fill the Migration API gaps. This means throttling is still a risk. This also explains why 3rd party migration tools can get throttled even if they are configured to use the Migration API.
When discussing the scope of what the API handles, we focus on the target environment, not the source. The Migration API does not care what your source is; it can be SharePoint on-premises, a file system, or anything. You will have to package up the source content in a very specific way as discussed further below.
Supported Migration API capabilities include:
- List items
- System fields (created, created by, modified, modified by)
- Managed metadata
- Version history
- List item ID
To be clear, our deep dive only tested the first 3 from the list above. We also only tested calendar/event list items.
Unsupported Migration API capabilities include, but not limited to:
- Classic discussion boards
- Structure (sites, subsites, list, and libraries all must be pre-provisioned)
- Navigation elements
- List items declared as records
Although we did not dig far enough to find out if list/library folders can be created with the Migration API, I suspect that it can. Also of note, there is a post on Migrating web parts using the Migration API. We could not get that to work in our limited time and do not believe any tool vendors have implemented this as of Spring 2020; however, I think this could be quite valuable.
What is the process for using the Migration API?
The documentation (same link as further above) can get you started, but the basic overview is:
- Organize your assets that you want to migrate (files, list content, user mapping information)
- Create and populate your Azure storage blobs/queue:
- A Content blob must be created – this is where you put files you want to go to document libraries
- A Manifest blob must be created – this is where you put XML files defining a plethora of information including user mappings and list items as well as the location Microsoft puts log files when the migration completes
- A Report queue must be created – this is where the Microsoft puts status message as the migration API runs
- Kick off the migration. There is only one line in the actual API:
Guid jobId = ClientContext.Site.CreateMigrationJob(webId, containerBlobUrl, manifestBlobUrl, queueReportUrl);
- The URLs to the blobs/queue are generated such that they provide the appropriate read/list (container blog) and read/list/write (manifest blob, report queue) access for a specific period-of-time
- Monitor theAzure queue based on the Job Id provided by the API call
- Review the log files when complete
It seems simple, but the devil is in the details – and those details are in the 8 manifest files. Digging into each of these is beyond the scope of this post and is quite a chore, but you can get a 14,000 line (you read that right) C# file that pre-defines the classes used to generate these manifest files from here. Pretty quickly you start to realize that this is not trivial.
Interestingly, I believe these manifest files date back to the stsadm import/export commands which explain the use of ugly XML (vs. ugly JSON, I suppose).
If this has not scared you off and you want to dig further, here are some additional resources that might help:
- General Migration Performance Guidance
- Office 365 Migration API using C# (from 2016, but gets deeper into the code)
- Migration API Demo (from 2016, but this is an actual code sample)
- Demystifying the New Migration API for SharePoint (from 2015)
- What is the Office 365 migration API (from 2015)
- Content Matrix Support for the Migration API (v 8.8)
- ShareGate Migration Limitations with Insane Mode (“Insane Mode” refers to the Migration API)
Understanding the Migration API is very useful when migrating to SharePoint Online. For tool vendors, it is imperative that they make use of it. For Office 365 IT, systems integrators, and consultants, knowing the Migration API can give you insight into what migration tools are doing. Here at ThreeWill we can take what we know around the Migration API to better serve our clients by improving custom migration tooling we have.
A big thanks and shout out to Divya Pinnu, who did a lot of the groundwork for our migration API POC effort.