Tim Coalson is a Senior Consultant in the Transformation Practice at ThreeWill. Tim has been developing solutions on the SharePoint platform for over 15 years and has been a developer/consultant for over 30 years. Tim has been involved in migrating SharePoint on-premises farms to the Microsoft Cloud, Power Apps, and Power Automate (aka Flow) which are part of the Microsoft no code/low code solutions.
When most people think about a SharePoint migration, they usually think about the movement of data from one environment to another. This is certainly an important part of the migration process, with the SharePoint Migration Accelerator from ThreeWill being the focus of this article.
However, it is also important to understand that managing users through this change is instrumental to the adoption of the new platform. New environments have new features and sometimes old features are deprecated in the new environment. Understanding these changes up-front will help prepare users for these changes and hopefully build some excitement about what is possible with the new features. And creating a new target environment with coordinated branding, consistent templates, and intuitive navigation is also very important to make this transition as smooth as possible.
A Comprehensive Approach
The approach to a successful migration can be viewed as legs of a three-legged stool (see image below). The failure of any one of the legs will result in an outcome that is not as stable as it could be. The three legs are as follows: the creation of a well-planned target environment, migration of the data from the source to the new target (the focus of this article), and the management of users through this change.
To that end, ThreeWill is dedicated to three practice areas that focus on these “legs” to ensure a successful migration. Client Success is the practice that maximizes the business value of the migration through user adoption and organizational change management techniques. Transformation is the practice that moves data from the source to the destination. And Innovation is the practice that helps define the target environment to enhance employee experiences, increase engagement end empower workers to work together better. This blog will focus primarily on the automation framework that has been developed to facilitate the efficient migration of the data from the source to the target.
Migrating: The Basics
Migration software products such as Content Matrix and ShareGate come with a graphical user interface that allows connections to the source and target environments and offers a number of different migration options ranging from the migration of a full site collection to a single list item in a site. The user interfaces are a valuable tool for creating these connections, starting a job to migrate the data, and monitoring the progress of the job. And if your migration effort is relatively small, this is probably all you need.
Scaling Up the Migration
While some migration efforts may be small (less than 100 sites), some enterprise migration efforts involve hundreds or thousands of sites to be migrated so doing this manually using only the graphical interface is probably not feasible. For these types of efforts, ThreeWill has created an automation framework to not only handle the migration of the data but to also handle some pre-migration and post-migration tasks that are normally part of the migration process. This can be referred to as a ThreeWill SharePoint Migration Accelerator. This is possible because most migration vendors support PowerShell cmdlets to perform migration and validation tasks.
In addition to the vendor-supplied cmdlets, ThreeWill has also created other PowerShell scripts to support some of the pre-migration tasks such as “checking-in files with no checked-in version” and post-migration tasks such as “comparing list counts between the source and target environments”. In essence, a migration framework has been built that can be extended to perform a series of tasks and report the progress/output of the tasks to a database and SharePoint list. The details of the migration jobs that are triggered through the automation framework are viewable in the Vendor interfaces. Other details of pre and post-migration activities can be viewed in a SharePoint list or SQL Server database. Management views of the migration progress are available in a SharePoint custom list so everyone on the team can see the current status.
Managing the Migration through the Inventory List
The foundation of the SharePoint Migration Accelerator is a SharePoint list that we normally call the “Inventory List”. This list contains all of the source URLs to be migrated as well as the target location to which each of these source URLs should be migrated. “Starting” and “Ending” columns are defined for each step of the migration workflow so we can track the progress of each step that is involved in the overall site migration. In its most simple form, this would include a “Full Migration Started” and “Full Migration Completed” column that the automation script would populate with a date/time as the Full Migration starts and completes.
In addition, a “Status” column is defined that provides the overall progress with a single value (i.e. Not Started, Full Migration Complete, Validation Complete, etc.) that the automation script also populates as it progresses through the workflow of migration activities. Some customer migrations require incremental or delta migrations to provide a window of time for users to do User Acceptance Testing (UAT) after the initial Full migration. But since business never stops, the source site will continue to get updated during the UAT period, so one or more Incremental or delta migrations are included in the plan to pick-up these changes. To track these additional migration activities, additional columns would be added to the inventory list to support tracking of these steps in the process. From a management perspective, this list becomes the one source of truth that everyone involved in the project can review to understand the overall status.
Example Inventory List
Migrating the Data
With the Inventory list in place, groups of sites are selected to be migrated using the ThreeWill SharePoint Migration Accelerator. Sites are normally grouped into what we call “waves”. And, yes, there is a Wave column created in the Inventory list so that everyone knows which sites are associated to a given Wave including the automation script which will use this value to pull in the right set of sites for migration. The SharePoint Migration Accelerator is flexible in that it can be installed and run concurrently on several machines. To date, automation has been run concurrently on as many as ten servers working on a single Wave of sites. And we have configured waves with as many as 500 sites.
Normally, this number is driven by the project plan and the amount of time provided for migration and validation of the data prior to turning over the site to the customer for UAT. In our experience, roughly fifteen percent of sites require some manual review after the full migration due to source and target counts not matching. Sometimes a subsequent list migration might be necessary after fixing some data or updating a list attribute to resolve an issue. So, time has to be planned for these activities when choosing the number of sites to go into a Wave.
Manual Migration versus Automated Migration
The choice to use the ThreeWill SharePoint Migration Accelerator versus running the jobs manually using the vendor interface is dependent upon a number of factors. One of the biggest is the number of sites to migrate. As the number of sites to migrate increases, the automation framework is more cost-effective as it can handle the migration of sites with fewer human resources. The project schedule can also impact the choice of using automation or not. If a larger number of sites need to be migrated in a shorter time-frame, automation may make the most sense. If the project schedule is more forgiving, there may be time to execute the jobs manually.
While there is no exact science to these numbers, the table below represents a good rule of thumb:
|Number of Sites||Suggested Migration Technique|
|1-30||Most Vendor Consoles will accommodate the migration of a small number of sites|
|31-100||A combination of using the Vendor Console along with some custom PowerShell scripts to handle pre-migration and post-migration steps|
|101+||The ThreeWill SharePoint Migration Accelerator begins to make more sense as you approach a larger number of sites since this can automate a set of steps that could include pre-migration, migration and post-migration scripts. The automation can easily scale up to more and more servers as necessary to handle a larger number of sites in less time.|
In terms of time savings, a high-level estimation of the time savings that result from leveraging the automation is roughly 5 minutes for each job that needs to be configured and 3 minutes for validation of each site/subsite that is included in these jobs. For example, to create the connections and configure a migration for a site collection, the time would be 5 minutes. If the site collection contained 10 sites, the validation time would be 30 minutes so the savings to use automation would be 35 minutes. If the sites are small, manual validation would be less than 30 minutes, so these are averages.
While these savings are small when you are talking about a smaller set of sites, they become very large during an enterprise migration where the migration includes 10,000 or more sites. While this may seem like a crazy number, a current effort involves roughly 20,000 sites and the automation framework is allowing us to double the migration capacity leveraging 2 Consultants which ultimately saves both time and money.
I’ve created this graph to try to demonstrate the savings of using automation based on the number of sites being migrated. For 1,000 sites, the savings is 67 hours which is the sum of time for configuration and time for validation. As you increase the number of sites, the savings become much greater. Depending upon any specific customizations and extra set-up required for the automation, there could be some offset of the savings due to the cost to update the automation but that would easily be recovered in a larger migration.
Choosing the Right Approach
In the end, the choice of which approach to use is a collaborative decision that is made with each customer to ensure their needs are met. The number of sites to migrate, size and complexity of the sites, number of resources available to help in the effort, and the timelines dictated by the Project Plan are all factors that will need to be considered to choose the right approach.
For the most successful outcome, ThreeWill recommends using an approach the includes all three legs of the three-legged stool; Client Success, Transformation, and Innovation. Innovation will ensure enhanced employee engagement through simplified workplace collaboration. Transformation will provide the migration expertise to move the data from the source environment to the new platform. And Client Success will be involved all along the way to be sure Users are educated, prepared, and excited to use the new environment.