Introduction

The more I learn about PowerShell, the more impressed I am with the power it has to do most anything.  Whether it is reading and writing from the Windows FileSystem, ActiveDirectory, SQL Server or SharePoint, PowerShell can do it all.  And because most SharePoint migration vendors provide PowerShell cmdlets to perform migration functions, automating migrations can be developed that leverages these cmdlets to initiate and monitor migration and migration-related activities in a predictable and orchestrated fashion.

Our Automating Migrations Project

At ThreeWill, we have been evolving an automation framework for automating migrations over time to increasingly add functionality. This so that we can provide more value to our customers when it comes to migrations.  In particular, I have been working with an automation framework focused on automating migrations for SharePoint.  This framework currently supports migrations for Metalogix Content Matrix.  We can easily interface with ShareGate and other vendors that support PowerShell cmdlets and APIs to monitor migration job progress.

SQL Server is usually a supported database that vendors use to log migration activity.  This allows the automation framework to initiate jobs through the vendor’s PowerShell cmdlets.  Then SQL Server table(s) are monitored for the progress of those jobs.

In addition, ThreeWill has also written some custom PowerShell scripts that can be leveraged to perform pre-migration and post-migration activities to check for important pre-requisites (I.e. running workflows or check-in files with no checked-in version) or to perform post-migration activities (I.e. validation, display “this site has been migrated” banners on source sites, etc).  The possibilities are endless.

In this article, I’d like to highlight some of the modules and features in PowerShell I think are important to migrations and many other automation-related processes.

Important Modules or Features in PowerShell

PnP PowerShell (module)

PnP PowerShell is a library of PowerShell commands that, to use the definition from Microsoft, “allows you to perform complex provisioning and artifact management actions towards SharePoint”.  For our process of automating migrations, we use PnP PowerShell to read and write to SharePoint Migration Inventory list so we can capture dates and times of each step in our migration workflow.  The SharePoint Migration Inventory list contains source and target urls , a Wave field where we can establish a set of sites to feed to our automation process for a given migration cycle, and a number of “Started”/”Completed” columns that are used to capture date/times for each step of the migration process.

These “Started/Completed” columns typically include Full Migration Started, Full Migration Completed, Validation Started, Validation Completed, Incremental Migration Started and Incremental Migration Completed.  These columns are updated with a date/time as the automation process navigates the migration workflow.  We also have a Migration Status column that denotes the current state of the migration (I.e. Not Started, Full Migration Started, Full Migration Completed, Validation Started, Validation Completed, etc).  This column is updated in conjunction with the “Started/Completed” columns to reflect the furthest completed step in the migration workflow.

Example code to retrieve SharePoint websites to be migrated

Here’s an example of code to retrieve a list of SharePoint websites to be migrated based on the assigned Wave number:

$camlQuery = "<View><Query><Where><Eq><FieldRef Name='$($Constants.WaveColumnInternalName)'/><Value Type='Text'>$($waveNumber)</Value></Eq></Where></Query></View>"
$listItems = Get-PnPListItem -List $Constants.InventoryListTitle -Query $camlQuery

There are similar PnP commands to do other CRUD operations on a SharePoint list. They are helpful to track status during an automated process.  Updates to the Migration Status field and date/time updates to the Started/Completed columns help provide a good high-level view of progress.

Start-Process (cmdlet in Microsoft.PowerShell.Management module)

The Start-Process cmdlet allows the automation framework to start a migration job as a separate process. The automation framework can then monitor by reading the SQL Server table(s) where the migration job logs its progress.  The process object that is returned can be monitored to determine when the process has exited. This is so we know the launched PowerShell process has completed.

Example code of Start-Process cmdlet

Here’s an example of the Start-Process cmdlet being used to launch another Powershell instance with a filename stored in the $psFile variable.  Five parameter values ($p1 – $p5) are being passed to this Powershell script that is being started.

Write-Verbose "[$($startDate)] Starting Copy-MLAllSharePointSiteContent from

$($siteInfo.SourceSiteUrl) to $($siteInfo.TargetSiteUrl)."

$proc = Start-Process PowerShell.exe -PassThru:$true -Argument "-File ""$($psFile)""",

$p1, $p2, $p3, $p4, $p5

return $proc

In this example, the variable $psFile references another PowerShell script file that contains code to call the Content Matrix cmdlet, Copy-MLAllSharePointSiteContent, which triggers a migration job based on the various parameter settings. You can see from the code below that there are many options when triggering a migration job.

#Full migration
 if($migrationScope -eq "Full")
 {
           $output = $SourceCollection | Copy-MLAllSharePointSiteContent -Target $Target `
                    -CopySiteWebParts:$copySiteWebParts `
                    -CopyPermissionLevels -CopySitePermissions -CopyAccessRequestSettings -CopyAssociatedGroups `
                    -WebTemplateMappingTable ( New-MetalogixSerializableObject "Metalogix.DataStructures.Generic.CommonSerializableTable``2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]" "Metalogix.Core" $webTemplateMappingTable) `
                    -CopyLists -OverwriteSites -RecursivelyCopySubsites:$copyChildSites -CopyContentTypes -CopyNavigation -CopySiteFeatures -MergeSiteFeatures `
                    -ApplyThemeToWeb:$applyThemeToWeb -PreserveMasterPage:$preserveMasterPage `
                    -CopyCustomContent -CopyUncustomizedFiles -RunNavigationStructureCopy -CopyGlobalNavigation -CopyCurrentNavigation -CopyListPermissions -OverwriteLists -CheckModifiedDatesForLists -CopyNintexForms:$copyNintexForms `
                    -CopyCustomizedFormPages:$copyCustomizedFormPages `
                    -CopyListOOBWorkflowAssociations -CopyContentTypeOOBWorkflowAssociations -CopyListSharePointDesignerNintexWorkflowAssociations -CopyContentTypeSharePointDesignerNintexWorkflowAssociations `
                    -CopyViewWebParts:$copyViewWebParts -CopyFormWebParts:$copyFormWebParts `
                    -CopyFolderPermissions -CopyListItems -UseAzureOffice365Upload -EncryptAzureMigrationJobs -CopyDocumentWebParts `
                    -ExistingWebPartsAction "Preserve" -CopyRootPermissions -MapRolesByName -ClearRoleAssignments -CopyItemPermissions `
                    -OverwriteItems -CheckModifiedDatesForItemsDocuments -ReattachPageLayouts -CopyVersions -CopySubFolders -MaximumVersionCount "5" `
                    -PreserveItemIDs -MigrationMode "Full" `
                    -LogSkippedItems -ForceRefresh -CorrectingLinks -LinkCorrectTextFields -LinkCorrectionScope "SiteCollection" `
                    -UseComprehensiveLinkCorrection:$useComprehensiveLinkCorrection `
                    -MapGroupsByName -OverwriteGroups -MapMissingUsersToLoginName $mapMissingUsersToLoginName -OverrideCheckouts `
                    -Transformers ( New-MetalogixSerializableObject "Metalogix.Transformers.TransformerCollection" "Metalogix.Actions" $transformers -Enumerate) `
                    -jobdatabase $jobDatabase `
                    -jobtitle $jobTitle -Certificate $certificateName -RunRemotely:$runRemotely

Microsoft.PowerShell.Management module

This management module contains cmdlets that help you manage Windows in PowerShell.  Our framework for automating migrations can run on multiple servers at one time. Changes to the framework needed to be easily distributed to all of the participating migration servers.  Commands like New-PSDrive, Copy-Item, Get-ChildItem, Get-Content, Set-Content were invaluable to delete the old code and then replace it.  Next, the code was replaced on the migration server. After this, the files were edited through PowerShell to make server-specific changes to the code.

For example, to help mitigate throttling issues when doing migrations, each server would use a different Office 365 account. Part of our code deployment would then be to update the account information in one of the script files. This is done to assign the appropriate Office 365 account.  In conclusion, the cmdlets in this module allowed us to do 3 things. Delete old code on a remote server, copy the new code to the remote server, and then edit files on the remote server to update as necessary.

Example of code to make a backup of a file and then open it and edit the content using a regular expression

#make a copy of the migrationscheduler file on this machine
$backupFileName = $backupFolder + "\migrationscheduler.ps1"
Copy-Item -Path $migrationSchedulerPath -Destination $backupFileName -Force
#get contents of migrationscheduler
$fileContents = Get-Content -Path $migrationSchedulerPath -Force
#replace ContentSince in migrationscheduler
$fileContents -replace 'contentSince = "[^"]*"', "contentSince = $contentSince" | 
Set-Content  -Path $migrationSchedulerPath

Custom modules

The automation framework contains many lines of code so the ability to create custom modules in PowerShell is very helpful.  Our automation code is separated into quite a few modules to help organize the cmdlets. They are organized based on their responsibility in the overall migration process.  Modules are files that end in a .psm1 file extension rather than .ps1.  They are simple to create and recommended to help organize your code.   Here’s a link to get you started:  https://docs.microsoft.com/en-us/powershell/scripting/developer/module/how-to-write-a-powershell-script-module?view=powershell-7

Get-Content (cmdlet in Microsoft.PowerShell.Management module)

In our automation scripts, we are usually reading and writing from a SharePoint list.  However, there are many other applications that require reading from a .csv file.  I love this command as it makes working with data exported from Excel to CSV so easy:

$webUrls = Get-Content –Path C:\CSVFileThatContainsWebURLs.csv | ConvertFrom-CSV

$webUrls variable now contains a collection of objects that I can loop through one by one using a for each loop.  Or I can use Where-Object {$_.Url -eq “http://www.threewill.com“} to retrieve all objects in the collection that have a URL property value of “http://www.threewill.com”.

Write-Host (cmdlet in Microsoft.PowerShell.Utility module)

In addition to logging to a database, sometimes you just want to write out helpful information to the host to help debug.  With Write-Host, you can customize the color of text by using the –ForeGroundColor and –BackGroundColor parameters.

Example to create a prompt for the user with color so it stands out

Write-Host "User should be in domain\id format (i.e. threewill\coalsont)" -BackgroundColor black -ForegroundColor red

Conclusion

I hope this brief discussion of some of the PowerShell features we have used in our automation has caused you to consider applications where automation can help you.  If you are relatively new to PowerShell and want to do more learning, I recommend a course in PluralSight by Adam Bertram titled “Windows PowerShell Toolmaking Fundamentals”.  Adam demonstrates some User and Computer management scenarios interacting with Active Directory. This can also open your eyes to other ways you can leverage PowerShell in your daily job.

Related Content: