Migrating large corporate SharePoint intranets
- Approach (part I of IV)
- Analysis (part II of IV)
- Migration (part III of IV)
- Verification (part IV of IV)
Migration
In this third part of the series I will discuss the actual migration itself. If all previous steps were done right, the migration should be the easy part of the complete project.
Tool selection
I have found that for migrations to work almost autonomous only three tools are needed:
- Excel, and to be more specific a comma separated file (.csv) that is being used as an input file for the migration
- PowerShell to script and automate the migration. Basically this consists of the following tasks:
- Read input from .csv
- [optional] do mutations / transformations on the input.csv (needed for a more specialised migration-scenario)
- Use ShareGate PowerShell cmdlets
- Process ShareGate logging
- ShareGate, the migration tool that does the actual migration
In addition of using PowerShell to automate the migration, we have seen that PowerShell is also a great to to analyse your environment. When combining these, the output .csv of the analysis phase can serve as the input for the migration phase.
Space allocation
Migrating to a new SharePoint environment (SharePoint online or SharePoint on-premise) will require storage and most likely lots it it. Make sure that the destination SharePoint farm is prepared to handle the amount of data. You can save some storage for on-premise migrations by turning down the SQL transaction logging. Also temporary disable SQL database mirroring can safe you from lots of unnecessary logging.
Migration actions
Your migration process will look something like the image displayed above. See this as your base-migration plan that can be altered depending on your situation. In general you do not want the users to be able to modify the source- site while the migration is in process. That means it is needed to remove these permissions before the migration starts. But since, most likely user permissions needs to be migrated too, the first step in this migration-plan is to export the user permissions.
Next step is the migration itself. The migration is done using PowerShell that processes a input.csv, uses ShareGate to migrate and handles the logging generated by ShareGate. One rule I found to be very helpful is to log everything to file but only show critical events on the screen. The code-samples below contain such filtering mechanisms to display only the critical messages.
Finally when the migration has completed it is time to check the log and do some basic validations. Once this is done, permissions can be imported so that the environment becomes accessible for the users. Next communicate the migration results to the site owner and inform them about anything noticeable information or issues.
Migration scenarios
To describe the details of the migration itself we have to distinguish different scenarios. Each will be described in detail below. As described before the migration is done using the ShareGate PowerShell cmdlets. The cmdlets to use depend on the type of migration that is needed.
I use the following table as a general guideline:
| Migration type | ShareGate cmdlet |
|---|---|
| site migration | Copy-Site |
| list migration | Copy-List |
| item migration | Copy-Content (not the list definition itself) |
First scenario: as-is migration
In this scenario no change is needed to the data and everything should be preserved as-is. One case I have used this is to ‘archive’ project that are no longer in use on an on-premise environment and to store them in SharePoint Online for archive. Of course this scenario can also be used to migrate to SharePoint Online for projects that are still in use.
Input
The input consists of a simple .csv file that contains all of the source site URLs. Creating this input.csv can either be done manually or by using the analyse-script described in the: Analysis (part II of IV) using recursion.
At the present time it has become a more common rule to use site collections instead of sites. When migrating from sites to site collections the Input csv can simply be extended to include a destination URL instead of using a single destination site URL as described below. Also notice that Sharegate can now also promote subsites to top-level sites as described in this article O365.tips
environment.ps1
These variables are used in the scripts below, it contains the name of the input file, a start index, end index and a destination site URL. Making use of a start and end index it is possible to migrate only a segment of the complete input file. Migrating in smaller batches gives the opportunity to do some in-between migration validations. Another reason could be that sometimes the migration cannot continue in the evening because of security patches e.g.
1$inputFile = ".\input.csv"
2$startIndex = 5
3$endIndex = 10
4$dstSiteUrl = "https://your.sharepoint.com/sites/20190101/targetsite"
0-show.ps1
Useful script to simply display the lines of the input file that will be migrated using the given $startIndex = 5 and $endIndex = 10.
1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
2
3. ".\environment.ps1"
4
5if ($null -eq $endIndex) {
6 Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
7 break
8}
9
10if ($null -eq $dstSiteUrl) {
11 Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12 break
13}
14
15$input = [Array](Import-Csv -Path $inputFile)
16
17$i = 1
18foreach ($line in $input) {
19 if ($i -lt $startIndex) {
20 $i++
21 continue
22 }
23
24 $url = $line.SiteUrl
25 Write-Host "$i-$endIndex [$url] -> [$dstSiteUrl]" -ForegroundColor DarkGray
26
27 if ($i++ -ge $endIndex) {
28 Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
29 break
30 }
31}
1-migrate.ps1
The actual migration script. Basically just a plain PowerShell script, but makes use of the cmdlets provided by ShareGate such as: Connect-Site, New-MappingSettings, New-CopySettings and the Copy-Site. One rule I use is to log everything, but only display error and warning messages to the screen output. This way you do not get flooded by an information overload.
1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
2
3. ".\environment.ps1"
4
5if ($null -eq $endIndex) {
6 Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
7 break
8}
9
10if ($null -eq $dstSiteUrl) {
11 Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12 break
13}
14
15$input = [Array](Import-Csv -Path $inputFile)
16$dstSite = Connect-Site -Url $dstSiteUrl -Browser
17
18$i = 1
19foreach ($line in $input) {
20 if ($i -lt $startIndex) {
21 $i++
22 continue
23 }
24
25 $url = $line.SiteUrl
26 $projectNr = $url.split("/")[-1]
27 Write-Host "$i-$endIndex [$url] -> [$dstSiteUrl/$projectNr]" -ForegroundColor DarkGray
28
29 $srcSite = Connect-Site -Url $url
30
31 $mappings = New-MappingSettings
32 Import-UserAndGroupMapping -MappingSettings $mappings -Path "$(Get-Location)\UserAndGroupMappings.sgum" | Out-Null
33 Import-SiteTemplateMapping -MappingSettings $mappings -Path "$(Get-Location)\SiteTemplateMappings.sgwtm" | Out-Null
34 Import-PermissionLevelMapping -MappingSettings $mappings -Path "$(Get-Location)\PermissionLevelMappings.sgrm" | Out-Null
35
36 Write-Host $(Get-Date -Format "dddd MM/dd/yyyy HH:mm") -ForegroundColor White
37
38 $copysettings = New-CopySettings -OnContentItemExists Skip -OnSiteObjectExists Skip
39
40 $result = Copy-Site -MappingSettings $mappings -CopySettings $copysettings -Site $srcSite -DestinationSite $dstSite -NoCustomPermissions -NoWorkflows -NoSiteFeatures #-NoContent #-NoCustomizedFormsAndViews -NoNavigation
41 Write-Host " Errors:" $result.Errors "Warnings:" $result.Warnings "Copied:" $result.ItemsCopied
42
43 Write-Host $(Get-Date -Format "dddd MM/dd/yyyy HH:mm") -ForegroundColor White
44
45 $statusMessage = "OK"
46 if ($result.Warnings -gt 0) {
47 $statusMessage = "Warning"
48 Write-Host "Warning during copy" -ForegroundColor Yellow
49 }
50 if ($result.Errors -gt 0) {
51 $statusMessage = "ERROR"
52 Write-Host "Error during copy" -ForegroundColor Red
53 }
54
55 $filename = "$($i)-$($statusMessage).csv"
56 Export-Report $result -Path "./log/$filename" -Overwrite | Out-Null
57
58 Write-Host "Showing logfile errors and warnings ./log/$filename" -ForegroundColor Green
59 $input = Import-Csv "./log/$filename"
60
61 $input | ForEach-Object {
62 if ($_.Status -eq "Error") {
63 Write-Host "Error [$($_.Title)]" -NoNewline -ForegroundColor Magenta
64 Write-Host $_.Details -ForegroundColor Red
65
66 }
67 if ($_.Status -eq "Warning") {
68 Write-Host "Warning [$($_.Title)]" -NoNewline -ForegroundColor Magenta
69 Write-Host $_.Details -ForegroundColor Yellow
70 }
71
72 }
73
74 Write-Host
75 if ($i++ -ge $endIndex) {
76 Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
77 break
78 }
79}
Download the complete solution here
Second scenario: list migrations into sites based on custom template
The second scenario I like to show is one where the destination site(s) are based on a custom developed / configured template. As described in the customizations part, there are situations in which it is better to develop a custom tailored template to fully suite the business case. One method of developing such templates is by making use of the OfficeDev PnP.
So when the site itself should not be copied but the content does you can choose to copy lists and list items only. In this example I will show you how to migrate lists, and I mean a lots of them.
In this migration we split-up the input files into two separate. One .csv that contains the relation between the source-sites an destination-sites. And a second file that contains the lists that need to be migrated. Note that once again this lists.csv is generated by the same PowerShell we used for analysis.
By opening this generated file that contains all the lists in all the website in Excel we can easily filter out the lists that should not be migrated. Examples of lists that you may wish to exclude are:
- Web part gallery
- Pages libraries
- Tasks libraries
- Libraries with no list items
- Libraries based on a specific base-template
- Libraries that have not been modified in the last year
environment.ps1
An updated environment, see the description above in previous scenario
1$sitesFile = ".\sites.csv"
2$listsFile = ".\lists.csv"
3$startIndex = 5
4$endIndex = 10
5$dstSiteUrl = "https://your.sharepoint.com/sites/20190101/targetsite"
Input (sites) Note I do not display lists.csv because it is generated.
0-show.ps1
1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
2
3. ".\environment.ps1"
4
5if ($null -eq $endIndex) {
6 Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
7 break
8}
9
10if ($null -eq $dstSiteUrl) {
11 Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12 break
13}
14
15$input = [Array](Import-Csv -Path $listsFile)
16
17$i = 1
18foreach ($line in $input) {
19 if ($i -lt $startIndex) {
20 $i++
21 continue
22 }
23
24 $url = $line.Url
25 $url2 = $line.ListName
26 Write-Host "$i-$endIndex [$url] -> [$url2]" -ForegroundColor DarkGray
27
28 if ($i++ -ge $endIndex) {
29 Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
30 break
31 }
32}
0-validate.ps1
The validation script has been updated to also check if the source list still exists.
1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
2
3. ".\environment.ps1"
4
5Add-Type -Path "$PSScriptRoot\Microsoft.SharePoint.Client.dll"
6Add-Type -Path "$PSScriptRoot\Microsoft.SharePoint.Client.Runtime.dll"
7
8if ($null -eq $endIndex) {
9 Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
10 break
11}
12
13function checkIfWebExists($url, $listName) {
14 $clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url);
15 $clientContext.RequestTimeout = 5000
16 $currentWeb = $clientContext.Web;
17 $lists = $currentWeb.Lists
18 $clientContext.Load($lists)
19 $clientContext.Load($currentWeb)
20
21 try {
22 $clientContext.ExecuteQuery();
23 Write-Host " OK" -ForegroundColor Green
24 $listExists = $lists | Where-Object { $_.Title -eq $listName }
25 if (-Not $listExists) {
26 Write-Host $listName "not found" -ForegroundColor Red
27 }
28 }
29 catch {
30 Write-Host " $url does not exist" -ForegroundColor Red
31 }
32}
33
34Write-Host "Processing ($startIndex-$endIndex)" -ForegroundColor Green
35$input = Import-Csv -Path $listsFile
36
37$i = 1
38foreach ($line in $input) {
39 if ($i -lt $startIndex) {
40 $i++
41 continue
42 }
43
44 $url = $line.Url
45
46 Write-Host "[$i / $endIndex] $url $($line.ListName)" -NoNewline -ForegroundColor Yellow
47 checkIfWebExists $url $line.ListName
48
49 if ($i++ -ge $endIndex) {
50 Write-Host "break because of endIndex limit" -ForegroundColor Red
51 break
52 }
53}
54
1-migrate-lists.ps1
And finally the migration script I use for list migrations
1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
2
3. ".\environment.ps1"
4
5if ($null -eq $endIndex) {
6 Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
7 break
8}
9
10$input = [Array](Import-Csv -Path $listsFile)
11$sites = [Array](Import-Csv -Path $sitesFile)
12
13$i = 1
14foreach ($line in $input) {
15 if ($i -lt $startIndex) {
16 $i++
17 continue
18 }
19
20 $url = $line.Url
21 $dstSiteUrl = $sites | Where-Object { $_.SourceUrl -eq $url }
22
23 if ($null -eq $dstSiteUrl) {
24 Write-Host "Destination URL not found"
25 break
26 }
27 $dstSiteUrl = $dstSiteUrl.DestURL
28 Write-Host "$i-$endIndex["$url/$($line.ListName)"] -> ["$dstSiteUrl"]" -ForegroundColor DarkGray -NoNewline
29
30 $srcSite = Connect-Site -Url $url
31 $dstSite = Connect-Site -Url $dstSiteUrl
32 $srcList = Get-List -Site $srcSite -name $line.ListName
33 if ($null -eq $srcList) {
34 Write-Host "Source list not found"
35 break
36 }
37
38 $mappings = New-MappingSettings
39 Import-UserAndGroupMapping -MappingSettings $mappings -Path "$(Get-Location)\UserAndGroupMappings.sgum" | Out-Null
40
41 $copysettings = New-CopySettings -OnContentItemExists Skip
42 $result = Copy-List -List $srcList -DestinationSite $dstSite -CopySettings $copysettings -MappingSettings $mappings
43 Write-Host " Errors:" $result.Errors "Warnings:" $result.Warnings "Copied:" $result.ItemsCopied
44
45 $statusMessage = "OK"
46 if ($result.Warnings -gt 0) {
47 $statusMessage = "Warning"
48 Write-Host "Warning during copy" -ForegroundColor Yellow
49 }
50 if ($result.Errors -gt 0) {
51 $statusMessage = "ERROR"
52 Write-Host "Error during copy" -ForegroundColor Red
53 }
54
55 $filename = "$i-$($statusMessage).csv"
56 Export-Report $result -Path "./log/$filename" -Overwrite | Out-Null
57
58 Write-Host "Showing logfile errors and warnings ./log/$filename" -ForegroundColor Green
59 $input = Import-Csv "./log/$filename"
60
61 $input | ForEach-Object {
62 if ($_.Status -eq "Error") {
63 Write-Host "Error [$($_.Title)]" -NoNewline -ForegroundColor Magenta
64 Write-Host $_.Details -ForegroundColor Red
65
66 }
67 if ($_.Status -eq "Warning") {
68 Write-Host "Warning [$($_.Title)]" -NoNewline -ForegroundColor Magenta
69 Write-Host $_.Details -ForegroundColor Yellow
70 }
71 }
72
73 if ($i++ -ge $endIndex) {
74 Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
75 break
76 }
77}
Download the complete solution here