Migrating large corporate SharePoint intranets

Migration

In this third part of the series I will discuss the actual migration itself. If all previous steps were done right, the migration should be the easy part of the complete project.

Tool selection

I have found that for migrations to work almost autonomous only three tools are needed:

medium shadow rotate

  1. Excel, and to be more specific a comma separated file (.csv) that is being used as an input file for the migration
  2. PowerShell to script and automate the migration. Basically this consists of the following tasks:
  3. Read input from .csv
  4. [optional] do mutations / transformations on the input.csv (needed for a more specialised migration-scenario)
  5. Use ShareGate PowerShell cmdlets
  6. Process ShareGate logging
  7. ShareGate, the migration tool that does the actual migration

In addition of using PowerShell to automate the migration, we have seen that PowerShell is also a great to to analyse your environment. When combining these, the output .csv of the analysis phase can serve as the input for the migration phase.

Space allocation

Migrating to a new SharePoint environment (SharePoint online or SharePoint on-premise) will require storage and most likely lots it it. Make sure that the destination SharePoint farm is prepared to handle the amount of data. You can save some storage for on-premise migrations by turning down the SQL transaction logging. Also temporary disable SQL database mirroring can safe you from lots of unnecessary logging.

Migration actions

shadow

Your migration process will look something like the image displayed above. See this as your base-migration plan that can be altered depending on your situation. In general you do not want the users to be able to modify the source- site while the migration is in process. That means it is needed to remove these permissions before the migration starts. But since, most likely user permissions needs to be migrated too, the first step in this migration-plan is to export the user permissions.

Next step is the migration itself. The migration is done using PowerShell that processes a input.csv, uses ShareGate to migrate and handles the logging generated by ShareGate. One rule I found to be very helpful is to log everything to file but only show critical events on the screen. The code-samples below contain such filtering mechanisms to display only the critical messages.

Finally when the migration has completed it is time to check the log and do some basic validations. Once this is done, permissions can be imported so that the environment becomes accessible for the users. Next communicate the migration results to the site owner and inform them about anything noticeable information or issues.

Migration scenarios

To describe the details of the migration itself we have to distinguish different scenarios. Each will be described in detail below. As described before the migration is done using the ShareGate PowerShell cmdlets. The cmdlets to use depend on the type of migration that is needed.

I use the following table as a general guideline:

Migration type ShareGate cmdlet
site migration Copy-Site
list migration Copy-List
item migration Copy-Content (not the list definition itself)

First scenario: as-is migration

In this scenario no change is needed to the data and everything should be preserved as-is. One case I have used this is to ‘archive’ project that are no longer in use on an on-premise environment and to store them in SharePoint Online for archive. Of course this scenario can also be used to migrate to SharePoint Online for projects that are still in use.

shadow

Input

shadow

The input consists of a simple .csv file that contains all of the source site URLs. Creating this input.csv can either be done manually or by using the analyse-script described in the: Analysis (part II of IV) using recursion.

At the present time it has become a more common rule to use site collections instead of sites. When migrating from sites to site collections the Input csv can simply be extended to include a destination URL instead of using a single destination site URL as described below. Also notice that Sharegate can now also promote subsites to top-level sites as described in this article O365.tips

environment.ps1

These variables are used in the scripts below, it contains the name of the input file, a start index, end index and a destination site URL. Making use of a start and end index it is possible to migrate only a segment of the complete input file. Migrating in smaller batches gives the opportunity to do some in-between migration validations. Another reason could be that sometimes the migration cannot continue in the evening because of security patches e.g.

1$inputFile = ".\input.csv"
2$startIndex = 5
3$endIndex = 10
4$dstSiteUrl = "https://your.sharepoint.com/sites/20190101/targetsite"

open raw file

0-show.ps1

Useful script to simply display the lines of the input file that will be migrated using the given $startIndex = 5 and $endIndex = 10.

 1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
 2
 3. ".\environment.ps1"
 4
 5if ($null -eq $endIndex) {
 6    Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
 7    break
 8}
 9
10if ($null -eq $dstSiteUrl) {
11    Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12    break
13}
14
15$input = [Array](Import-Csv -Path $inputFile)
16
17$i = 1
18foreach ($line in $input) {
19    if ($i -lt $startIndex) {
20        $i++
21        continue
22    }
23
24    $url = $line.SiteUrl
25    Write-Host "$i-$endIndex [$url] -> [$dstSiteUrl]" -ForegroundColor DarkGray
26
27    if ($i++ -ge $endIndex) {
28        Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
29        break
30    }
31}

open raw file

1-migrate.ps1

The actual migration script. Basically just a plain PowerShell script, but makes use of the cmdlets provided by ShareGate such as: Connect-Site, New-MappingSettings, New-CopySettings and the Copy-Site. One rule I use is to log everything, but only display error and warning messages to the screen output. This way you do not get flooded by an information overload.

 1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
 2
 3. ".\environment.ps1"
 4
 5if ($null -eq $endIndex) {
 6    Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
 7    break
 8}
 9
10if ($null -eq $dstSiteUrl) {
11    Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12    break
13}
14
15$input = [Array](Import-Csv -Path $inputFile)
16$dstSite = Connect-Site -Url $dstSiteUrl -Browser
17
18$i = 1
19foreach ($line in $input) {
20    if ($i -lt $startIndex) {
21        $i++
22        continue
23    }
24    
25    $url = $line.SiteUrl
26    $projectNr = $url.split("/")[-1]
27    Write-Host "$i-$endIndex [$url] -> [$dstSiteUrl/$projectNr]" -ForegroundColor DarkGray
28    
29    $srcSite = Connect-Site -Url $url
30
31    $mappings = New-MappingSettings	
32    Import-UserAndGroupMapping -MappingSettings $mappings -Path "$(Get-Location)\UserAndGroupMappings.sgum" | Out-Null
33    Import-SiteTemplateMapping -MappingSettings $mappings -Path "$(Get-Location)\SiteTemplateMappings.sgwtm" | Out-Null
34    Import-PermissionLevelMapping -MappingSettings $mappings -Path "$(Get-Location)\PermissionLevelMappings.sgrm" | Out-Null
35
36    Write-Host $(Get-Date -Format "dddd MM/dd/yyyy HH:mm") -ForegroundColor White
37
38    $copysettings = New-CopySettings -OnContentItemExists Skip -OnSiteObjectExists Skip
39
40    $result = Copy-Site -MappingSettings $mappings -CopySettings $copysettings -Site $srcSite -DestinationSite $dstSite -NoCustomPermissions -NoWorkflows -NoSiteFeatures #-NoContent #-NoCustomizedFormsAndViews -NoNavigation
41    Write-Host " Errors:" $result.Errors "Warnings:" $result.Warnings "Copied:" $result.ItemsCopied
42
43    Write-Host $(Get-Date -Format "dddd MM/dd/yyyy HH:mm") -ForegroundColor White
44
45    $statusMessage = "OK"
46    if ($result.Warnings -gt 0) {
47        $statusMessage = "Warning"
48        Write-Host "Warning during copy" -ForegroundColor Yellow
49    }
50    if ($result.Errors -gt 0) {
51        $statusMessage = "ERROR"
52        Write-Host "Error during copy" -ForegroundColor Red
53    }
54
55    $filename = "$($i)-$($statusMessage).csv"
56    Export-Report $result -Path "./log/$filename" -Overwrite | Out-Null
57
58    Write-Host "Showing logfile errors and warnings ./log/$filename" -ForegroundColor Green
59    $input = Import-Csv "./log/$filename"
60
61    $input | ForEach-Object {
62        if ($_.Status -eq "Error") {
63            Write-Host "Error [$($_.Title)]" -NoNewline -ForegroundColor Magenta
64            Write-Host $_.Details -ForegroundColor Red
65
66        }
67        if ($_.Status -eq "Warning") {
68            Write-Host "Warning [$($_.Title)]" -NoNewline -ForegroundColor Magenta
69            Write-Host $_.Details -ForegroundColor Yellow
70        }
71
72    }
73
74    Write-Host
75    if ($i++ -ge $endIndex) {
76        Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
77        break
78    }
79}

open raw file

Download the complete solution here

Second scenario: list migrations into sites based on custom template

The second scenario I like to show is one where the destination site(s) are based on a custom developed / configured template. As described in the customizations part, there are situations in which it is better to develop a custom tailored template to fully suite the business case. One method of developing such templates is by making use of the OfficeDev PnP.

So when the site itself should not be copied but the content does you can choose to copy lists and list items only. In this example I will show you how to migrate lists, and I mean a lots of them.

In this migration we split-up the input files into two separate. One .csv that contains the relation between the source-sites an destination-sites. And a second file that contains the lists that need to be migrated. Note that once again this lists.csv is generated by the same PowerShell we used for analysis.

By opening this generated file that contains all the lists in all the website in Excel we can easily filter out the lists that should not be migrated. Examples of lists that you may wish to exclude are:

  1. Web part gallery
  2. Pages libraries
  3. Tasks libraries
  4. Libraries with no list items
  5. Libraries based on a specific base-template
  6. Libraries that have not been modified in the last year

environment.ps1

An updated environment, see the description above in previous scenario

1$sitesFile = ".\sites.csv"
2$listsFile = ".\lists.csv"
3$startIndex = 5
4$endIndex = 10
5$dstSiteUrl = "https://your.sharepoint.com/sites/20190101/targetsite"

open raw file

Input (sites) Note I do not display lists.csv because it is generated.

shadow

0-show.ps1

 1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
 2
 3. ".\environment.ps1"
 4
 5if ($null -eq $endIndex) {
 6    Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
 7    break
 8}
 9
10if ($null -eq $dstSiteUrl) {
11    Write-Host "check environment.ps1, missing dstSiteUrl" -ForegroundColor Yellow
12    break
13}
14
15$input = [Array](Import-Csv -Path $listsFile)
16
17$i = 1
18foreach ($line in $input) {
19    if ($i -lt $startIndex) {
20        $i++
21        continue
22    }
23
24    $url = $line.Url
25    $url2 = $line.ListName
26    Write-Host "$i-$endIndex [$url] -> [$url2]" -ForegroundColor DarkGray
27
28    if ($i++ -ge $endIndex) {
29        Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
30        break
31    }
32}

open raw file

0-validate.ps1

The validation script has been updated to also check if the source list still exists.

 1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
 2
 3. ".\environment.ps1"
 4
 5Add-Type -Path "$PSScriptRoot\Microsoft.SharePoint.Client.dll" 
 6Add-Type -Path "$PSScriptRoot\Microsoft.SharePoint.Client.Runtime.dll"
 7
 8if ($null -eq $endIndex) {
 9    Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
10    break
11}
12
13function checkIfWebExists($url, $listName) {
14    $clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url);
15    $clientContext.RequestTimeout = 5000
16    $currentWeb = $clientContext.Web;
17    $lists = $currentWeb.Lists
18    $clientContext.Load($lists)
19    $clientContext.Load($currentWeb)
20    
21    try {
22        $clientContext.ExecuteQuery();
23        Write-Host " OK" -ForegroundColor Green 
24        $listExists = $lists | Where-Object { $_.Title -eq $listName }
25        if (-Not $listExists) {
26            Write-Host $listName "not found" -ForegroundColor Red
27        }
28    } 
29    catch {
30        Write-Host " $url does not exist" -ForegroundColor Red
31    }
32}
33
34Write-Host "Processing ($startIndex-$endIndex)" -ForegroundColor Green
35$input = Import-Csv -Path $listsFile
36
37$i = 1
38foreach ($line in $input) {
39    if ($i -lt $startIndex) {
40        $i++
41        continue
42    }
43
44    $url = $line.Url
45
46    Write-Host "[$i / $endIndex] $url       $($line.ListName)" -NoNewline -ForegroundColor Yellow
47    checkIfWebExists $url $line.ListName
48
49    if ($i++ -ge $endIndex) {
50        Write-Host "break because of endIndex limit" -ForegroundColor Red
51        break
52    }
53}
54

open raw file

1-migrate-lists.ps1

And finally the migration script I use for list migrations

 1# Copyright (C) www.jurjan.info - All Rights Reserved (MIT License)
 2
 3. ".\environment.ps1"
 4
 5if ($null -eq $endIndex) {
 6    Write-Host "check environment.ps1, missing endIndex" -ForegroundColor Yellow
 7    break
 8}
 9
10$input = [Array](Import-Csv -Path $listsFile)
11$sites = [Array](Import-Csv -Path $sitesFile)
12
13$i = 1
14foreach ($line in $input) {
15    if ($i -lt $startIndex) {
16        $i++
17        continue
18    }
19    
20    $url = $line.Url
21    $dstSiteUrl = $sites | Where-Object { $_.SourceUrl -eq $url }
22
23    if ($null -eq $dstSiteUrl) {
24        Write-Host "Destination URL not found"
25        break
26    }
27    $dstSiteUrl = $dstSiteUrl.DestURL
28    Write-Host "$i-$endIndex["$url/$($line.ListName)"] -> ["$dstSiteUrl"]" -ForegroundColor DarkGray -NoNewline
29
30    $srcSite = Connect-Site -Url $url
31    $dstSite = Connect-Site -Url $dstSiteUrl
32    $srcList = Get-List -Site $srcSite -name $line.ListName
33    if ($null -eq $srcList) {
34        Write-Host "Source list not found"
35        break
36    }
37
38    $mappings = New-MappingSettings	
39    Import-UserAndGroupMapping -MappingSettings $mappings -Path "$(Get-Location)\UserAndGroupMappings.sgum" | Out-Null
40
41    $copysettings = New-CopySettings -OnContentItemExists Skip
42    $result = Copy-List -List $srcList -DestinationSite $dstSite -CopySettings $copysettings -MappingSettings $mappings
43    Write-Host " Errors:" $result.Errors "Warnings:" $result.Warnings "Copied:" $result.ItemsCopied
44
45    $statusMessage = "OK"
46    if ($result.Warnings -gt 0) {
47        $statusMessage = "Warning"
48        Write-Host "Warning during copy" -ForegroundColor Yellow
49    }
50    if ($result.Errors -gt 0) {
51        $statusMessage = "ERROR"
52        Write-Host "Error during copy" -ForegroundColor Red
53    }
54
55    $filename = "$i-$($statusMessage).csv"
56    Export-Report $result -Path "./log/$filename" -Overwrite | Out-Null
57   
58    Write-Host "Showing logfile errors and warnings ./log/$filename" -ForegroundColor Green
59    $input = Import-Csv "./log/$filename"
60
61    $input | ForEach-Object {
62        if ($_.Status -eq "Error") {
63            Write-Host "Error [$($_.Title)]" -NoNewline -ForegroundColor Magenta
64            Write-Host $_.Details -ForegroundColor Red
65
66        }
67        if ($_.Status -eq "Warning") {
68            Write-Host "Warning [$($_.Title)]" -NoNewline -ForegroundColor Magenta
69            Write-Host $_.Details -ForegroundColor Yellow
70        }
71    }
72
73    if ($i++ -ge $endIndex) {
74        Write-Host "break because of endIndex limit" -ForegroundColor DarkYellow
75        break
76    }
77}

open raw file

Download the complete solution here

Continue reading this series here