Import/Export/Copy Lists

One of the challenges I'm currently trying to solve is related to the site directory. After my upgrade the site directory is a bit messed up (the upgrade seemed to have merged columns). In an attempt to get the site directory in a state that I can work with I'm looking at several possibilities. One of those was configure the site directory the way I want it, export the Sites list and then import back after my upgrade completes. In order to accomplish this I had to create the ability to import and export lists (turns out this alone won't solve my problems but the commands I've created could be extremely handy in many scenarios).

Stsadm has the ability out of the box to export and import web sites but it doesn't support doing just lists. However, the content deployment API does allow you to import and export any object type (right down to an individual list item). So the solution was rather simple - I just created two new commands which function almost identically to the existing import and export commands except that they work with lists (export takes in an URL representing a view of the list and import takes in a target web to import the list in to).

I also created a copy command which wraps up the import and export so that both can be done in one step (passing "-deletesource" to the copylist command will make it function like a move command). While working on all this I also decided that I got tired of writing the same parameter validation code over and over again so I created a new base class (SPOperation) and some parameter classes (SPParam and SPParamCollection). These classes use several validator classes that help with more complex validations. Note however that I didn't re-invent the wheel here - these classes are all almost straight copies of what Microsoft uses internally in the stsadm.exe assembly itself (if you use Reflector to look at stsadm.exe you'll see these classes defined there - I didn't want to deal with lots of reflection to use these so I just recreated them thereby allowing me to make modifications to suite my own needs where necessary).

The code to work with the content deployment API is pretty straightforward. Stefan Goßner does a really good job at explaining how to use the API in his blog Deep Dive into SharePoint Content Deployment and Migration API so I won't reiterate it here. The commands I created are detailed below.

1. gl-exportlist

This command has almost exactly the same parameters as the built in "export" command. The only thing of note is that the default value for the versions flag is 4 (or All versions for files and list items) in mine whereas the default for the export operation is 1 (or Last major version for files and list items). I went back and forth on whether I should keep it the same but decided in the end that for my purposes when I want to copy a list I usually want to retain everything so I chose that as the default. I also considered whether I wanted the caller to pass in the list name or GUID or something else and in the end I decided that the easiest thing would be to use the URL to a view.

I created a simple helper method which takes this passed in url and searches all lists and all views until it finds a match (there's probably an easier way to do this but this works and I didn't have a lot of time to mess with it). The syntax of the command can be seen below:

c:\>stsadm -help gl-exportlist stsadm -o gl-exportlist Exports a list. Parameters: -url <list view url> -filename <export file name> [-overwrite] [-includeusersecurity] [-haltonwarning] [-haltonfatalerror] [-nologfile] [-versions <1-4> 1 - Last major version for files and list items 2 - The current version, either the last major or the last minor 3 - Last major and last minor version for files and list items 4 - All versions for files and list items (default)] [-cabsize <integer from 1-1024 megabytes> (default: 25)] [-nofilecompression] [-includedescendants <All | Content | None>] [-excludedependencies (Specifies whether to exclude dependencies from the export package when exporting objects of type SPFile or SPListItem)] [-quiet]

Here’s an example of how to export the Documents list from the root site collection:

stsadm –o gl-exportlist -url "http://intranet/Documents/Forms/AllItems.aspx" -filename "c:\docs" -includeusersecurity -versions 4 -nofilecompression

Running the above will create a folder called "docs" in the root c: drive. You'll also find a log file there containing the same information that was dumped to the console. This same output can then be used by the gl-importlist command to copy the list to another site collection (note that the target could be on any web app in any farm).

Update 11/3/2008: I've added a new option, excludedependencies, which allows you to exclude exporting dependent items - this will get around a lot of issues that some people have had when trying to import and were receiving errors about dependent items not existing.

2. gl-importlist

This command has almost exactly the same parameters as the built in "import" command. One additional parameter of note is the "-retargetlinks" parameter. If this parameter is specified then you must also specify the "-sourceurl" parameter which corresponds to a view of the source list. The "retargetlinks" parameter tells the code to find any links that point to items in the source list and make them point to the target list. This becomes helpful in a move operation. Note that only items identified by the list items BackwardLinks collection will get modified (unfortunately I'm not 100% clear about when and how this collection is set so I don't claim that using this will fix every link to your items, though I am planning on creating another command that will loop through every field of every item of every list of every site and adjust the field value by replacing one value with another). The syntax of the command can be seen below:

C:\>stsadm -help gl-importlist

stsadm -o gl-importlist

Imports a list.

Parameters:
        -url <url of a web site to import to>
        -filename <import file name>
        [-includeusersecurity]
        [-haltonwarning]
        [-haltonfatalerror]
        [-nologfile]
        [-updateversions <1-3>
                1 - Add new versions to the current file (default)
                2 - Overwrite the file and all its versions (delete then insert)
                3 - Ignore the file if it exists on the destination]
        [-nofilecompression]
        [-quiet]
        [-retargetlinks (resets links pointing to the source to now point to the target)]
        [-sourceurl <url to a view of the original list> (use if retargetlinks)]
        [-retainobjectidentity]
        [-copysecuritysettings (must provide sourceurl and includeusersecurity)]

Here’s an example of how to import the Documents list that we exported above and re-target any links pointing to the original documents so that they now point to the new documents (note that just like the built in import operation if you specify nofilecompression for the export you must also specify it for the import):

stsadm –o gl-importlist -url "http://teamsites/" -filename "c:\docs" -includeusersecurity -updateversions 2 -nofilecompression -retargetlinks -sourceurl "http://intranet/Documents/Forms/AllItems.aspx"

As with the gl-exportlist operation a log file will be generated which will include all the details that were dumped to the console.

Update 10/8/2007: I've added a retainobjectidentity identity parameter to the gl-importlist command which allows you keep all IDs the same after the import. Note that the source cannot exist in the content database or you will get an error using this switch.

3. gl-copylist

If you've got the ability to import and export a list then why not make it a one step operation to copy a list or move a list. Once I had the code for the import and export the copy was nothing more than a wrapper which would call into the appropriate methods. By adding an extra parameter to support deleting the source list I now also had a move command. The syntax of the command is just the combination of the import and export minus that which I don't need the user to set. The syntax of the command can be seen below:

C:\>stsadm -help gl-copylist

Copies a list to a new web.

Parameters:
        -sourceurl <list view url to copy>
        -targeturl <url of a web site to copy to>
        [-includeusersecurity]
        [-haltonwarning]
        [-haltonfatalerror]
        [-nologfile]
        [-versions <1-4>
                1 - Last major version for files and list items
                2 - The current version, either the last major or the last minor
                3 - Last major and last minor version for files and list items
                4 - All versions for files and list items (default)]
        [-updateversions <1-3>
                1 - Add new versions to the current file (default)
                2 - Overwrite the file and all its versions (delete then insert)

                3 - Ignore the file if it exists on the destination]
        [-quiet]
        [-retargetlinks (resets links pointing to the source to now point to the target)]
        [-deletesource]
        [-temppath <temporary folder path for storing of export files>]
        [-includedescendants <All | Content | None>]
        [-excludedependencies (Specifies whether to exclude dependencies from the export package when exporting objects of type SPFile or SPListItem)]

Here’s an example of how to perform the same operation that the above import and export commands were doing but with just one step while at the same time deleting the source list:

stsadm –o gl-copylist -sourceurl "http://intranet/Documents/Forms/AllItems.aspx" -targeturl "http://teamsites/" -includeusersecurity -updateversions 2 -versions 4 -retargetlinks

If the source list was deleted then the result of the export operation are saved to a temp folder, the path to which is displayed to the user upon completion.

Update 11/3/2008: I've added a new option, excludedependencies, which allows you to exclude exporting dependent items - this will get around a lot of issues that some people have had when trying to import and were receiving errors about dependent items not existing.  I've also added the includedescendants parameter to the copy command to bring it in line with the export command.

4. gl-getlistschemaxml

This is a command that I really only created to help me debug something. I was going to delete it but figured I'd keep it around in case someone found a use for it. All the command does is grab an instance of the SPList object corresponding to the view URL that was passed in and then outputs the SchemaXml property (using an XmlTextWriter to provide formatting). Maybe someone will find a use for it (perhaps in creating a feature definition file or something). The syntax of the command can be seen below:

C:\>stsadm -help gl-getlistschemaxml

Returns the XML schema for a list.

Parameters:
        -url <list view url>

Here’s an example of how to get the schema XML for the Documents list:

stsadm –o gl-getlistschemaxml -url "http://intranet/Documents/Forms/AllItems.aspx"

The output for this is rather lengthy so I won't bother showing it here.

Update 11/16/2007: I've updated the gl-copylist and gl-importlist commands to now support the functionality of the gl-copylistsecurity command. The content above has been updated to reflect this change where appropriate. Note that for this to work for the gl-importlist command the sourceurl and includeusersecurity parameters must be specified.