MOSS MVP

I've moved my blog to http://blog.falchionconsulting.com!. Please update your links. This blog is no longer in use--you can find all posts and comments at my new blog; I will no longer be posting to this site and comments have been disabled.

Saturday, October 6, 2007

A Better Import

Some people may have read my post from a few days ago regarding the retargetting of the content query web part which was necessary because I had to move a sub-site from one site collection to another. In order to do the move I used the built-in import command. The problem that I sought to address with the retargetting of the content query web part was the result of the import creating new object identifiers for the lists and other such items that were imported. The only thing is that my solution only addressed the symptoms and not the actual problem and it did not address the other similar issues (such as data form web parts which suffered the same issue).

The real problem was with the import itself - internally the built-in import command sets the RetainObjectIdentity property of the SPImportSettings object to false and there is no option that you can pass into the command to override this. This is most unfortunate as this particular property has a profound affect on how objects are imported. If you set this to false then all objects are given a new ID - this is great if your intention is to duplicate a web or list within a given content database but if your intention is to move a web or list then resetting all the object IDs can really mess some things up.

For instance - if you have use the IT Team Workspace template from the Fantastic 40 application templates and attempt to move an instance of that web to another site collection via the export/import commands you will be rather annoyed to find that all of the data form web parts on the main page no longer function. If you attempt to open the web parts in SharePoint Designer to fix them you'll find that you are unable to. The only solution is to export the web part, change the old IDs to the new IDs and then re-import the web part.

But this headache could have all been avoided if we could have told the import statement to retain the original object identifiers. So, as a solution to this problem I built a new version of the import command called gl-import2. My version functions exactly the same as the built-in version but adds to it the ability to specify whether you wish to retain the original object identifiers. If you do not wish to do this then it will call out to the built-in command thereby only using my custom version if you wish to retain object IDs.

Fortunately I already had most of the code as I had to create it for the gl-importlist command that I created. The additional code I figured out by simply using Reflector to look at what Microsoft was doing with the built-in import command. In this way I was able to make sure that my command preserved all the functionality of the original. The core code is shown below:

   1: /// <summary>
   2: /// Sets up the import object.
   3: /// </summary>
   4: /// <param name="settings">The settings.</param>
   5: /// <param name="compressFile">if set to <c>true</c> [compress file].</param>
   6: /// <param name="filename">The filename.</param>
   7: /// <param name="haltOnFatalError">if set to <c>true</c> [halt on fatal error].</param>
   8: /// <param name="haltOnWarning">if set to <c>true</c> [halt on warning].</param>
   9: /// <param name="includeusersecurity">if set to <c>true</c> [includeusersecurity].</param>
  10: /// <param name="logFile">if set to <c>true</c> [log file].</param>
  11: /// <param name="quiet">if set to <c>true</c> [quiet].</param>
  12: /// <param name="updateVersions">The update versions.</param>
  13: /// <param name="retainObjectIdentity">if set to <c>true</c> [retain object identity].</param>
  14: internal static void SetupImportObject(SPImportSettings settings, bool compressFile, string filename, bool haltOnFatalError, bool haltOnWarning, bool includeusersecurity, bool logFile, bool quiet, SPUpdateVersions updateVersions, bool retainObjectIdentity)
  15: {
  16:  settings.CommandLineVerbose = !quiet;
  17:  settings.HaltOnNonfatalError = haltOnFatalError;
  18:  settings.HaltOnWarning = haltOnWarning;
  19:  settings.FileCompression = compressFile;
  20:  
  21:  if (!compressFile)
  22:  {
  23:   if (string.IsNullOrEmpty(filename) || !Directory.Exists(filename))
  24:   {
  25:    throw new SPException(SPResource.GetString("DirectoryNotFoundExceptionMessage", new object[] { filename }));
  26:   }
  27:  }
  28:  else if (string.IsNullOrEmpty(filename) || !File.Exists(filename))
  29:  {
  30:   throw new SPException(SPResource.GetString("FileNotFoundExceptionMessage", new object[] { filename }));
  31:  }
  32:  
  33:  if (!compressFile)
  34:  {
  35:   settings.FileLocation = filename;
  36:  }
  37:  else
  38:  {
  39:   string path;
  40:   Utilities.SplitPathFile(filename, out path, out filename);
  41:   settings.FileLocation = path;
  42:   settings.BaseFileName = filename;
  43:  }
  44:  
  45:  if (logFile)
  46:  {
  47:   if (!compressFile)
  48:   {
  49:    settings.LogFilePath = Path.Combine(settings.FileLocation, "import.log");
  50:   }
  51:   else
  52:    settings.LogFilePath = Path.Combine(settings.FileLocation, filename + ".import.log");
  53:  }
  54:  
  55:  
  56:  if (includeusersecurity)
  57:  {
  58:   settings.IncludeSecurity = SPIncludeSecurity.All;
  59:   settings.UserInfoDateTime = SPImportUserInfoDateTimeOption.ImportAll;
  60:  }
  61:  settings.UpdateVersions = updateVersions;
  62:  settings.RetainObjectIdentity = retainObjectIdentity;
  63: }
  64:  
  65: /// <summary>
  66: /// Imports the site providing the option to retain the source objects ID values.  The source object
  67: /// must no longer exist in the target content database.
  68: /// </summary>
  69: /// <param name="filename">The filename.</param>
  70: /// <param name="targeturl">The targeturl.</param>
  71: /// <param name="haltOnWarning">if set to <c>true</c> [halt on warning].</param>
  72: /// <param name="haltOnFatalError">if set to <c>true</c> [halt on fatal error].</param>
  73: /// <param name="noFileCompression">if set to <c>true</c> [no file compression].</param>
  74: /// <param name="includeUserSecurity">if set to <c>true</c> [include user security].</param>
  75: /// <param name="quiet">if set to <c>true</c> [quiet].</param>
  76: /// <param name="logFile">if set to <c>true</c> [log file].</param>
  77: /// <param name="retainObjectIdentity">if set to <c>true</c> [retain object identity].</param>
  78: public void ImportSite(string filename, string targeturl, bool haltOnWarning, bool haltOnFatalError, bool noFileCompression, bool includeUserSecurity, bool quiet, bool logFile, bool retainObjectIdentity)
  79: {
  80:  if (!retainObjectIdentity)
  81:  {
  82:   // Use the built in "import" command.
  83:   ImportSite(filename, targeturl, haltOnWarning, haltOnFatalError, noFileCompression, includeUserSecurity,
  84:        quiet, logFile);
  85:   return;
  86:  }
  87:  
  88:  SPImportSettings settings = new SPImportSettings();
  89:  SPImport import = new SPImport(settings);
  90:  
  91:  SetupImportObject(settings, !noFileCompression, filename, haltOnFatalError, haltOnWarning, includeUserSecurity, logFile, quiet, SPUpdateVersions.Append, retainObjectIdentity);
  92:  
  93:  using (SPSite site = new SPSite(targeturl))
  94:  {
  95:   settings.SiteUrl = site.Url;
  96:   ExportList.ValidateUser(site);
  97:  
  98:   string dirName;
  99:   Utilities.SplitUrl(Utilities.ConvertToServiceRelUrl(Utilities.GetServerRelUrlFromFullUrl(targeturl), site.ServerRelativeUrl), out dirName, out m_webName);
 100:   m_webParentUrl = site.ServerRelativeUrl;
 101:   if (!string.IsNullOrEmpty(dirName))
 102:   {
 103:    if (!m_webParentUrl.EndsWith("/"))
 104:    {
 105:     m_webParentUrl = m_webParentUrl + "/";
 106:    }
 107:    m_webParentUrl = m_webParentUrl + dirName;
 108:   }
 109:   if (m_webName == null)
 110:   {
 111:    m_webName = string.Empty;
 112:   }
 113:  }
 114:  
 115:  EventHandler<SPDeploymentEventArgs> handler = new EventHandler<SPDeploymentEventArgs>(OnSiteImportStarted);
 116:  import.Started += handler;
 117:  
 118:  try
 119:  {
 120:   import.Run();
 121:  }
 122:  catch (SPException ex)
 123:  {
 124:   if (retainObjectIdentity && ex.Message.StartsWith("The Web site address ") && ex.Message.EndsWith(" is already in use."))
 125:   {
 126:    throw new SPException(
 127:     "You cannot import the web because the source web still exists.  Either specify the \"-deletesource\" parameter or manually delete the source web and use the exported file.", ex);
 128:  
 129:   }
 130:   else
 131:    throw;
 132:  }
 133:  finally
 134:  {
 135:   Console.WriteLine();
 136:   Console.WriteLine("Log file generated: ");
 137:   Console.WriteLine("\t{0}", settings.LogFilePath);
 138:   Console.WriteLine();
 139:  }
 140: }
 141:  
 142: /// <summary>
 143: /// Called when [site import started].
 144: /// </summary>
 145: /// <param name="sender">The sender.</param>
 146: /// <param name="args">The <see cref="Microsoft.SharePoint.Deployment.SPDeploymentEventArgs"/> instance containing the event data.</param>
 147: private void OnSiteImportStarted(object sender, SPDeploymentEventArgs args)
 148: {
 149:  SPImportObjectCollection rootObjects = args.RootObjects;
 150:  if (rootObjects.Count != 0)
 151:  {
 152:   if (rootObjects.Count != 1)
 153:   {
 154:    for (int i = 0; i < rootObjects.Count; i++)
 155:    {
 156:     if (rootObjects[i].Type == SPDeploymentObjectType.Web)
 157:     {
 158:      rootObjects[i].TargetParentUrl = m_webParentUrl;
 159:      rootObjects[i].TargetName = m_webName;
 160:      return;
 161:     }
 162:    }
 163:   }
 164:   else
 165:   {
 166:    rootObjects[0].TargetParentUrl = m_webParentUrl;
 167:    rootObjects[0].TargetName = m_webName;
 168:   }
 169:  }
 170: }
The syntax of the command I created can be seen below. With luck Microsoft will realize the benefits of this capability and will add the parameter to the built-in command with the next service pack. Note that I also added the same ability to retain object IDs to my gl-importlist command.

C:\>stsadm -help gl-import2

stsadm -o gl-import2

An improved version of the built-in "import" command.  Allows the ability to retain the original object identifiers.

Parameters:
        -url <URL to import to (parent if retainobjectidentity, otherwise full url of target location)>
        -filename <import file name>
        [-includeusersecurity]
        [-haltonwarning]
        [-haltonfatalerror]
        [-nologfile]
        [-updateversions <1-3>
            1 - Add new versions to the current file (default)
            2 - Overwrite the file and all its versions (delete then insert)
            3 - Ignore the file if it exists on the destination]
        [-nofilecompression]
        [-quiet]
        [-retainobjectidentity (the source must not exist in the content database and the url parameter must refer to the parent web)]

Here's an example of how to import a web into a site collection while maintaining all the original object IDs:

stsadm -o gl-import2 -url "http://intranet/hr" -filename "c:\export.cmp" -includeusersecurity -retainobjectidentity

 

Update 10/11/2007: A clarification is needed on the impact of the retainobjectidentity. This parameter will affect how the URL is provided. If you set retainobjectidentity then the url must correspond to the targets parent web - a new web site will be created using the name of the exported site (so if the exported site was "http://intranet/test1" and you want the new path to be "http://teamsites/sites/site1/test1" you would specify the url as "http://teamsites/sites/site1" - note that /sites/site1/test1 cannot already exist - don't use createweb). If you do not specify retainobjectidentity then the url parameter must be the target url (this allows you to specify a different web name) but the target web must already exist with a matching template (use createweb to create the placeholder web). If you wanted to do the same as described above but without retaining the object IDs then you would specify the url as "http://teamsites/sites/site1/test1". Also - I discovered that the retainobjectidentity switch introduces another limitation - it really only works with webs that are not sub-webs of another web (so first level child objects of a site collection). This is a limitation of the deployment API and not my command.

19 comments:

Daryll said...

SUPERB!! - This solved a massive problem that I've had with moving the application templates. MS would do well to follow your lead my friend.

Thank you so much for sharing your fix.

Ryanj1 said...

Gary,

Have you encountered any problems with imports when versioning is enabled on the site? I have had some hit and miss problems with documents that have multiple versions. It generates a version not found error. I have not been able to pin point the problem. The command I am using is:
stsadm.Exe -o import -url http://spqteams/moss2007/ -filename E:\Backup\moss2007.bak -includeusersecurity -updateversions 2

Thanks,
Ryan Johnson

Gary Lapointe said...

Unfortunately there are just tons of issues with the built-in deployment API - you may want to search through the MSDN forums for I believe I've seen this specific issue discussed (can't remember if there was a solution though).

Anonymous said...

I am slightly confused by your update on this post. You say it doesn't work with subwebs? Does this mean I can only really use this to move an entire site collection around? Can I export http://localhost/myweb to another server and the same target url?

Gary Lapointe said...

The limitation is just when using retainobjectidentity - you can move first level child webs, just not sub-webs of those - so you should be able to move http://localhost/myweb but to be honest the deployment API is so flaky I can't promise that it will work smoothly.

Eric said...

Hi Gary,

Thanks for the solution!

I think I might be doing something wrong, but when I attempt to move my site with retainobjectidentity set, the operation shows successful but no new site was created.

Any possible cause?

Gary Lapointe said...

If you try to run without retainobjectidentity does it add the site?

Eric said...

Hi Gary,

When I run the command without the retainobjectidentity, my site is added but with web part errors.

Gary Lapointe said...

Eric - that specific problem is actually why I added the retainobjectidentity parameter - if you have list view web parts, for example, that reference a specific list those web parts will be broken unless you retain the identity of the list (the list view web part references the list by it's GUID and not it's name or path). There are other web parts that have this same limitation.

Eric said...

Gary,

Thanks alot for your help! The problem has been resolved. It seems that the problem lies with exporting the site using Designer 2007.

Anyway, your efforts are a great help to the community!

Anonymous said...

Thanks for the tool. I am getting an error when runing import with the -retainobjectidentity switch. It says that the name already exists. I am importing to a new site collection. Thanks

Gary Lapointe said...

Try creating the site collection without specifying a site template then import on top of that site collection.

Simon Baynes said...

Gary,

You're a genius, thank you very much for this...you've scored me some serious brownite points!

Thanks again.

Simon Baynes.

Anonymous said...

Hi Gary,

I am a newbie to Sharepoint. I do not have any issues at present, but am trying to understand how I would actually use your code above to add the gl-import2 operation to stsadm.
Thank you.

Orville

Gary Lapointe said...

The download page has the install instructions.

Kieran said...

I'm getting various access denied messages along the lines of:

Debug: Security check failed in OnListItemImport
Warning: Access denied.

when trying to run this command, even though I am using the farm admin account.

Any idea's what to look for?

all the best

Kieran said...

Hi Gary

When using the stsadm -o gl-import2 command I am getting an error relating to a list already existing, the import then seems to fail. Should it not handle this or am I missing something?

I then used the UI to delete the existing list and ran the import again. It then seemed to fail at the same point saying the list exists. In the the UI the list appears but when I click it, reports that the list does not exist.

Any pointers?

Gary Lapointe said...

Validate that you are in fact a farm admin and that you are a local admin and have dbo rights to all the databases.

The destroyer of all he ever wanted said...

Hi All,
I am hoping that someone will read this who has done what I am currently trying to do and am experiencing issues with. I am exporting a site from a WSS 3 SP2 server into a MOSS 2007 SP2 server. The site has child/sub-sites and several meeting and document workspaces. When I imported I received an error regarding each workspace imported. Here is the error from the import log, "[4/1/2010 2:01:22 AM]: Progress: Importing ListItem /teams/pm/Partnerships/WorldView/_catalogs/masterpage?id=1.
[4/1/2010 2:01:22 AM]: Verbose: Deleting...
[4/1/2010 2:01:22 AM]: Warning: File cannot be deleted will try to append the file instead.
*** Inner exception:
Cannot remove file "default.master". Error Code: 158.
at Microsoft.SharePoint.Library.SPRequest.AddOrDeleteUrl(String bstrUrl, String bstrDirName, Boolean bAdd, UInt32 dwDeleteOp, Int32 iUserId, Guid& pgDeleteTransactionId)
at Microsoft.SharePoint.Deployment.FileSerializer.DeleteFileForOverride(Object fileOrListItem, SPWeb web, String fileUrl, SPImportSettings settings)
[4/1/2010 2:01:22 AM]: Progress: Importing ListItem /teams/pm/Partnerships/WorldView/_catalogs/masterpage?id=2."

I have the updated STSADM extensions installed on both systems as well. Any help you can provide will be greatly appreciated!