MOSS MVP

I've moved my blog to http://blog.falchionconsulting.com!. Please update your links. This blog is no longer in use--you can find all posts and comments at my new blog; I will no longer be posting to this site and comments have been disabled.

Tuesday, December 18, 2007

Save List Items and Files to Disk

I've seen numerous examples of people needing to save all the files from a document library or custom list (containing attachments) to disk. I didn't necessarily need the ability myself for the upgrade we are doing but I did need a quick way to generate lots of different samples to make sure that my gl-addlistitem command was working correctly. So I decided to create a new command which would make my testing easier as well as help the many out there that have the need of saving lots of files out to disk. The command I created is gl-exportlistitem2. I already had an gl-exportlistitem command which used the deployment API and I just wasn't feeling very creative with the name so I just added "2" (maybe "savelistdata" is better???). The command does two key things - saves all the files to a specified path and creates a Manifest.xml file that contains information about the files and any list items that were in the list. This information can then be used by the gl-addlistitem command to actually import the data into another list. For this initial version I've kept things fairly simple - there's no compression, no security information, and no version history. I'm only storing the file(s) (if present) and any field data (perhaps I'll look to handle more data in the future but for now this met my needs). The nice thing is that if you don't need any of the other information then what I created actually works better than using the deployment API as mine actually takes folder location into account whereas the deployment API is extremely buggy when it comes to folders. I also included a simplified version of the code which just simply dumps all the files to disk without the manifest information (the command does not use this but I kept it in the source in case anyone needed it). The code to do all of this is really straightforward - I decided to break it up into two chunks - the first gathers all the necessary data from the list and stores it in some custom data classes and the second takes those classes and saves to disk and creates the actual manifest file:

   1: /// <summary>
   2: /// Gets the item data.
   3: /// </summary>
   4: /// <param name="web">The web.</param>
   5: /// <param name="list">The list.</param>
   6: /// <param name="ids">The ids.</param>
   7: /// <returns></returns>
   8: private static List<ItemInfo> GetItemData(SPWeb web, SPList list, List<int> ids)
   9: {
  10:  List<ItemInfo> itemData = new List<ItemInfo>();
  11:  
  12:  foreach (SPListItem item in list.Items)
  13:  {
  14:   if (!(ids.Count == 0 || ids.Contains(item.ID)))
  15:    continue;
  16:  
  17:   ItemInfo info = new ItemInfo();
  18:   itemData.Add(info);
  19:   info.ID = item.ID;
  20:   
  21:   if (item.File != null)
  22:   {
  23:    info.File = new FileDetails(item.File.OpenBinary(), item.File.Name, item.File.Author, item.File.TimeCreated);
  24:    info.Title = item.File.Name;
  25:   }
  26:   else
  27:    info.Title = item.Title;
  28:  
  29:   info.FolderUrl = item.Url.Substring(list.RootFolder.Url.ToString().Length, item.Url.LastIndexOf("/") - list.RootFolder.Url.ToString().Length);
  30:  
  31:   try
  32:   {
  33:    foreach (string fileName in item.Attachments)
  34:    {
  35:     SPFile file = web.GetFile(item.Attachments.UrlPrefix + fileName);
  36:     info.Attachments.Add(new FileDetails(file.OpenBinary(), file.Name, file.Author, file.TimeCreated));
  37:    }
  38:   }
  39:   catch (ArgumentException)
  40:   {}
  41:  
  42:   foreach (SPField field in list.Fields)
  43:   {
  44:    if (!field.ReadOnlyField && 
  45:     field.InternalName != "Attachments" && 
  46:     field.InternalName != "FileLeafRef" &&
  47:     item[field.InternalName] != null)
  48:    {
  49:     info.FieldData.Add(field.InternalName, item[field.InternalName].ToString());
  50:    }
  51:   }
  52:  }
  53:  return itemData;
  54: }
  55:  
  56: /// <summary>
  57: /// Gets the item data from XML.
  58: /// </summary>
  59: /// <param name="itemData">The item data.</param>
  60: /// <param name="manifestPath">The manifest path.</param>
  61: private static void SaveItemData(List<ItemInfo> itemData, string manifestPath)
  62: {
  63:  if (string.IsNullOrEmpty(manifestPath))
  64:   throw new ArgumentNullException("manifest", "No directory was specified for the manifest.");
  65:  
  66:  if (!Directory.Exists(manifestPath))
  67:   Directory.CreateDirectory(manifestPath);
  68:  
  69:  string dataPath = Path.Combine(manifestPath, "Data");
  70:  
  71:  StringBuilder sb = new StringBuilder();
  72:  
  73:  XmlTextWriter xmlWriter = new XmlTextWriter(new StringWriter(sb));
  74:  xmlWriter.Formatting = Formatting.Indented;
  75:  
  76:  xmlWriter.WriteStartElement("Items");
  77:  
  78:  foreach (ItemInfo info in itemData)
  79:  {
  80:   xmlWriter.WriteStartElement("Item");
  81:  
  82:   if (info.File != null)
  83:   {
  84:    string folder = Path.Combine(dataPath, info.FolderUrl.Trim('\\', '/')).Replace("/", "\\");
  85:    if (!Directory.Exists(folder))
  86:     Directory.CreateDirectory(folder);
  87:  
  88:    xmlWriter.WriteAttributeString("File", Path.Combine(folder, info.File.Name));
  89:    xmlWriter.WriteAttributeString("Author", info.File.Author.LoginName);
  90:    xmlWriter.WriteAttributeString("CreatedDate", info.File.CreatedDate.ToString());
  91:    File.WriteAllBytes(Path.Combine(folder, info.File.Name), info.File.File);
  92:   }
  93:   xmlWriter.WriteAttributeString("LeafName", info.Title);
  94:   xmlWriter.WriteAttributeString("FolderUrl", info.FolderUrl);
  95:     
  96:   xmlWriter.WriteStartElement("Fields");
  97:   foreach (string key in info.FieldData.Keys)
  98:   {
  99:    xmlWriter.WriteStartElement("Field");
 100:    xmlWriter.WriteAttributeString("Name", key);
 101:    xmlWriter.WriteString(info.FieldData[key]);
 102:    xmlWriter.WriteEndElement(); // Field
 103:   }
 104:   xmlWriter.WriteEndElement(); // Fields
 105:  
 106:   xmlWriter.WriteStartElement("Attachments");
 107:   foreach (FileDetails file in info.Attachments)
 108:   {
 109:    string folder = Path.Combine(Path.Combine(dataPath, info.FolderUrl.Trim('\\', '/')).Replace("/", "\\"), "item_" + info.ID);
 110:    if (!Directory.Exists(folder))
 111:     Directory.CreateDirectory(folder);
 112:    
 113:    xmlWriter.WriteElementString("Attachment", Path.Combine(folder, file.Name));
 114:  
 115:    File.WriteAllBytes(Path.Combine(folder, file.Name), file.File);
 116:   }
 117:   xmlWriter.WriteEndElement(); // Attachments
 118:  
 119:   xmlWriter.WriteEndElement(); // Item
 120:  }
 121:  
 122:  xmlWriter.WriteEndElement();
 123:  xmlWriter.Flush();
 124:  
 125:  File.WriteAllText(Path.Combine(manifestPath, "Manifest.xml"), sb.ToString());
 126: }
 127:  
 128: #region Private Classes
 129:  
 130: private class FileDetails
 131: {
 132:  public byte[] File = null;
 133:  public string Name = null;
 134:  public SPUser Author = null;
 135:  public DateTime CreatedDate = DateTime.Now;
 136:  public FileDetails(byte[] file, string name, SPUser author, DateTime createdDate)
 137:  {
 138:   File = file;
 139:   Name = name;
 140:   Author = author;
 141:   CreatedDate = createdDate;
 142:  }
 143: }
 144: private class ItemInfo
 145: {
 146:  public FileDetails File = null;
 147:  public string FolderUrl = null;
 148:  public List<FileDetails> Attachments = new List<FileDetails>();
 149:  public Dictionary<string, string> FieldData = new Dictionary<string, string>();
 150:  public int ID = -1;
 151:  public string Title = null;
 152: }
 153: #endregion

The syntax of the command can be seen below:

C:\>stsadm -help gl-exportlistitem2

stsadm -o gl-exportlistitem2

Exports list items to disk (exported results can be used with addlistitem).

Parameters:
        -url <list view url to export from>
        -path <export path>
        [-id <list item ID (separate multiple items with a comma)>]
Here's an example of how to do export list items:
stsadm -o gl-exportlistitem2 -url "http://intranet/documents/forms/allitems.aspx" -path "c:\documents"
Note that a "Data" folder will be created under the path specified - all files will be put in this folder and the folder structure will mirror that of the list. The Manifest.xml file will be in the root of the folder specified. Attachments will be stored in sub-folders using the name "item_{ID}" where {ID} is the item ID. Once exported you could then use the gl-addlistitem command to import these items to another list:
stsadm -o gl-addlistitem -url "http://intranet/documents2/forms/allitems.aspx" -datafile "c:\documents\manifest.xml" -publish
Update 1/31/2008: I've modified this command so that it now also supports exporting web part pages. The resultant exported manifest file can be used in conjunction with the gl-addlistitem command so that web part pages can be properly imported using that command.

Wednesday, December 12, 2007

Add List Item

I'm about at my wits end with the deployment API. I had a fairly simple requirement of adding a custom list item into the site directory so that the top tasks would have some additional items post upgrade. So I thought, no problem - I'll just create the list item manually in my test environment, export it with gl-exportlistitem and then use gl-importlistitem to import during the upgrade. Only problem was that because the field IDs of the list have changed the deployment API decided to recreate all the columns thus duplicating all of them and putting the data into the new column rather than the old so I couldn't even just delete the new columns without having to recreate the data. Also - I noticed that SP 1 just released yesterday - I only saw two fixes to the deployment stuff and neither addressed any of the dozen or so bugs that I've identified on this blog and reported to Microsoft - very frustrating. So my solution to this was to abandon the deployment API entirely and create a new command for importing list items: gl-addlistitem. I anticipate that I'll be expanding this more and more over time as requirements present themselves (I may even create a new export command that can work with this but so far I haven't needed to do so). The command itself is pretty simple - it offers two ways of adding data: the first is for simple stuff and allows you to just set field/value pairs via a single parameter; the second method allows you to specify an XML file thus allowing you to set more complex field values and add more than one item at a time as well as adding more than one attachment. Both methods support adding items to a document library or custom list. I'm currently not handling the case when the file already exists but I may allow for that in the future if necessary. The code itself is pretty straightforward as well. The only thing that tripped me up was creating new folders. This turned out to be easy but it was a trick finding the right place to create the folder - I thought I could just use list.Folders.Add() but that doesn't work - you have to use the following: SPListItem newFolder = list.Items.Add("", SPFileSystemObjectType.Folder, folderPath.Trim('/')); newFolder.Update(); Trick is that you have can't create "/SubFolder1/SubFolder2" if SubFolder1 doesn't already exist so you have to split the path and systematically create each folder in turn:

   1: /// <summary>
   2: /// Adds the item.
   3: /// </summary>
   4: /// <param name="web">The web.</param>
   5: /// <param name="list">The list.</param>
   6: /// <param name="itemData">The item data.</param>
   7: /// <param name="publish">if set to <c>true</c> [publish].</param>
   8: internal static void AddItem(SPWeb web, SPList list, List<ItemInfo> itemData, bool publish)
   9: {
  10:     bool itemsAdded = false;
  11:     foreach (ItemInfo itemInfo in itemData)
  12:     {
  13:         SPListItem item;
  14:  
  15:         SPFolder folder = GetFolder(web, list, itemInfo);
  16:         if (itemInfo.File == null)
  17:         {
  18:  
  19:             if (list.BaseType == SPBaseType.DocumentLibrary)
  20:             {
  21:                 string title = Guid.NewGuid().ToString();
  22:                 if (!string.IsNullOrEmpty(itemInfo.LeafName))
  23:                     title = itemInfo.LeafName;
  24:  
  25:                 SPFile file;
  26:                 if (itemInfo.Author == null)
  27:                     file = folder.Files.Add(title, new byte[] { });
  28:                 else
  29:                     file = folder.Files.Add(title, new byte[] {}, itemInfo.Author, itemInfo.Author, itemInfo.CreatedDate, DateTime.Now);
  30:  
  31:                 item = file.Item;
  32:             }
  33:             else
  34:                 item = list.Items.Add(folder.ServerRelativeUrl, SPFileSystemObjectType.File);
  35:         }
  36:         else
  37:         {
  38:             // We have a file so we need to handle it.
  39:             // Make sure the leaf and folder properties are set.
  40:             if (string.IsNullOrEmpty(itemInfo.LeafName))
  41:                 itemInfo.LeafName = itemInfo.File.Name;
  42:  
  43:             if (list.BaseType != SPBaseType.DocumentLibrary)
  44:             {
  45:                 // The list is not a document library so we'll add the file as an attachment.
  46:                 item = list.Items.Add(folder.ServerRelativeUrl, SPFileSystemObjectType.File);
  47:  
  48:                 item.Attachments.Add(itemInfo.LeafName, File.ReadAllBytes(itemInfo.File.FullName));
  49:             }
  50:             else
  51:             {
  52:                 // We've got the right folder so now add the file.
  53:                 SPFile file;
  54:                 if (itemInfo.Author == null)
  55:                     file = folder.Files.Add(folder.ServerRelativeUrl + "/" + itemInfo.LeafName,
  56:                                                File.ReadAllBytes(itemInfo.File.FullName));
  57:                 else
  58:                     file = folder.Files.Add(folder.ServerRelativeUrl + "/" + itemInfo.LeafName, File.ReadAllBytes(itemInfo.File.FullName), itemInfo.Author, itemInfo.Author, itemInfo.CreatedDate, DateTime.Now);
  59:  
  60:                 // Get the SPListItem object from the added file.
  61:                 item = file.Item;
  62:             }
  63:  
  64:         }
  65:         // Set the field values
  66:         foreach (string fieldName in itemInfo.FieldData.Keys)
  67:         {
  68:             SPField field = item.Fields.GetFieldByInternalName(fieldName);
  69:  
  70:             if (field.Type == SPFieldType.URL)
  71:                 item[fieldName] = new SPFieldUrlValue(itemInfo.FieldData[fieldName]);
  72:             else
  73:                 item[fieldName] = itemInfo.FieldData[fieldName];
  74:  
  75:         }
  76:  if (itemInfo.Author != null && item.Fields.ContainsField("Created By"))
  77:   item["Created By"] = itemInfo.Author;
  78:  
  79:  if (item.Fields.ContainsField("Created"))
  80:   item["Created"] = itemInfo.CreatedDate;
  81:         
  82:         item.Update();
  83:         
  84:         if (list.BaseType != SPBaseType.DocumentLibrary)
  85:         {
  86:             try
  87:             {
  88:                 // Add any attachments
  89:                 foreach (FileInfo att in itemInfo.Attachments)
  90:                 {
  91:                     item.Attachments.Add(att.Name, File.ReadAllBytes(att.FullName));
  92:                 }
  93:             }
  94:             catch (ArgumentException)
  95:             {
  96:                 throw new SPException("List does not support use of attachments.  Item added but not published.");
  97:             }
  98:         }
  99:         else if (itemInfo.Attachments.Count > 0)
 100:         {
 101:             throw new SPException("List does not support use of attachments.  Item added but not published.");
 102:         }
 103:         item.Update();
 104:  
 105:  
 106:         // Publish the changes
 107:         if (publish)
 108:         {
 109:             PublishItems.Settings settings = new PublishItems.Settings();
 110:             settings.Test = false;
 111:             settings.Quiet = true;
 112:             settings.LogFile = null;
 113:             PublishItems.PublishListItem(item, list, settings, "stsadm -o addlistitem");
 114:         }
 115:         itemsAdded = true;
 116:     }
 117:     if (itemsAdded)
 118:         list.Update();
 119: }
 120:  
 121: /// <summary>
 122: /// Gets the folder.
 123: /// </summary>
 124: /// <param name="web">The web.</param>
 125: /// <param name="list">The list.</param>
 126: /// <param name="itemInfo">The item info.</param>
 127: /// <returns></returns>
 128: internal static SPFolder GetFolder(SPWeb web, SPList list, ItemInfo itemInfo)
 129: {
 130:     if (string.IsNullOrEmpty(itemInfo.FolderUrl))
 131:         return list.RootFolder;
 132:  
 133:     SPFolder folder = web.GetFolder(list.RootFolder.Url + "/" + itemInfo.FolderUrl);
 134:  
 135:     if (!folder.Exists)
 136:     {
 137:         if (!list.EnableFolderCreation)
 138:         {
 139:             list.EnableFolderCreation = true;
 140:             list.Update();
 141:         }
 142:  
 143:         // We couldn't find the folder so create it
 144:         string[] folders = itemInfo.FolderUrl.Trim('/').Split('/');
 145:  
 146:         string folderPath = string.Empty;
 147:         for (int i = 0; i < folders.Length; i++)
 148:         {
 149:             folderPath += "/" + folders[i];
 150:             folder = web.GetFolder(list.RootFolder.Url + folderPath);
 151:             if (!folder.Exists)
 152:             {
 153:                 SPListItem newFolder = list.Items.Add("", SPFileSystemObjectType.Folder, folderPath.Trim('/'));
 154:                 newFolder.Update();
 155:                 folder = newFolder.Folder;
 156:             }
 157:         }
 158:     }
 159:     // Still no folder so error out
 160:     if (folder == null)
 161:         throw new SPException(string.Format("The folder '{0}' could not be found.", itemInfo.FolderUrl));
 162:     return folder;
 163: }

The syntax of the command can be seen below:

C:\>stsadm -help gl-addlistitem

stsadm -o gl-addlistitem

Adds a list item or items to an existing list.

Parameters:
        -url <list view url to import into>
        [-filename <file to add to the list (will add as attachment if not document library)>]
        [-leafname <name to give the file if specified (be sure to include the file extension)>]
        [-folderurl <url to place the specified file if not at the root (list relative)>]
        {[-fielddata <semi-colon separated list of key value pairs: "Field1=Val1;Field2=Val2"> (use ';;' to escape semi-colons in data values)] |
         [-datafile <path to a file with xml settings matching the following format: <Items><Item File="{optional file path}" FolderUrl="{optional folder url}" LeafName="{optional display name of file}" Author="{optional username}" CreatedDate="{optional create date}"><Fields><Field Name="Name1">Value1</Field><Field Name="Name2">Value2</Field></Fields><Attachments><Attachment>{path to file}</Attachment></Attachments></Item></Items> >]}
        [-publish]
Here's an example of how to do add a single item to a site directory:
stsadm -o gl-addlistitem -url "http://intranet/sitedirectory/siteslist/allitems.aspx" -fielddata "Title=Sales Site;URL=http://intranet/sales, Sales Home;Region=Local;Division=Sales;TopSite=true" -publish
To do the same thing but this time also add an attachment you would do the following:
stsadm -o gl-addlistitem -url "http://intranet/sitedirectory/siteslist/allitems.aspx" -filename "c:\sales\MissionStatement.txt" -fielddata "Title=Sales Site;URL=http://intranet/sales, Sales Home;Region=Local;Division=Sales;TopSite=true" -publish
To add a file to a document library you would do this:
stsadm -o gl-addlistitem -url "http://intranet/documents/forms/allitems.aspx" -filename "c:\file1.doc" -leafname "Test File 1.doc" -fielddata "Title=Title1;ImportDate=12/12/2007" -publish
To do the same thing using a file and at the same time add more than one item you'd do this:
stsadm -o gl-addlistitem -url "http://intranet/documents/forms/allitems.aspx" -datafile "c:\items.xml" -publish
Here's the XML that was provided:
<Items>
    <Item File="c:\file1.doc" LeafName="Test File 1.doc" Author="domain\user" CreatedDate="12/18/2007 11:55 AM">
        <Fields>
            <Field Name="Title">Title1</Field>
            <Field Name="ImportDate">12/12/2007</Field>
        </Fields>
    </Item>
    <Item File="c:\file2.xml" FolderUrl="Documents/SubFolder1/SubFolder2">
        <Fields>
            <Field Name="Title">Title2</Field>
            <Field Name="ImportDate">12/12/2007</Field>
        </Fields>
    </Item>
</Items>
To add items to a custom list (not a document library) you could use the following:
stsadm -o gl-addlistitem -url "http://intranet/TestList/allitems.aspx" -datafile "c:\items.xml" -publish

Here's the XML that was provided:

<Items>
    <Item>
        <Fields>
            <Field Name="Title">Title1</Field>
            <Field Name="ImportDate">12/12/2007</Field>
        </Fields>
        <Attachments>   
            <Attachment>c:\file1.doc</Attachment>
            <Attachment>c:\file2.doc</Attachment>
        </Attachments>
    </Item>
    <Item FolderUrl="TestList/SubFolder1/SubFolder2">
        <Fields>
            <Field Name="Title">Title2</Field>
            <Field Name="ImportDate">12/12/2007</Field>
        </Fields>
    </Item>
 </Items>

Update 12/18/2007: I've updated this command so that it now also supports the setting of an Author and CreatedDate attribute when using XML as the data source. I've also fixed a bug with how folders were being created (previously they'd get created but were not visible via the browser). I've also fixed some other issues when dealing with different types of input data and target list types. Note also that this command is compatible with the exportlistitem2 command. All content above has been updated to reflect these changes. Update 1/31/2008: I've modified this command so that it now also supports importing web part pages. You can use the exportlistitem2 command to export pages and then use the resultant exported manifest file in conjunction with the gl-addlistitem command so that web part pages can be properly imported.