MOSS MVP

I've moved my blog to http://blog.falchionconsulting.com!. Please update your links. This blog is no longer in use--you can find all posts and comments at my new blog; I will no longer be posting to this site and comments have been disabled.

Saturday, October 25, 2008

A Better execadmsvcjobs STSADM Command

This is something that's been bugging me for a long time - when you run the out of the box execadmsvcjobs command on a server it only ensures that pending jobs on that one server are executed - when it completes it doesn't mean that jobs on other servers in the farm have completed.  This gets real annoying when you are using a script to deploy solution because end up getting errors about pending timer jobs needing to complete.

I tried a couple of different approaches to address this problem - the first was to use WMI to execute the execadmsvcjobs command remotely on each server.  Problem with this approach is that for some reason the security context kept getting to changed to "NT AUTHORITY\ANONYMOUS LOGON" even though the process showed that it was running as my executing account - never figured out what the heck was going on with that so I decided to try a different approach.  The next thing I tried was to reverse engineer the out of the box command and change it to execute all jobs for each server, not just the local server.  This appeared to work but upon further inspection it became clear that it wasn't working at all - there's definitely something going on that gets whacked out when executing this way - so I was left with trying to find another approach.

What I eventually ended up with was a simple command that leveraged what I had done while trying to recreate the out of the box execadmsvcjobs command but instead of executing the job on each server it simply blocks until the jobs have all completed.  It's not exactly what I wanted but the end result is the same - the command blocks my script until the pending jobs have finished on each server thus allowing my subsequent commands to run without error.  The name of this new command is gl-execadmsvcjobs.

   1: using System;
   2: using System.Collections.Generic;
   3: using System.Text;
   4: using Lapointe.SharePoint.STSADM.Commands.OperationHelpers;
   5: using Microsoft.SharePoint;
   6: using Microsoft.SharePoint.Administration;
   7: using System.Threading;
   8:  
   9: namespace Lapointe.SharePoint.STSADM.Commands.TimerJob
  10: {
  11:     public class ExecAdmSvcJobs : SPOperation
  12:     {
  13:         /// <summary>
  14:         /// Initializes a new instance of the <see cref="ExecAdmSvcJobs"/> class.
  15:         /// </summary>
  16:         public ExecAdmSvcJobs()
  17:         {
  18:             SPParamCollection parameters = new SPParamCollection();
  19:             parameters.Add(new SPParam("local", "l"));
  20:  
  21:             StringBuilder sb = new StringBuilder();
  22:             sb.Append("\r\n\r\nExecutes pending timer jobs on all servers in the farm.\r\n\r\n\r\n\r\nParameters:");
  23:             sb.Append("\r\n\t[-local]");
  24:             Init(parameters, sb.ToString());
  25:         }
  26:  
  27:         /// <summary>
  28:         /// Gets the help message.
  29:         /// </summary>
  30:         /// <param name="command">The command.</param>
  31:         /// <returns></returns>
  32:         public override string GetHelpMessage(string command)
  33:         {
  34:             return HelpMessage;
  35:         }
  36:  
  37:         /// <summary>
  38:         /// Executes the specified command.
  39:         /// </summary>
  40:         /// <param name="command">The command.</param>
  41:         /// <param name="keyValues">The key values.</param>
  42:         /// <param name="output">The output.</param>
  43:         /// <returns></returns>
  44:         public override int Execute(string command, System.Collections.Specialized.StringDictionary keyValues, out string output)
  45:         {
  46:             output = string.Empty;
  47:  
  48:             Execute(Params["local"].UserTypedIn);
  49:  
  50:             return OUTPUT_SUCCESS;
  51:         }
  52:  
  53:         /// <summary>
  54:         /// Executes the timer jobs.
  55:         /// </summary>
  56:         /// <param name="local">if set to <c>true</c> [local].</param>
  57:         public static void Execute(bool local)
  58:         {
  59:             Execute(local, false);
  60:         }
  61:  
  62:         /// <summary>
  63:         /// Executes the timer jobs.
  64:         /// </summary>
  65:         /// <param name="local">if set to <c>true</c> [local].</param>
  66:         /// <param name="quiet">if set to <c>true</c> [quiet].</param>
  67:         public static void Execute(bool local, bool quiet)
  68:         {
  69:             // First run the OOTB execadmsvcjobs on the local machine to make sure that any local jobs get executed
  70:             if (!quiet)
  71:                 Console.WriteLine("\r\nExecuting jobs on {0}", SPServer.Local.Name);
  72:  
  73:             Utilities.RunStsAdmOperation("-o execadmsvcjobs", quiet);
  74:             // If local was passed in then we're basically just using the OOTB command - I included this for testing only - it's not
  75:             // really helpful otherwise.
  76:             if (!local)
  77:             {
  78:                 foreach (SPServer server in SPFarm.Local.Servers)
  79:                 {
  80:                     // Only look at servers with a valid role.
  81:                     if (server.Role == SPServerRole.Invalid)
  82:                         continue;
  83:  
  84:                     // Don't need to check locally as we just ran the OOTB command locally so skip the local server.
  85:                     if (server.Id.Equals(SPServer.Local.Id))
  86:                         continue;
  87:  
  88:                     bool stillExecuting;
  89:                     if (!quiet)
  90:                         Console.WriteLine("\r\nChecking jobs on {0}", server.Name);
  91:  
  92:                     do
  93:                     {
  94:                         stillExecuting = CheckApplicableRunningJobs(server, quiet);
  95:  
  96:                         // If jobs are still executing then sleep for 1 second.
  97:                         if (stillExecuting)
  98:                             Thread.Sleep(1000);
  99:                     } while (stillExecuting);
 100:                 }
 101:             }
 102:         }
 103:         /// <summary>
 104:         /// Checks for applicable running jobs.
 105:         /// </summary>
 106:         /// <param name="server">The server.</param>
 107:         /// <param name="quiet">if set to <c>true</c> [quiet].</param>
 108:         /// <returns></returns>
 109:         private static bool CheckApplicableRunningJobs(SPServer server, bool quiet)
 110:         {
 111:             foreach (KeyValuePair<Guid, SPService> current in GetProvisionedServices(server))
 112:             {
 113:                 SPService service = current.Value;
 114:                 SPAdministrationServiceJobDefinitionCollection definitions = new SPAdministrationServiceJobDefinitionCollection(service);
 115:                 if (CheckApplicableRunningJobs(server, definitions, quiet))
 116:                     return true; // We've found running jobs so no point looking any further.
 117:  
 118:                 SPWebService service2 = service as SPWebService;
 119:                 if (service2 != null)
 120:                 {
 121:                     foreach (SPWebApplication webApplication in service2.WebApplications)
 122:                     {
 123:                         definitions = new SPAdministrationServiceJobDefinitionCollection(webApplication);
 124:                         if (CheckApplicableRunningJobs(server, definitions, quiet))
 125:                             return true;
 126:                     }
 127:                 }
 128:             }
 129:             return false;
 130:         }
 131:  
 132:         /// <summary>
 133:         /// Checks for applicable running jobs.
 134:         /// </summary>
 135:         /// <param name="server">The server.</param>
 136:         /// <param name="jds">The job definitions to consider.</param>
 137:         /// <param name="quiet">if set to <c>true</c> [quiet].</param>
 138:         /// <returns></returns>
 139:         private static bool CheckApplicableRunningJobs(SPServer server, SPAdministrationServiceJobDefinitionCollection jds, bool quiet)
 140:         {
 141:             bool stillExecuting = false;
 142:  
 143:             foreach (SPJobDefinition definition in jds)
 144:             {
 145:                 if (string.IsNullOrEmpty(definition.Name))
 146:                     continue;
 147:  
 148:                 bool isApplicable = false;
 149:                 if (!definition.IsDisabled)
 150:                     isApplicable = ((definition.Server == null) || definition.Server.Id.Equals(server.Id));
 151:  
 152:                 if (!isApplicable)
 153:                 {
 154:                     // If it's not applicable then we don't really care if it's running or not.
 155:                     continue;
 156:                 }
 157:                 
 158:                 if (!quiet)
 159:                     Console.Write("Waiting on {0}.\r\n", definition.Name);
 160:  
 161:                 stillExecuting = true;
 162:             }
 163:             return stillExecuting;
 164:         }
 165:  
 166:  
 167:         /// <summary>
 168:         /// Gets the provisioned services.
 169:         /// </summary>
 170:         /// <param name="server">The server.</param>
 171:         /// <returns></returns>
 172:         private static Dictionary<Guid, SPService> GetProvisionedServices(SPServer server)
 173:         {
 174:             Dictionary<Guid, SPService> dictionary = new Dictionary<Guid, SPService>(8);
 175:             foreach (SPServiceInstance serviceInstance in server.ServiceInstances)
 176:             {
 177:                 SPService service = serviceInstance.Service;
 178:                 if (serviceInstance.Status == SPObjectStatus.Online)
 179:                 {
 180:                     if (dictionary.ContainsKey(service.Id))
 181:                         continue;
 182:                     dictionary.Add(service.Id, service);
 183:                 }
 184:             }
 185:             return dictionary;
 186:  
 187:         }
 188:  
 189:         /// <summary>
 190:         /// This class mimics the internal equivalent and is used because the base class is abstract.
 191:         /// </summary>
 192:         internal class SPAdministrationServiceJobDefinitionCollection : SPPersistedChildCollection<SPAdministrationServiceJobDefinition>
 193:         {
 194:             /// <summary>
 195:             /// Initializes a new instance of the <see cref="SPAdministrationServiceJobDefinitionCollection"/> class.
 196:             /// </summary>
 197:             /// <param name="service">The service.</param>
 198:             internal SPAdministrationServiceJobDefinitionCollection(SPService service) : base(service)
 199:             {
 200:             }
 201:  
 202:             /// <summary>
 203:             /// Initializes a new instance of the <see cref="SPAdministrationServiceJobDefinitionCollection"/> class.
 204:             /// </summary>
 205:             /// <param name="webApplication">The web application.</param>
 206:             internal SPAdministrationServiceJobDefinitionCollection(SPWebApplication webApplication) : base(webApplication)
 207:             {
 208:             }
 209:         }
 210:  
 211:     }
 212: }

The help for the command is shown below:

C:\>stsadm -help gl-execadmsvcjobs

stsadm -o gl-execadmsvcjobs

Executes pending timer jobs on all servers in the farm.


Parameters:
        [-local]

The following table summarizes the command and its various parameters:

Command Name Availability Build Date
gl-execadmsvcjobs WSS v3, MOSS 2007 Released: 10/25/2008

Parameter Name Short Form Required Description Example Usage
local l No If passed in then do not consider other servers in the farm - this basically just treats the command exactly as the out of the box execadmsvcjobs command (in fact it just calls out to that command). -local

-l

The following is an example of how to make sure that all pending timer jobs have run on all servers in the farm:

stsadm -o gl-execadmsvcjobs

5 comments:

Neil J Thomson said...

Very helpful. Had been using the SPAdmin APIs to get jobs to execute and was unable to reliably detect when jobs were finished. Kept getting (can't remove solution as jobs still running) exeption.

Used the approach (outlined here) of having the STSADMIN command line version to ensure execution completed and voila! Problem solved.

Nicely explained and commented solution. My compliments.

Neil J Thomson said...

Developing a sharepoint web part installer using the SP APIs and had failures using the suggested API approach of executing timer jobs.

However, even with getting the list of timer jobs and looking through them to see which solution-deployment jobs were still running, would get 50% + exceptions after executing retract that remove solution could not be done as jobs still running.

I'm guessing that the running jobs list wasn't accurate, but from looking at the solutions outlined here which use reflection to get at internal classes not published by MS, I'm guessing that the public SP Admin APIs aren't 100% capable in this area.

Using this description and the downloaded project, I found that programmatically calling the STSADMIN command line utility to execjobs, which reliably solved the problem.

My compliments on a clear description and code to resolve the problem.

tripwire said...

Hi Gary,

I ran this command from my CA server as a way to see if the timer jobs were actually running on all boxes. As it turns out it got stuck in a loop when it hit the next server and confirmed my suspicions.

I then went to the box in question and ran the standard execsvcadmjobs command. It completed successfully with the following:

Executing .
Executing job-watson-trigger.
Executing job-antivirus.
Operation completed successfully.

However when I went back to the other server your command was still displaying:

Waiting on job-antivirus.
Waiting on job-watson-trigger.

Could this be permission related? I ran the command under the context of the SharePoint Setup account which is the same user applied to the WSS Timer Service.

Chints said...

Hi Gary,

Very good stuffs. But could you please show us where to integrate above .Net code?

Thanks,

Gary Lapointe said...

Go to the downloads page and download the WSP for your environment. Follow the install instructions (there's nothing you need to do with the code - it's there only if you wish to see the what and the how of the command).