I’m currently working for a customer that has an Orchestrator runbook that should always be in a running state.
It monitors whether a certain type of Incident Request has been created in SCSM and if so, it sends out an email message.
If this particular runbooks stops, then no more email notifications are sent out for this type of IRs.
The Orchestrator database resides on a cluster of which frequent failovers happen.
Sometimes during such a failover (when a failover takes too long) Orchestrator looses its connection to the SQL instance and the runbook is stopped.
The customer only notices that the runbook has stopped when users start complaining that they no longer receive email notifications for new Incident Requests.
Since they have SCOM deployed as well, such reactive “monitoring” is of course a big no no
Therefor I thought to build them a script based monitor using Silects MPAuthor, which will notify us if the runbook stops.
In the following steps, I will show how I created the monitor and which scripts have been used to query the Orchestrator webservice.
First, I needed to write the scripts needed to query the webservice.
The result will be placed inside a SCOM propertybag so that the Health service can receive the result:
This is what I finally came up with:
$API = New-Object -ComObject “MOM.ScriptAPI”
$PropertyBag = $API.CreatePropertyBag()
$user = “<<Serviceaccount>>”
$pass = ConvertTo-SecureString “<<Password>>” -AsPlainText -Force
$creds = New-Object System.Management.Automation.PsCredential($user,$pass)
$url = >:81/Orchestrator2012/Orchestrator.svc/Jobs()?
$expand=Runbook&$filter=(Status eq ‘Running’) and (Runbook/Name eq ‘>’)&
$filter=(Status eq 'Running') and (Runbook/Name eq '<<RunbookName>>')&$select=Runbook/Name,Status
$request = [System.Net.HttpWebRequest]::Create($url)
$request.Credentials = $creds
$request.Timeout = 120000
$request.ContentType = “application/atom+xml,application/xml”
$request.Method = “GET”
$response = $request.GetResponse()
$requestStream = $response.GetResponseStream()
$readStream=new-object System.IO.StreamReader $requestStream
$Output = $readStream.ReadToEnd()
$Result=$Output -match “<d:Status>Running</d:Status>”
Once we have the script, we can almost begin building our monitor in Silect but first we need to do some more preparing on the server which will execute the script.
As you know, a monitor runs on specific targets.
Within MPAuthor, we will also be asked against which target we will run our monitor.
A target can be defined in several ways. I like to create a custom registry value on our monitor target.
As I want to run the monitor locally on our Orchestrator server, I went ahead and created the following key:
With this in place, we can now open up MPAuthor and start the madness
Create a Blank Management Pack
1. Give your MP a logical name
2. Specify where to store it
3. Accept the default referenced other MPs
4. Select to create an empty Management Pack
5. On the Create page, review your settings and click the create button
Create the Discovery rule and Target
1. Right-click the Discoveries node and select a new registry target
2. Select the server where you want to browse towards to lookup the custom the registry value which we created earlier
3. In the browse windows select our ‘SCORWatcherNode’
4. Once select, it should look like this
5. Give your target a logical name
6. Give your discovery a logical name
7. Select the Windows Computer Class as your target base class
8. On the expression page , accept the default expression
9. On the discovery schedule, enter which schedule suits your needs best.
I choose 4 hours
10. On the Create page, review your settings and click the create button
11. Once completed, you should now see your Discovery and your Target
Create the Monitor
1. Right-click the Monitors pane and select create newscript monitor
2. On the Script page, enter a name for the script and paste in the script which was mentioned above
3. The second screen can be a bit confusing.
It starts off with a 3-state monitor.
In this case, we only need a 2 state monitor, therefor only keep the UnderWarning and OverError states (these statenames can be customized as well, I kept them at defaults for this example).
You have to click each state and modify the criteria for the selected monitor.
The Propertyname should match the propertyname you entered in the propertybag of the powershell script.
As we have a 2 statesmonitor, we should do this for both states
4. Now select to target our script monitor against all instances discovered by our newly created target. Parent monitor should be Availability
5. Give your monitor a logical name
6. On the Scedule page, select a schedule that suits your needs.
I’ve choosen to run our monitor every 10 minutes
7. On the Alert page, setup the way you want alerts to be generated.
8. On the create page, click the Create button
9. We now can see the monitor has been saved in the MP
Importing the MP into SCOM
From MPAuthor, we can import the newly created MP directly into SCOM but for this example I’ve imported it manually
1. Open the SCOM console and navigate to the administration pane
2. Right click the Management Packs node and select Import
3. Select the xml file for the management pack which we just created and hit the import button. After a while, the following should be shown
4. Next up, I connected to the Operations Manager eventlog of the orchestrator server to see if the MP got downloaded.
After a while, I saw that so far everything was going smoothly
5. Now that the MP has arrived at the server which will run the monitor, we have to wait until the discovery has picked up our registry key which indicates the target for our monitor. To follow up on this process, you can go back into the monitoring pane and change the scope to point to our Target class. After 5 to 10 minutes, our Target should appear
6. If we then open up the health explorer, we will then see our monitor and in this case it has already gone into a monitored state
7. To test the monitor, I stopped the runbook and after a while it went into error (as expected) and place our server into red
8. If we go into the alerts view, then we also see the alert
And that’s it.
That’s how easy it is to monitor the state of an orchestrator runbook using scom and some MPAuthoring magic!
As a follow-up article, I’ll show you how you can define a recovery task which will call an orchestrator runbook to rectify the problem and to restart the runbook.
Please feel free to re-use this logic for any kind of script monitor you need.
If you got any questions/remarks, then don’t hesitate to contact me
Hope it helps