You are browsing the archive for 2012 September.

Orchestrator 2012 : Monitor SNMP Trap activity affected by trap content

11:07 am in Orchestrator by Christopher Keyaert

Hi

During a migration project from Opalis 6.3 to Orchestrator 2012, I had to migrate a runbook that was using the Monitor SNMP Trap activity. The idea of this runbook is to receive a trap that is composed of two OIDs, first one contains the target name and the second one contains the description and to raise an alert in Operations Manager (SCOM) based on there values.

image

image

In Opalis, as I explained here : http://scug.be/christopher/2011/06/17/system-center-opalis-monitor-snmp-trap-activity/ this configuration is working. Now, it’s time to migrate it ! As this is a quite simple runbook, I just did a runbook export in Opalis, an import in Orchestrator and started the runbook.

To test the good working of my newly imported runbook, I sent several SNMP traps to my orchestrator server. I noticed that the Monitor SNMP Trap activity catch the trap, published the result correctly, but when I checked the content of the trap, the result was really not what I was expecting. Let’s me explain what I did in detail.

I used the command line application SNMPTrapGen ( http://www.snmpsoft.com/freetools/snmptrapgen.html ) , to send a SNMP trap to the Orchestrator server. Below the content of my SNMP Trap :

As my trap contained the string ‘azerty’, I was expecting to receive the same value from the Monitor SNMP Trap activity in Orchestrator, but I received the following value :

I did several tests and every time that I sent a SNMP Trap that was containing a string to Orchestrator, the Monitor SNMP Trap activity published a suite of numbers, which was really not the contain that I was expecting. I also did a network capture on the Orchestrator to check if the trap content was correct and yes, it was.

As it was not working when I sent a string value, I tried to sent an integer value to Orchestrator :

and there, Orchestrator returned the right value.

After all my tests, I observed that the Monitor SNMP Trap activity always returned a suite of number when the SNMP trap was containing a string value. In that project, It’s what I need to do, I have to pass a string value to this activity. As this was working perfectly with Opalis and not anymore with Orchestrator, I continued my investigation and I focused on the Monitor SNMP Trap activity itself.

By default, this activity  use the Microsoft SNMP Trap Services :

image

I decided to choose the “No dependency’” connection. For that configuration, the first step is to stop and disable the SNMP Trap service on the Orchestrator server.

image

I changed the connection setting of the Monitor SNMP Trap activity and I started my runbook again.

clip_image002[5]

I sent a new trap, which was containing a string value, to my Orchestrator with the SNMPTrapGen command line..

And this time, when I checked the value returned by the Monitor SNMP Trap activity, it was correct Smile

Yes, We have a solution to get this working ! I don’t know exactly why this was working correctly with the Microsoft SNMP Trap Service in Opalis 6.3 and not anymore with the same service in Orchestrator 2012. If you have me more information about it, please contact me.

As I changed the Monitor SNMP Trap activity connection to No Dependency, we are now limiting to run only one instance of the Monitor SNMP Trap activity on this Orchestrator server (on the same port),  which was not the case by using the Microsoft SNMP Trap Service connection. By changing the connection type, this activity is taking the control on the defined port and it seems logical that it cannot be shared with another activity.

Now that I can only use one Monitor SNMP Trap activity, what can I do if I need to receive SNMP Traps from several locations  ? Well, You will have to redesign your runbooks to get only one entry point for all the SNMP traps :

image

Configured the Monitor SNMP Trap activity to get and publish all the different OIDs that you will need .

image

On the link between the Monitor SNMP Trap and the Invoke Runbook activities, apply a filter based on the SNMP Trap sender IP address.

image

You have to define all the published OIDs value that you need to pass to your Invoke Runbook activity.

image

And finally pass these information to your original runbook by replacing the Monitor SNMP Trap activity with an Initialize Data activity .

image

I hope this post could help you to configure the SNMP Trap monitor with Orchestrator 2012.

Christopher Keyaert

SCOM 2007 R2 : Configuration not loaded…again

12:56 pm in Operations Manager by Christopher Keyaert

Hi Guys,

If you follow my blog, we probably already read my previous post about : SCOM 2007 R2 : Configuration not loaded
Two days after this issue was resolved, my customer had again the exact same symptoms… First thing that I did, is to check the BaseManagedEntity table and  this time no inconsistency was found, everything seems to be ok.

After some search, I found that KB : Configuration may not update in System Center Operations Manager http://support.microsoft.com/kb/2635742/EN-US

The System Center Management Configuration service uses a timestamp to determine when new configuration data needs to be calculated for agents and management servers.  If the system clock on an agent is faster than the system clock on the Management Server, discovery data from this agent will set the timestamp for one or more managed instances hosted by that agent to the current agent system clock time.  The System Center Configuration Management service will delay calculating configuration updates for the instances on that agent until the system clock on the Management Server is current with the timestamp for that discovery data.  If the agent system clock was significantly faster than Management Server system time when discovery data was sent, or the agent continues to send data with a future timestamp, then it is possible that the management group would experience the symptoms listed above.

Setting the agent system clock time to match the Management Server system clock time will not reset the timestamp for the existing discovery data and the issue will remain until the Management Server system clock time exceeds the discovery data by the grooming interval, when the obsolete discovery data will be groomed normally.

Let check if we are in that case or not. The first thing to do is to run the following 3 queries :

Capture3

As we could see, there is a difference of 3 hours 30 min between the current time and some data inserted in the DB. “We’ve got data from the future” Open-mouthed smile. We have now to identify which computer is causing that problem in our environment.

For that, we have to run 3 new queries : (I modified the queries to return the computer that has inserted data with more than 1 hour in advance.)

Capture4

This second query didn’t return any on environment. we will now run the last query :

Capture5

It returned the exact same server that we had with the first query. Let’s have a quick look on that server. On the left the impacted server, on the right the Domain Controller located on the same site.

Capture

As you could see the time on the server is not correct. We have now to correct the time on the server, and also the existing data in the DB. For the time on the server, just modify it from the user interface, for the data in the data, we will have to run the following commands against the affected tables.

All data are now at the right time, a restart of the 3 SCOM services on the RMS should fix the problem.

Cheers

Christopher