You are browsing the archive for Operations Manager.

SCOM 2007 : The error is 0x80FF0066(0x80FF0066).

3:57 pm in Uncategorized by Christopher Keyaert

Hi Guys,

First day back to work after the Xmas holidays and first error of the year Open-mouthed smile
Below 3 events that I found on a customer server, that was not reporting to Operations Manager anymore.

Event Type:    Error
Event Source:    HealthService
Event Category:    Health Service
Event ID:    1220
Date:        1/3/2012
Time:        4:22:36 PM
User:        N/A
Computer:    xxxx
Description:
Received configuration cannot be processed. Management group "xxxx". The error is 0x80FF0066(0x80FF0066).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

image

Event Type:    Error
Event Source:    HealthService
Event Category:    Health Service
Event ID:    7022
Date:        1/3/2012
Time:        4:22:35 PM
User:        N/A
Computer:    xxxx
Description:
The Health Service has downloaded secure configuration for management group xxxx, and processing the configuration failed with error code 0x80FF0066(0x80FF0066).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

image

Event Type:    Warning
Event Source:    HealthService
Event Category:    Health Service
Event ID:    7004
Date:        1/3/2012
Time:        4:22:35 PM
User:        N/A
Computer:    xxxx
Description:
The Health Service received a secure message from management group xxxx which was encrypted using the wrong public key.  This message has been discarded and the public key will be re-published.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

image

The root cause is a missing registry key on the server. Take a look to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\UCBOM2007\SSDB\References

On the non working server :
image

On a working server :
image

When you compare the both servers, you notice that the following registry key is missing :

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\xxxx\SSDB\References\92f8f803-0763-f491-2480-274bfc4126f9]
@="01020202020202020202020202020202020202020200000000000000000000000000000000000000"

Just import the missing registry key from the working server to the non working server, and after a few seconds, the event below will appear, problem fixed Smile

Event Type:    Information
Event Source:    OpsMgr Connector
Event Category:    None
Event ID:    21025
Date:        1/3/2012
Time:        4:28:37 PM
User:        N/A
Computer:    xxxx
Description:
OpsMgr has received new configuration for management group xxxx from the Configuration Service.  The new state cookie is "24 58 21 8C 7C D4 70 68 B2 87 1A 52 EA BE 7F 66 43 10 FC A7 "

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

image

If you want more information about this issue, I invite you to read the two followings posts :

http://thoughtsonopsmgr.blogspot.com/2009/02/eventid-7024.html
http://www.systemcentercentral.com/tabid/60/indexId/34058/tag/Forums+Operations_Manager/Default.aspx#vindex56241

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

SCOM2007 : New Server OS MP, version 6.0.6957.0 has some bad bugs

9:16 am in Uncategorized by Christopher Keyaert

Hi All,

As you probably already noticed, Microsoft has released some days ago the new New Base OS MP 6.0.6957.0.

This management pack adds a lot of new cool functionalities that Kevin Holman explained on his blog : http://blogs.technet.com/b/kevinholman/archive/2011/09/30/opsmgr-new-base-os-mp-6-0-6956-0-adds-cluster-shared-volume-monitoring-bpa-and-many-changes.aspx

Now after several days, we’re starting to notice that this new management pack has also some “bugs” :

1: Report execution might fail because of lacking proper security settings on a Stored Procedure;
2: Importing the MP might fail when one has many overrides in place for diskspace. All because of a spelling error;
3: Knowledge is out of date for the new default values in the free space monitors;
4: The BPA monitors can be noisy for Server 2008R2 systems;
5: The “performance by utilization” report section dealing with Logical Disk % Idle time is upside down: the lowest idle time values are on top (100% Idle time is the lowest) and the highest idle time (anything close to 10% or even less) are on the bottom.
6: Impossible to deploy the new reports on SQL2005

Marnix Wolf and Bob Cornelissen started to report these bugs on their own blogs :

http://thoughtsonopsmgr.blogspot.com/2011/10/more-than-one-lets-ask-for-new-improved.html
http://www.bictt.com/blogs/bictt.php/2011/10/04/asking-for-a-quick-fix

Marnix also took the really good initiative to open a bug report on connect :

image

If you also noticed these bugs, please go to the link below and share your experience :

https://connect.microsoft.com/OpsMgr/feedback/details/692944/new-server-os-mp-version-6-0-6956-0-has-some-bad-bugs

This is not a flame or something like that, just an initiative to get a fixed and working MP very soon with all the great new stuff that was added and without at least most of the issues!

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

SCOM2007 : Performance Degradation / Maintenance Mode Scheduling Tool

12:58 pm in Uncategorized by Christopher Keyaert

Hi Everyone,

Last week, I’ve been called by one of my customer, who told me that they saw a huge performance degradation of their SCOM2007 R2 environment.

This customer has a quite big infrastructure, around 2200 servers, using Nworks MP, NetApp MP, Oracle Enterprise Manager connector, Opalis 6.3, Scheduling maintenance mode ….. This environment was working well since a long time and no major modification has been done so far.

The symptoms :

  • Taking more 10 min for connecting to the console.
  • Console navigation extremely slow or not responding.
  • High CPU usage on the RMS.
  • Very high CPU usage on the DB Cluster.
  • Connection lost time to time to the DBs

After we checked everything we could, it was time to call the Microsoft Premier Support. After several days of investigation with them, we didn’t identity clearly the root cause. so, It was time to turn off one by one each connectors, each third party software that were connecting to Operations Manager.

It took time but we finally found which application was causing our nightmare since several days…. it was the Maintenance Mode Scheduling Tool.

This tool has been released part of the Administration Resource Kit for System Center Operations Manager 2007 R2 published by Operations Manager Product Team some months ago. More information available here : http://blogs.technet.com/b/jimmyharper/archive/2011/07/14/maintenance-mode-scheduling-tool.aspx

This is really an excellent tool that provide you an user interface for scheduling maintenance mode of your servers. My customer was using it a lot for disabling the monitoring during batch processing, planned reboot and so on.

Since we turned it off by disabling the rule Maintenance Mode Workflow, the Operations Manager infrastructure went back to business. So I continued to investigate on why this tools was causing all our troubles and I finally found this post : http://blogs.technet.com/b/momteam/archive/2011/06/21/schedule-maintenance-mode-reskit-tool-info.aspx

Best Practices:

  • Do not go beyond 20 Jobs scheduled.  Anything over this will start to place too much load on your Root Management Server
  • Do not schedule more then 20 items in one Job.  If you need to go over this please create a group and target this.  Note: The MP has a bug that only limits you to select up to six objects.  I have attached an updated MP that corrects this issue below.
  • When scheduling a group make sure to select system.group then select the group.  If you select the group itself the tool lists the individual group membership.

My customer had more than 40 Jobs scheduled…. so we were clearly not respecting the best practice and that was explaining why our performance degradation came slowly day per day.

Currently, the only solution that we have for scheduling maintenance mode without using that’s tool is to come back to a PowerShell script that we will run from a windows schedule task.

Resources :
System Center Operations Manager 2007 R2 Evaluation : http://technet.microsoft.com/en-us/systemcenter/om/bb498233

Technet Forums : http://social.technet.microsoft.com/Forums/en-US/category/systemcenteroperationsmanager

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

Best of MMS 2011 – Operations Manager 2012: Overview, Setup and Configuration

9:24 am in Uncategorized by Christopher Keyaert

image001

This session will provide you with an overview of the capabilities in OpsMgr 2012. With Operations Manager 2012 public beta, this session will get you started by explaining the new OpsMgr 2012 configuration – without an RMS!! We’ll also look at configuring high availability and pooling of management servers, as well as other new concepts to be sure that you are ready to get the most out of the upcoming beta.

Watch the Video : Operations Manager 2012: Overview, Setup and Configuration

Resources :
System Center Operations Manager 2012 Beta : Installation step by step
System Center Operations Manager 2012 Beta : Available for download

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

System Center Operations Manager 2012 Beta : Installation step by step

8:47 am in Uncategorized by Christopher Keyaert

As you are already aware, System Center Operations Manager 2012 Beta is now available for public download. Smile The download link is available in my previous post : System Center Operations Manager 2012 Beta : Available for download

Once the file downloaded, you have to uncompressed it and let takes a look to the SCOM2012 installer.
image

Here the first screen of the new System Center Operations Manager 2012 installer.
image

So let’s go, click on install and we have now the license agreement.image

In the following screen, you could choose which components you want to install. In my lab, I’m using a separate SQL Server which is also hosting the SQL Reporting service role. So I will uncheck the box for the Reporting Server component for the moment and we will install it later in this post, directly on the SQL server.image

If you keep the box checked without having the Sql Reporting service installed on your current computer, you will be block later in the setup process.image

image

So, we continue the setup process without the Reporting Server component. Here we just have to choose the Installation folder.
image

The new prerequisites checker is now in progress.
image

And the result. In my lab, two blocking prerequisites.
image

First one, I have to install the Report Viewer Controls. This is available as free download at the following url : http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3841
image

The second one is the IIS 6 Metabase Compatibility role missing.image

Go the Server Manager, add role and now check the IIS 6 Metabase Compatibility.image

We recheck the prerequisites, and we just still have two warnings. As I’m using Virtual Machines in a lab environment, we could ignore it and continue the setup process.
image

Totally new environment, so we create a new management group.image

We are now at the SQL part of the setup, I specified my SQL 2008 R2 server name and the setup directly detects that the SQL Server Full-Text-Search is not installed on it.
image

Go to the SQL Server and run the SQL installer.
image

In the setup, we just have to add a component to an existing instance and we check the Full-Text-Search component.
image

Go back to our SCOM2012 installation, the setup is now ok with the configuration of the Operational database and to let us continue.
image

Now, the parameters for the Warehouse database.image

For the web console, we choose the Default Web Site :image

Choose your authentication mode for the web console. As it’s for internal use only, the Mixed Authentication is the one that we have to choose.image

The credentials part is one of the most interesting and important of the setup. I decided to use one dedicated account as Server Action Account. This account will be use for agent deployment, tasks, …. and a second account for reading and writing into the databases.
image

We have to create two news users domain.
image

Adding them as local admin of our future SCOM2012 management server and also as local Admin of the SQL Server.
image

Now, there is a part on which I’m not really sure. For installing the DB, is the setup will use my current credentials ? Or is the setup will use the service account that we provided. (I will update the post when I ‘ll receive the answer Winking smile)

In doubt, we go to the SQL server credentials and we add the SCOM2012_DB account as SysAdmin. Like this, the setup process will be able to create the DBs and assign it the rights.
image

image

Go back to the SCOM Installation, We just have to provide the credentials of the two accounts that we created.
image

My lab VMs don’t have internet access, so I’m not able to participate to the Customer Experience Improvement Program, but if you could, please do it.
image

SCOM2012 seems to be part now of the Microsoft Update program. Option that I will test when I will have a lab with internet access.
image

The installation Summary where we could review all the information we provided.
image

Installation in progress……
image

Woua, setup completed Smile
image

I opened the new SCOM2012 console… and first surprise…. I’ve got already some alerts Confused smile
image

So, It’s seems that the Action Account that we provided during needs to have Read/Write to the WareHouse Db ? What for ? As we already provided a separate service account for the DBs access. Personally, I think that this is a little bug of this first beta release, I will retrieve it to the Microsoft Connect Site.

Ok, but how to solve this ? Simply go to the SQL server, in the credentials part. We open the properties of the Action Account (SCOM2012_AA) and in the User Mapping part, we set the same rights than the SCOM2012_DB account has on the OperationsManagerDW database.
image

As we already are in the SQL Studio, we could also remove the SysAdmin right that we granted to the SCOM2012_DB account during the setup process. This is not needed anymore.
image

Once done, all the alerts disappeared Smile
image

Ok, so now, what’s about the reporting ? Go to the SQL server where we have the Sql Reporting Services installed and Run the SCOM2012 Setup, click on Install.
image

Select only the Reporting Server component.
image

Installation folder location.
image

Prerequisites checker.
image

We have to specify one management server of our SCOM2012 infrastructure.
image

We are now able to select the SQL Server instance for reporting services.
image

Red Cross, this is not good Confused smile
image

Hopefully, Microsoft did a great work on the SCOM2012 installer, and simply by clicking on the Red Cross, we have the reason of the failure.
image

We simply have to start the SQL Server Agent service and to set it to Automatically.
image

We specify the DB reader account that we already used before.
image

If you server has internet access, please choose Yes.
image

As I said before, my lab servers don’t have internet access, so I will handle the updates manually.
image

The installation Summary report.
image

Go for the installation.
image

image

Once done, we’re now able to generate reports in your all new SCOM 2012 infrastructure.
image

Stay tune for next posts about SCOM2012 Smile

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

System Center Operations Manager 2012 Beta : Available for download

5:39 pm in Uncategorized by Christopher Keyaert

System Center Operations Manager 2012 Beta is now available for public download.

Here’s what’s new

  • Rich application performance monitoring and diagnostics for .NET applications plus JEE application health monitoring
  • Support for monitoring heterogeneous environments
  • Integrated network device monitoring and alerts
  • Simplified management infrastructure (no more RMS)
  • Common console across datacenter and clouds with customizable dashboards

Microsoft Download link : DOWNLOAD
Microsoft Official announcement : Read it

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

How to start, stop and delay an Opalis Policy

12:00 pm in Uncategorized by Christopher Keyaert

Hi all,

Today, I will present a little case that I need to deal with at one of my customer. My case Study is quite simple, I’ve got a Monitor Policy which creates an Incident/Ticket each time that a critical alert is raised in Operations Manager.

I will simplify this policy to the one below :

clip_image001

The scenario, a maintenance is planned on my Incident management platform, so at the convenience time I need to stop my Monitor Policy, wait the end of the maintenance window and after start my Monitor Policy again. As Opalis allows us to orchestrate our activities, I want to automate that task.

No object presents in the default integration packs allow us to do these actions easily.

· How could I stop an Opalis policy through another policy ?
· How could I schedule the start of an Opalis policy ?
· How could I pause my policy for some minutes ?

So we just have to create it with the Opalis Quick Integration Kit (QIK). J

Basically, I need to schedule an action, stop a policy, wait the end of the maintenance window and start my policy.

clip_image003

I invite you to read carefully the post of Adam Hall, Snr. Technical Product Manager for System Center Opalis/Orchestrator : Migrating a PowerShell script to an Opalis Activity

Thanks to this article and some PowerShell command I’ve been able to create three 3 new Opalis objects :

clip_image004

1. Sleep object: As input it simply takes a number of seconds and your policy will pause for the define time.
2. Start At: Specify a date and time for starting/continuing your policy.

3. Stop Policy:
This object allows you to stop an Opalis policy from another one. It uses the sp_StopAllRequestsForPolicy SQL store procedure. As input it needs the Opalis Sql server name, database name, and the name of the policy that you want to stop.

Be carefull, the policy name that you specified must be unique, because this object will stop all the policies that are using the specified policy name. For example, if you’ve go more than one policy that is named “1.Monitor”, this object will stop all of them, and that certainly not what you want because that impact directly the others running policies. So be careful with it and always use an unique name for the policy that you want to top.
Now, we will use these news objects for stopping the Monitor Policy below during a maintenance window and restart it after, for that we created a Maintenance Window Policy.

Let’s configuring it

Monitor policy (Policy that we want to plan the stop)
clip_image001[1]

Maintenance window policy

clip_image003[1]

Start At: We decided to plan the stop of the Monitor policy the 05/17/2011 at 04:28 PM.clip_image005

Stop Policy: We want to stop the Monitor policy (which is my OEA-IncidentCreation).
image

Sleep: Here we defined a maintenance window of 5 minutes.clip_image007

Start Policy: Simply use the Trigger Policy object provided in the default integration pack for starting the Monitor Policy.clip_image008

 

Let’s testing it

1. Now, we just have to start the Maintenance Window policy and go on weekend, Opalis will orchestrate the maintenance for us.
clip_image009

2. The Start At object will wait the specified date/time for releasing/continuing the rest of the policy.
clip_image010

3. At the defined date/time, the policy is released and our Monitor Policy well is stopped by the Maintenance Policy.
clip_image011

4. Our monitor policy is stopped
clip_image012

5. The sleep object will now pause the policy for the duration of your maintenance window.
clip_image013

6. When the maintenance window is ended, the monitor policy will start automatically.clip_image015

7. Our Monitor Policy has been well started.
clip_image016

You can directly download the integration pack which is containing these news objects and the source code: Download.

Remember, this is provided “As is” without any support and/or guarantee.

Feel free to contact me through the blog comments for any questions and/or remarks.

Christopher KEYAERT

Microsoft System Center Operations Manager 2012: Overview

6:50 am in Uncategorized by Christopher Keyaert

The first of four sessions on Operations Manager 2012 has been presented at the Tech Ed North America 2011. This session provides you with an overview of the capabilities in Operations Manager 2012.

 

ACS Part II : ACS Reports Deployment & Access

5:58 pm in Uncategorized by Christopher Keyaert

Previous post about ACS :

ACS Part I : Introduction & Collector Installation

Reports Deployment

Now that you have a running ACS Collector, you have to publish the ACS Reports on your SQL Reports Services server. 

1. From the Operations Manager source, copy the files and folders present in the ReportModels\ACS to temporary folder, for example: D:\ACS
clip_image002

2. Open a command prompt
3. Go to the folder you just created (D:\ACS)
4. Executing the following command

UploadAuditReports.cmd reportsrvfqdn http://reportsrvfqdn/ReportServer ACSFOLDER

clip_image003

5. Don’t take care of the two warnings

clip_image004

6. Start your web Brower and go to http://reportsrvfqdn/Reports

clip_image006

7. Click on Show Details and go to DB Audit.
8. Adapt the Connection String field to point to your ACS Database
For example:
data source= xxxx\SQLDB1;initial catalog=OperationsManagerAC;Integrated Security=SSPI

9. If you are using the same reporting for Operations Mananger and ACS, select also the option Credentials supplied by the user running the report and check Use as Windows Credentials when connection to the data source.

clip_image007

10. Click Apply.

Access Control List

1. Create a new Active Directory group for your Security Administrator and add them as member. (By example: SCOM2007-ACSAuditors)

2. Go to your Database server and add the group as Users for you ACS DB.

clip_image001

3. Grant your group as db_datareader of your ACS database.

clip_image002

4. Your Security Administrator could now access to the ACS Reports through the SQL Reporting Services Web Interface : http://reportsrvfqdn/Reports  > Audit Reports

The next post will be about the ACS Forwarder Configuration.
Feel free to contact in case of any remarks and/or comments.

Christopher KEYAERT

ACS Part I : Introduction & Collector Installation

5:40 pm in Uncategorized by Christopher Keyaert

Hi everyone,

With Operations Manager 2007, Microsoft introduces Audit
Collection Services (ACS) as an optional but integrated component of an OpsMgr
management group. By deploying and using the ACS components of Operations
Manager, the administrator will be able to store and present security audit
information.

What is the idea?

ACS Forwarder: It’s your servers/workstations where you
installed an OpsMgr Agent and for which you want to collect the security event
log.

ACS Collector: It’s an OpsMgr management server which will be
designated as an ACS collector.

ACS DB: ACS requires having his own database. Depending of the
numbers of you forwarder, the DB could grow really fast. Satya Vel, a System
Center Program Manager, published an Excel sheet for helping you to size the
ACS DB. (http://blogs.technet.com/b/momteam/archive/2008/07/02/audit-collection-acs-database-and-disk-sizing-calculator-for-opsmgr-2007.aspx)

ACS Reporting: ACS is using SQL Reporting Services, so you have
the choice to install a new fresh server, or using the one that you already
used for OpsMgr reports. If you want to use your existing SQL Reporting server
and want continue to be in a Microsoft supported configuration, each time that one
of your Security Administrators want to generate an ACS report, he will have
to enter his credentials.

The best practice is to generate ACS reports directly from
the SQL Reporting web interface and not directly from the integrated reporting
pane available in SCOM console. This is due to the fact that ACS reports could
contain sensitive information and you don’t want that all your SCOM Operators
could see that information. The other advantage, and that you just need to
provide the web url to you Security Administrators, no need to install the SCOM
Console.

Security Administrator: Is the person of you company that will
be able to generate ACS Reports through the web interface of SQL Reporting
Services.

Pre-requisites

I invite you to take a look to the Operations Manager Supported Configuration page available on Technet : http://technet.microsoft.com/en-us/library/bb309428.aspx

What do you need :

  • OpsMgr infrastructure.
  • Service Account (a simple domain user).
  • A database server (Grant your service account to
    interact with the DB Server.
  • A dedicated management server that you will use as ACS Collector. (Grant your service account as Local Administrator).
  • Active Directory Group which contains your
    Security Administrators.
  • A reporting server (Dedicated or the one used
    for OpsMgr reporting).

Collector Installation

1. Log on to your dedicated management server with
your service account.

2. Launch in the OpsMgr setup and click on Install
Audit Collection Server.

 

3. Choose Create a new database.

 

4. ACS uses a ODBC connection to SQL, here you can
modify the Data source name.

 

5. Select Remote database server

 

6. Select Windows authentication

 

7. I suggest keeping the default parameter, Use
SQL Server’s default data and logging file directories.

 

8. Number of day an event is retained in
database
, is the maximum age for which you’ll be able to generate ACS
report. Keep in my that higher the number of days is, more space your DB will
use.

 

9. In the case, we use only one ACS DB, select Local.

 

10. Summary of the installation options

 

11.  
Click ok to confirm Authentication information

 

12.  
Installation of the ACS Collector finished

Now, you have your first collector installed [:)]
The next post will be about the publication of the ACS reports on the reporting server.
Feel free to contact in case of any remarks and/or comments.

Christopher KEYAERT