While working on a new System Center Configuration Manager 1511 server that I was building, I noticed that each of the Component Status logs were returning 0 messages for the previous 1 day. If I set the threshold back to 1 week, I could see events, but they had completely stopped a few days earlier. Something was preventing new logs from being created.
The Component Status logs are found under the Monitoring node, by expanding System Status and then selecting Component Status. The odd thing about this was that the icons and status seemed to be updating correctly, suggesting that the data for the logs was still being collected and processed, but SCCM could not display the logs.
The events written to statmgr.log looked something like this (in part):
*** exec sp_InsContentDistributionStatus N'', N'RTM00405', 0, 72057594038182832, 2361, '02/18/2016 07:04:10.240', N'\\myserver.com\dfs\Packages\Microsoft\Updates\Windows 7 Server 2008 R2 Office 2010 Patch Tuesday Updates 2015\95289d79-38a1-44a3-a2a7-5f54febcb70b', N'95289d79-38a1-44a3-a2a7-5f54febcb70b', N'95289d79-38a1-44a3-a2a7-5f54febcb70b', N'30', N'69', N''; SMS_STATUS_MANAGER 2/19/2016 10:54:34 AM 4648 (0x1228) *** [22001][8152][Microsoft][SQL Server Native Client 11.0][SQL Server]String or binary data would be truncated. : sp_InsContentDistributionStatus SMS_STATUS_MANAGER 2/19/2016 10:54:34 AM 4648 (0x1228)
The corresponding .sql file in Microsoft Configuration Manager\inboxes\statmgr.box\retry\ looked something like this (in part):
exec sp_InsContentDistributionStatus N'', N'RTM00405', 0, 72057594038182832, 2361, '02/18/2016 07:04:10.240', N'\\myserver.com\dfs\Packages\Microsoft\Updates\Windows 7 Server 2008 R2 Office 2010 Patch Tuesday Updates 2015\95289d79-38a1-44a3-a2a7-5f54febcb70b', N'95289d79-38a1-44a3-a2a7-5f54febcb70b', N'95289d79-38a1-44a3-a2a7-5f54febcb70b', N'30', N'69', N'';
As I looked through the .sql files containing the problem transactions, I noticed a pattern appear. Each .sql file contained a stored procedure named “sp_InsContentDistributionStatus”, and each of these stored procedures involved the source path to a Microsoft update in one of my Software Update Deployment Packages.
Nothing about the arguments to the stored procedure looked out of place, and in fact matched very closely to stored procedures that ran successfully once the jam in Microsoft Configuration Manager\inboxes\statmgr.box\retry\ was cleared.
I did some Googling and formulated a course of action to restore most of the missing log entries until I had time to troubleshoot the root cause.
- Open an elevated command prompt to the \statmgr.box\retry\ location. For example, on my server, where I have installed SCCM to the D: drive, the directory is located at:
D:\Program Files\Microsoft Configuration Manager\inboxes\statmgr.box\retry\
You’ll probably find dozens, if not hundreds of .sql files in this location. - Sort the files by date in descending order, to see the oldest at the bottom of the list, using the command:
dir /o:-d
This file contains the query that is blocking later transactions. - Open the SCCM server’s statmgr.log in cmtrace.exe and leave it open. On my server, this file is located at:
D:\Program Files\Microsoft Configuration Manager\Logs\statmgr.log - Search the log for the file name of oldest .sql server in the \retry\ directory. Depending on how long the problem has existed you may or may not find entries around the failure to submit the transaction.
- Move the oldest .sql file from the \retry\ directory into another location. This file will not be processed, and as such will result in some data loss, but it will allow you to proceed.
- Start the Configuration Manager Service Manager, expand Components, and then query, stop, query, start the SMS_STATUS_MANAGER component.
You should see activity in the statmgr.log as it begins to process the failed transactions, but it should be able to successfully process the transactions that were queued up behind the problem query. - Monitor the statmgr.log for failures. If another back-logged .sql file contains an invalid query, the statmgr.log will report the failure, and you will be able to capture both the events surrounding the failure and the .sql file containing the failed transaction for analysis.
- Repeat the process of moving each failed .sql file out of the \retry\ directory, then restarting the SMS_STATUS_MANAGER component until all of the back-logged .sql files have been processed.
As a first step to attempt to resolve the problem, I deleted the Software Update Deployment Package and then rebuilt it from the source path. The .sql files did not build up in Microsoft Configuration Manager\inboxes\statmgr.box\retry\, and the problem where the component status messages stopped being display did not recur. So, unfortunately, I seem to have fixed the problem by going straight to the nuclear option, and missed out on any opportunity for finding the root cause. While I don’t have an explanation about why the stored procedure errored out, I’m posting this description of my experience in hopes of helping others. If you can explain the cause of the problem, please do post a comment.
