This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to
Article Number 000385088 for more information about troubleshooting BMC products in containers.
Consider the following image:
Lets cover the Queues first. You should set the queues with your hardware capabilities in mind but basically here are the areas to consider:
ARS Feature
|
Default (OOB)
|
Application Friendly(Recommended)
|
Comments
|
| Process timeout | 5 | 20 |
Max is 60, but there should usually be no need to go beyond 20 or 30 seconds for this timeout as the longer the timeout the more
delay for external APIs to release a slow or non-responding thread back into the queue.
|
| Max-Entries-Per-Query | 0 | 5000 | Please see article 000097319 "Symptoms: Trouble merging records with Reconciliation, AIE can't update and CMDB2ASSET synch is not" for more info |
| 390620 Fast Queue | 2 [Min] / # of CPU Cores [Max] | 2 [Min] / 3 [Max] times the total count of CPU cores | Never exceed 32 Max threads even if you have more than 8 Cores |
| 390635 List Queue | 2 [Min] / # of CPU Cores [Max] | 2 [Min] / 5 [Max] times the total count of CPU cores | Never exceed 32 Max threads even if you have more than 8 Cores |
| 390626 Plugin Loopback RPC | Not set | 8 / 8 | Use of this queue depends on the application. Check the documentation for your application to see if it includes calls to this queue. |
| 390696 CMDB Admin | Not set | Do not set | This is the CMDB queue that acts similar to AR Server 390600 Admin queue. Only one thread is need. Do not add additional Max threads to this queue. |
| 390698 (*)RE/CMDB Queue | Not set | 4 [Min] / 16 [Max] |
We recommend no more than 16 threads Max for this queue as the higher the count of threads, the more will the CPU utilization grow.
"4 4" setting will use about 20% of the CPU when Recon job is triggered/per Job triggered. "8 8" will use 40% and so on.
Remember to add this value to your RE Server Settings or add this directly to ar.cfg:
RE-RPC-Socket: 390698
Setting the Maximum beyond 16 threads actually has a negative impact on performance. It is therefore recommended not to exceed 16 threads. Each create, modify, delete and merge modification to a CI triggers additional workflow by CMDB data consuming applications. Pushing the AR Server with excessive data changes on more than 16 threads will eventually overwhelm the AR Server. Please be conservative with this setting.
|
| 390699 RE/CMDB Queue | Unset | 4 [Min] / 8 [Max] | This queue can be set for Normalization. (See screen capture below for reference on how to set the actual NORM RPC Socket.) |
| Maximum Filters for an Operation | 10000 | 200000 [or higher] | This value is set to 500,000 after installing ITSM |
| Maximum Stack of Filters | 25 | 1000 | This value is set to 10,000 after installing ITSM |
| Plugin-Filter-API-Threads | Not Set | 4 12 | These threads are used when a filter makes a call to an external API or External Plugin (data vendor). |
| Plugin-ARDBC-Threads | Not set | 4 12 | These threads are for the C based ARS Server Plugin. With diminishing use of C plugin this setting is losing importance. |
| Plugin-AREA-Threads | Not set | 4 12 | Only if AREA is used. Do not add if AREA is not in use. |
JAVA PLUGIN Thread Settings
The following are java plugin server configuration settings. Each java plugin is bundled with a pluginsvr_config.xml file which is stored in either the ARS_HOME/pluginsvr and then each application has a location for it within its own install directory.
For example AtriumCore has two different java plugin server configuration files: ATRIUMCORE_HOME/cmdb/plugins/ne (NORM) and ATRIUMCORE_HOME/cmdb/plugins/shared (SHARED)
First we should configure the ARS Plugin Svr to add more threads: The configuration header includes configuration info on install:
<pluginsvr_config>
...
<numCoreThreads>5</numCoreThreads>
<numSelectorThreads>2</numSelectorThreads>
<workQueueMonitorLogInterval>0</workQueueMonitorLogInterval>
<workQueueTaskThreshold>5</workQueueTaskThreshold>
Make the following changes to the ARS Java Plugin config file (pluginsvr_config.xml)
<numCoreThreads>30</numCoreThreads>
<numSelectorThreads>2</numSelectorThreads>
<workQueueMonitorLogInterval>0</workQueueMonitorLogInterval>
<workQueueTaskThreshold>5</workQueueTaskThreshold>
Make the following changes for AtriumCore SHARED Java Plugin config file (pluginsvr_config.xml)
<numCoreThreads>10</numCoreThreads>
<numSelectorThreads>2</numSelectorThreads>
<workQueueMonitorLogInterval>0</workQueueMonitorLogInterval>
<workQueueTaskThreshold>5</workQueueTaskThreshold>
Make the following changes for NORM (NE) Java Plugin config file (pluginsvr_config.xml)
<numCoreThreads>10</numCoreThreads>
<numSelectorThreads>2</numSelectorThreads>
<workQueueMonitorLogInterval>0</workQueueMonitorLogInterval>
<workQueueTaskThreshold>5</workQueueTaskThreshold>
These plugin settings are not absolute. You will stil have to do some projection on the data size that will be processed in any given data set. Performance will be impacted by the CPU count and the amount of RAM.
Even the most powerfull systems we've seen still needed to be finetuned in a similar way. Please make this part of your deployment.
Finally, the Normalization settings. Please see the attached screen capture of the Edit Configuration panel in the Normalization console.
Some of the misunderstandings we've seen in the field include the overlap of the RE RPC Queue and the CMDB RPC Queue.
These are not the same thing.
Normalization engine makes a call back to the ARS using the Java AR API and this call can occur on any queue java server opens up for this bidirectional communication.
In this example we've set it to 390621, which is a queue usually created during the AtriumCore installation (it does not get set during ARS install) and this queue
has generally been unused while taking up at least 5 threads. You can use this queue for Normalization calls as long as you have nothing else set to use it.
The 390699 and 390698 are queues that are reserved for CMDB specifically. If you have set RE RPC Queue to 390698, then make sure that you're not setting it to the
same queue in the NORM console. That would need to use 390699. It would cause issues and cause errors that would only be visible to the system admin who would have access to the relevant logs.
Filter Maximums
One of the frequently overlooked settings that need to be changed on the ARS ARDBC are the Maximum totals for:
Filter-Max-Total: 10000
Filter-Max-Stack: 25
For applications they will need to be set to at least:
Filter-Max-Total: 500000
Filter-Max-Stack: 100
Please do this as soon as possible, especially when the arerror.log shows this error:
"Too many filters processed during this operation"