A) Useful High-Level questions to answer to help BMC Customer Support understand the problematic behavior

1) Is the performance problem consistent or does it appear to be intermittent?
If intermittent, does there seem to be any pattern to the performance problem? e.g. Sunday nights, when another particular job is running etc.

2) When did the performance problem(s) begin to occur in the BSA environment and were there any noticeable environment changes made at that time? e.g. BSA Upgrade, Database migration, Network change, changes to the virtualization infrastructure on which the BSA env resides etc.

3) Does the performance problem affect BSA Job execution time, BSA Console responsiveness or both?

4) Slow Job Performance:

i) If BSA Jobs are affected, is the problem confined to certain Job(s), certain Job Type(s) or does it appear to affect many Jobs of different types?
ii) How does the performance of these jobs differ from previous experience?

5) Slow BSA Console Performance:

i) If BSA Console responsiveness is affected, are there any particular user operations (e.g. saving a particular Component Template) which are responding slower than expected or is it an across-the-board slowness?
ii) How does the performance of these operations differ from previous experience?
iii) Do other user's BSA Consoles also have the same performance problems or is it confined to a single user's console?
iv) Is the BSA Console installed in the same subnet as the BSA Application Server?
v) Is the BSA Console running on user desktops or on a centralized Console Server e.g. Citrix
vi) Is there anti-virus software running on the BSA Console? Can that be temporarily disabled and the operation retested?
vii) Do the Console performance performance seem better or worse when logged in as certain roles? Does the BLAdmins role have the same problem?
 

6) If the BSA environment uses an Oracle Database, is the BMC-recommended gather_schema_stats job running weekly and has it been succeeding? This can be confirmed via a DBA, the Health and Value Dashboard or running a DB Diagnostic as described in the "Verifying that statistics are current" section of this page.

7) Has the Oracle Out of Box gather statistics job been disabled as recommended in the BSA documentation? Running both the BMC-recommended gather_schema_stats job and the Out Of Box Oracle version, even at different times, has been known to cause performance problems which is why the BSA documentation recommends disabling the Out Of Box Oracle version.

8) If the BSA environment uses a SQL Server Database, is the SQL Server sp_updatestats procedure scheduled to run weekly?  And does an index rebuild occur on a regular schedule?

9) Is the BSA Database Cleanup Job running regularly and successfully in this environment?

10) Run the Database Diagnostic under BSA Appserver Diagnostics. This tests DB latency by running many queries and returning their response time to the expected response time.

11) Make sure that the time on all the application servers and the Database servers are in SYNC. A time difference, even of minutes, on any of these servers can cause performance problems for jobs.

12) Are the BSA Appservers and BSA DB in the same network location? If Virtual Machines, are the BSA Application Servers and DB running on hosts in the same datacenter? We have seen issues where the DB Server is running on a host on a different, albeit close by, data center. Once the DB Server host was migrated to the same data center as the BSA Application Servers, the performance issue was resolved. The VM team should be consulted to verify this.

B) What items should be gathered and provided to BMC Customer Support?

In addition to answering the above questions, the following data should be provided to BMC Customer Support to help with problem diagnosis.

1) Generate Support Data: Run the “Generate Support Data” with the below options checked/selected for each Application Server in the environment. Note that the tool can be run once with all application server selected.This video discusses how to run the “Generate Support Data” utility. 

In addition to collecting the Appserver and Console logs (including rolled over log files) selecting the below options will also provide BSA Customer Support with extensive information about the BSA environment including:

1. Version information
2. Number of Application Servers in the BSA Environment
3. OS of Application Servers
4. Database Type and Version
5. Memory configuration
6. Application Servers configuration settings e.g. MaxWorkItemThreads

2) Job Run Logs of recent BSA DB Cleanup Jobs. There may be multiple jobs configured to run on different days e.g. Daily Cleanup and Weekly Cleanup. Please gather the job run logs of each Cleanup Job. This will help BMC Support confirm what cleanup commands are being run and whether they are running to completion.

3) Table Sizes: Run whichever of the following scripts (both contained in attached BSADBScripts.zip) is appropriate for your DB platform and return the output:

• Oracle: calc_tbl_size_oracle.sql
• SqlServer: calc_tbl_size_sqlserver.sql

4) Dump of DELETE_TASKS Table to review cleanup activity:  Run whichever of the following scripts/queries is appropriate for your DB platform and return the output:

• Oracle: check_del_task.sql (contained in attached BSADBScripts.zip)
• SqlServer: 'select (*) from delete_tasks' and export to a csv

4) Table Stats Check: Run whichever of the following scripts (both contained in attached BSADBScripts.zip) is appropriate for your DB platform and return the output:

• Oracle: lastAnalyzed_oracle.sql
• SqlServer: lastAnalyzed_sqlserver.sql

5) Index Fragmentation Check: Run whichever of the following scripts (both contained in attached BSADBScripts.zip) is appropriate for your DB platform and return the output:

• Oracle: indexFragmentation_oracle.sql
• SqlServer: indexFragmentation_sqlserver.sql

6) Database Reports:

• Oracle: Request that the site DBA provide Oracle AWR, ADDM and ASH reports for a time period during which the performance problems were observed. It is important that the reports are captured for a timeframe during which the problems were observed. Rather than a single report covering a 12 hour period, it is better to generate multi hourly reports which can then be correlated to Job Run logs, known times when Console performance was slow etc.
• SqlServer: Sql Server usually requires setting up a SQL Server Profiler session and capturing data, so that may require reproducing the problem again when the profiler session is running.

7) Health and Value Dashboard results zip file: The BSA Health and Value dashboard stores its results in the <File Server Storage>\dashboards\reports\latest directory. See below example. Please zip up the most-recent directory and provide the resultant zip file:

.

8) Job Performance Issues:

8) BSA Console Performance Issues:
   ​        The following log from the Windows BSA Console which is experiencing the problem as well as noting the timeframe which which the problem was observed so that this can be correlated to the events in the logs:
           i) %APPDATA%\1_2_1\Workspace1\.metadata\.log (no filename - just called .log)
           ii) %APPDATA%\Bladelogic\1_2_1\Workspace1\.metadata\.plugins\com.bladelogic.client.ui\* (all BLWorkbenchPlugin* files including rolled over)

C) Where to send the data?

The information collected above may result in a large amount of data to be provided to BMC Customer Support. File under 2GB may be attached to the support case. Larger files may be uploaded to https://mft.bmc.com which requires a valid support login, note in the case the email the for the acount used to login so the Support person handling the case can find the correct files.