How do I ensure the midtier component is configured correctly for better performance

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Health check areas
1. Compatibility
 Advice: Check the compatibility matrix for the midtier version.  If midtier version is from a couple of years ago, it may not have been tested with newer platform tools, eg: OS, Java, Tomcat.
       Example compatibility check:   Midtier 9.0 is compatible with:       Known incompatibility:  Java 8 is not compatible with Midtier 7.6.04 
        
2. Follow the Midtier Deployment knowledge article that pertains to the version of midtier in use:
On Windows OS, the JVM setting can be reviewed from the Tomcat Config panel - Java tab.   The Tomcat Config panel is accessed by executing the <Tomcat_install>\bin\tomcat#w.exe where # designates the Tomcat release..

• Set heap size the same for min and max.  Note:  These may be higher if the OS and Java are 64 bit,and  the OS physical RAM is greater than HeapSize+3GB.

• Set  the ratio between memory segments

• For *nix systems set font paths|

•   Set garbage collector settings like this

•  In case a problem happens, make sure JVM will log that information . The recommended settings are: 

•  CompressedOops . On or Off ?

•  Provide space for Classes storage

• Regarding monitoring tools 

• For Solaris only:  

                
3.  Load balancer and Network configuration
        For load balancer configuration, it’s the customers responsibility to verify that the load balancer is setup according to the following recommendations.
        
The required settings are:
 Load balancers in front of midtiers (between browsers and midtiers) 

•  Session affinity must be turned On using cookie-based persistence. This applies to midtiers installed in a Tomcat clustered environment as well as standalone.
•  Any session timeout set at the Loadbalancer level must be longer than the session timeout set in the midtier config tool.   If session cookies are used, then this requirement is mute.
•  Enable the OneConnect profile for F5 loadbalancers.   This means that traffic is balanced at the HTTP request level and not the TCP Connection level.
• If your load balancer or network infrastructure uses HTTP pipelining additional considerations may be needed.
  • Option 1: Avoid using Http Pipeline
  • Option 2: If you must use HTTP  Pipeline F5 only sets the session cookies at the first request, assumption is that due to keep-alive all the user's subsequent requests will be on the same TCP connection, so no need to send cookie or set cookie on every request/response.  However,  if "http pipelining" is turned on: multiple users share the same TCP connection and a single user could end up using multiple TCP connections. In that regards the cookies may not be set properly for some clients reusing a connection where the cookie was sent earlier.   In order to compensate  you can enable  the "Send cookie on every request" setting in F5. 

           
 Load balancer between Midtier and AR server

• Session affinity should be turned Off
• Under midtier connection settings, "Enable lifespan checkbox" should be ticked and the Connection Lifespan set to 15 minutes
• Default values for Connection Pool settings are ok for 90% of cases
• There will be a connection pool of 80 connections per AR server/ AR server Group. This pool would suffice to serve up to 4 or 5 times the number of users (320 to 400 human users)        
• NOTE:  If the high watermark under connection pool goes above 40 connections (50%), collect midtier performance logs and analyze if any operation is taking longer than 10 seconds.   Long running api/sql calls may be the root cause of the high watermark and can be linked to AR server logs. 
• OneConnect profile should NOT be used in the load balancer that resides between the midtiers and the AR Servers.

1.  For more information, please see this link https://support.f5.com/kb/en-us/solutions/public/7000/200/sol7208.html
2.  Avoid using a OneConnect profile for non-HTTP virtual servers that process more complex transactions, such as FTP or RTSP.   Doing so may result in traffic disruption and session failure. Even for simple non-HTTP protocols, an iRule may be required to manage connection reuse.

• Http profile should not be used in this load balancer. It can prevent the AR server to be added to the midtier config.jsp stating RPC errors

                Make sure network is not dropping packets
                Make sure network MTU size is adequate 4.  Midtier hotfixes
        For the most recent version, you can find the latest recommended hotfix here:  ftp://ftp.bmc.com/pub/ARRecommendedFixes/
        
5.  Have a single AR server configured under the Midtier config tool - AR server settings
        Having multiple AR servers configured under AR server settings will increase the amount of cache used. Extra AR servers can be added to help troubleshoot certain scenarios to help identify AR Server problems.  Once the issue is solved (eg. Schedule normalization engine), remove any extra AR servers from midtier configuration tools
    
6. Preload
        Preload is a technique that will reduce the time to render the form for the first time. It works by gathering a list of active links from the AR server(s), then traverses the forms linked to them and builds partial cache. Once a user accesses a form that was preloaded, the form will finish its cache process (compile HTML will still be displayed on midtier logs). 
In some cases forms or views cannot be identified by this process and then preload would not partially load those forms. 
       
How to identify if a form is fully loaded into cache? Access config.jsp page, login and then modify the url to go to config_cache_adv.jsp.  Review the View Building Information (sample displayed following) found at the bottom of the config_cache_adv.jsp page.

View Building Information

SERVER	FORM NAME	VIEW NAME	INLINE	LOCALE	TIMEZONE	USER NAME	GROUP-COMBO	HIT COUNT
clm-aus-019258	AR System Customizable Home Page	Default Administrator View	true	en_GB	Etc/GMT+12	Allen	Admin	1
clm-pun-033729	SHR:OverviewConsole	Overview Homepage Content	true	en_US	Etc/GMT+12	Allen	Admin	2

All the views for these particular users would be in the cache. 
Once a form is in this table (it is also in ViewStats.dat file), any process that rebuilds the cache (sync cache, or restarting mid-tier)  will fetch all these views.  Access to any of these views by users with same permissions will only see Compile JS.

Mid-tier will always collect information in ViewStats.dat files regardless of cache preferences or preload enabled or not. 
After some time has passed and ViewStats.dat reflects the forms actually in use;  preload or prefetchConfig.xml would only become a burden at restart times and should be disabled.
If the viewStats.dat has been removed, enableing preload or prefetchConfig.xml may help 

       
7.  Resource Check Intervals
        For production midtiers, the cache should not be set to sync automatically. There are 2 ways to achieve this:
        a. Disable Perform Check : This will still cache objects but won't ping AR server at regular intervals. However, this will disable  " sync cache" button that may be useful later.
        b. If you still need sync cache, but want to check cache manually :  increase the Resource Check intervals from 86400 (one day) to 86400000 (1000 days)

WARNING:  Setting the Check Intervals to 0 will disable midtier cache and would require midtier to retrieve Form, permissions and Active links from AR server on each form request.  This would have impact performance.          
8.  Enable Cache persistance
        This will save cache to disk when the JSP engine is shutdown or restarted.  On startup, the cache will be retrieved from disk and only refresh any content that has been updated.
        
9. When do I need to flush the cache?
        If content is promoted to production midtiers, start by syncing the cache. Midtier cache logs will show when this operation starts and finishes. If any content is not synced, try flushing the cache. If that fails, perform a hard cache flush for the whole cache to be rebuilt.  (See step 10).  Consider that hard cache flush will delete viewStats.dat and that file reflects the views more frequently used in your system. Without it rebuilding cache will be lengthier and depending on your configuration it may cache forms that are not really useful to users.
         If you only have a handful of forms and the midtier is version 9x, use the http://<midtier>:<port>/arsys/shared/config_cache_cleanup.jsp page to clear specific forms from the cache.   Note:  form and active links will be cleared.
       
 10. How to perform hard cache flush
        Stop Tomcat (Jsp Engine) Service.
        Clear out the contents of the 'work', 'temp' & 'logs' folders inside Tomcat X.X folder
        Clear out the contents of the 'cache', 'cachetemp', 'attstore' and 'logs' folders present inside the Mid-Tier folder.
        For versions 8.1.02 or earlier, Delete 'viewStats.dat' and 'viewStats.dat.bak' from <Mid-Tier folder>\WEB-INF\Classes  if present.
        For versions 9.0 or later, delete the content of the <Mid-Tierfolder>\WEB-INF\classess\viewServerStat directory
        Start the Tomcat (Jsp Engine) Service.
        Clear browser cache/cookies/etc completely. 

        
11.  Which logs should I have enabled in production:
        During normal operation (no issues observed)
            Go to Log Settings and enable the following categories
                Session Management, Performance,  Servlet
            Make sure 
                Log Level is set to: Info
                and log format: Detail text
                Maximum Log File Size (kb):  10240
                Maximum Number of Log Files : 10
        If a generic problem is observed 
            Go to Log Settings and enable the following categories
                Cache, Session Management, Performance,  Servlet
                Consider filtering the logs by User
            Make sure 
                Log Level is set to: Fine
                and log format: Detail text
                Maximum Log File Size (kb):  10240
                Maximum Number of Log Files : 100

12. How to make sure your form will render properly. 
Contact BMC support if help is needed . Provide evidence that this article has been followed (screenshots, midtier/WEB-INF/classes/config.properties file, jvm snapshot and versions from your midtier, OS, tomcat and JVM )