How do I ensure the midtier component is configured correctly for better performance |
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers. Health check areas 1. Compatibility Advice: Check the compatibility matrix for the midtier version. If midtier version is from a couple of years ago, it may not have been tested with newer platform tools, eg: OS, Java, Tomcat. Example compatibility check: Midtier 9.0 is compatible with: Third Party Prerequisite: Java 7 and above
Enabling Tech: Apache Tomcat 7 and above Release Date:04/30/2015 2. Follow the Midtier Deployment knowledge article that pertains to the version of midtier in use: Midtier 7.6.04 through 9.0 https://communities.bmc.com/docs/DOC-18162
Midtier 9.1 and later https://selfservice.bmc.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000h9kqCAA&type=FAQ On Windows OS, the JVM setting can be reviewed from the Tomcat Config panel - Java tab. The Tomcat Config panel is accessed by executing the <Tomcat_install>\bin\tomcat#w.exe where # designates the Tomcat release.. For other OS, review the JAVA_OPTS settings in one of the startup scripts.
JVM settings have the most impact With these settings, Heap size will be used and when reaching 70-90% of usage Garbage collector will be invoked and remove unused objects from it.
-Xms3072m -Xmx3072m
-XX:NewRatio=2
-Djava.awt.headless=true
-Dsun.java2d.fontpath=JREPATH/lib/fonts
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+HeapDumpOnOutOfMemoryError
-XX:ErrorFile=PATH/java_hs_err.log -XX:HeapDumpPath=PATH
-XX:+UseCompressedOops (enables it)
Some knowledge articles (including the midtier deployment knowledge article would recommend to enable it. It can be enabled, but be aware that we have seen problems with Java 6. If the JVM crashes and produces an hs_err_pid file, we can identify if this option was causing it, and then disable it. There are no definitive measurements of performance gain/loss with or without this flag enabled.
Option A: Play it safe and disable it. -XX:-UseCompressedOops
Option B: Enable it and in case it crashes disable it using Option A
For java 7 or lower: -XX:MaxPermSize=256m
For java 8 or better: -XX:MaxMetaspaceSize=512m
To monitor java performance of the server from a remote desktop, add
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=19999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false If jVisualVM can be used, gather a snapshot to have a better understanding of your JVM performance.
Add -d64
This forces the JVM to run in 64 bit mode. Without it, Solaris will run the JVM in 32 bit mode and heap will be limited to 2 GB. Allocating more than 2 GB heap in 32 bit mode will cause an error during runtime
3. Load balancer and Network configuration For load balancer configuration, it’s the customers responsibility to verify that the load balancer is setup according to the following recommendations. The required settings are: Load balancers in front of midtiers (between browsers and midtiers)
Load balancer between Midtier and AR server
Network health check
Make sure network is not dropping packetsMake sure network MTU size is adequate https://bmcapps.my.salesforce.com/knowledge/publishing/articlePreview.apexp?id=kA214000000d5fN http://www.microhowto.info/howto/persistently_change_the_mtu_of_a_network_interface_on_redhat.html
4. Midtier hotfixeshttp://www.microhowto.info/howto/change_the_mtu_of_a_network_interface_using_dhcp.html For the most recent version, you can find the latest recommended hotfix here: ftp://ftp.bmc.com/pub/ARRecommendedFixes/ 5. Have a single AR server configured under the Midtier config tool - AR server settings Having multiple AR servers configured under AR server settings will increase the amount of cache used. Extra AR servers can be added to help troubleshoot certain scenarios to help identify AR Server problems. Once the issue is solved (eg. Schedule normalization engine), remove any extra AR servers from midtier configuration tools 6. Preload Preload is a technique that will reduce the time to render the form for the first time. It works by gathering a list of active links from the AR server(s), then traverses the forms linked to them and builds partial cache. Once a user accesses a form that was preloaded, the form will finish its cache process (compile HTML will still be displayed on midtier logs). In some cases forms or views cannot be identified by this process and then preload would not partially load those forms. How to identify if a form is fully loaded into cache? Access config.jsp page, login and then modify the url to go to config_cache_adv.jsp. Review the View Building Information (sample displayed following) found at the bottom of the config_cache_adv.jsp page.
Once a form is in this table (it is also in ViewStats.dat file), any process that rebuilds the cache (sync cache, or restarting mid-tier) will fetch all these views. Access to any of these views by users with same permissions will only see Compile JS. Mid-tier will always collect information in ViewStats.dat files regardless of cache preferences or preload enabled or not. After some time has passed and ViewStats.dat reflects the forms actually in use; preload or prefetchConfig.xml would only become a burden at restart times and should be disabled. If the viewStats.dat has been removed, enableing preload or prefetchConfig.xml may help 7. Resource Check Intervals For production midtiers, the cache should not be set to sync automatically. There are 2 ways to achieve this: a. Disable Perform Check : This will still cache objects but won't ping AR server at regular intervals. However, this will disable " sync cache" button that may be useful later. b. If you still need sync cache, but want to check cache manually : increase the Resource Check intervals from 86400 (one day) to 86400000 (1000 days) WARNING: Setting the Check Intervals to 0 will disable midtier cache and would require midtier to retrieve Form, permissions and Active links from AR server on each form request. This would have impact performance. https://docs.bmc.com/docs/display/public/ars91/About+Mid+Tier+caching
8. Enable Cache persistance This will save cache to disk when the JSP engine is shutdown or restarted. On startup, the cache will be retrieved from disk and only refresh any content that has been updated. 9. When do I need to flush the cache? If content is promoted to production midtiers, start by syncing the cache. Midtier cache logs will show when this operation starts and finishes. If any content is not synced, try flushing the cache. If that fails, perform a hard cache flush for the whole cache to be rebuilt. (See step 10). Consider that hard cache flush will delete viewStats.dat and that file reflects the views more frequently used in your system. Without it rebuilding cache will be lengthier and depending on your configuration it may cache forms that are not really useful to users. If you only have a handful of forms and the midtier is version 9x, use the http://<midtier>:<port>/arsys/shared/config_cache_cleanup.jsp page to clear specific forms from the cache. Note: form and active links will be cleared. 10. How to perform hard cache flush Stop Tomcat (Jsp Engine) Service. Clear out the contents of the 'work', 'temp' & 'logs' folders inside Tomcat X.X folder Clear out the contents of the 'cache', 'cachetemp', 'attstore' and 'logs' folders present inside the Mid-Tier folder. For versions 8.1.02 or earlier, Delete 'viewStats.dat' and 'viewStats.dat.bak' from <Mid-Tier folder>\WEB-INF\Classes if present. For versions 9.0 or later, delete the content of the <Mid-Tierfolder>\WEB-INF\classess\viewServerStat directory Start the Tomcat (Jsp Engine) Service. Clear browser cache/cookies/etc completely. 11. Which logs should I have enabled in production: During normal operation (no issues observed) Go to Log Settings and enable the following categories Session Management, Performance, Servlet Make sure Log Level is set to: Info and log format: Detail text Maximum Log File Size (kb): 10240 Maximum Number of Log Files : 10 If a generic problem is observed Go to Log Settings and enable the following categories Cache, Session Management, Performance, Servlet Consider filtering the logs by User Make sure Log Level is set to: Fine and log format: Detail text Maximum Log File Size (kb): 10240 Maximum Number of Log Files : 100 12. How to make sure your form will render properly. Test any business crucial form using developer studio documentation found here
Rendering a form on midtier is an intensive process, it involves several API calls to AR server, including gathering Groups, Roles and Active Links. Once Midtier has gathered this information, it still needs to transform the form definitions to HTML and the active links to Javascript. When the user logs in, cache management will get the proper cached version for this user and send it to the browser for rendering. We encourage to test business crucial forms by using the "view in browser" feature from developer studio. This will run a full cycle from start to finish and will allow to detect issues in development or testing , possibly avoiding a situation in production systems Contact BMC support if help is needed . Provide evidence that this article has been followed (screenshots, midtier/WEB-INF/classes/config.properties file, jvm snapshot and versions from your midtier, OS, tomcat and JVM ) |