Description of problem encountered

Users experience sporadic, but not reproducible on demand ARRER 9201 errors. 

Even though the Load balancer of the environment has been set to a timeout of 90 minutes, and the last activity has occurred less than 90 minutes ago, the error sporadically appears.   The Load balancer uses session stickiness (session affinity enabled) with the IP-address.  

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.
 

You are likely to see this error if you are using a load balanced environment. To maintain a session, Mid-Tier uses the JSESSIONID, G and MJUID cookies. Every request that's sent by the browser contains these cookies and during the entire session the HTTP requests have to be sent to the same Mid-Tier server.

There are a few reasons why it might go wrong in a load balanced environment:

1. The load balancer in front of the Mid-Tier servers is supposed to use session affinity, so all sessions from one user are supposed to go to one dedicated Mid-Tier server. If you see a lot of 9201 errors it could be that session affinity (sticky bit) is not set up correctly.
2. When you set the session affinity to cookies and the session time out above the Mid tier session time out, the load balancer will create a cookie that will add to the browser to preserve the tracking, sometimes this cookie is set to expire even if there's activity on the browser, causing a 9201
3. It could be that the load balancer is set up with session affinity but that at one point it sends the HTTP request to the wrong server. Unless you run a clustered environment, the various JSP engines are not aware of each other. So if a request is supposed to be sent to server A but somehow ends up at server B, server B will not recognize the session ID and will throw the same error. For example, this could happen because a load balancer determines the first server is overloaded or not available and it automatically switches to server B. You need to check the load balancer logs and work out why it's making this decision.
4. Mid-Tier depends on a servlet engine (such as Tomcat) which in turn might use a web server (Apache, IIS). When a browser accesses a Mid-Tier page, its timeout will initially depend on whatever is set on the web server; Mid-Tier'S own timeout value has to fall within this. Therefore make sure that the web server and servlet engine's timeout parameter is always greater than the Mid-Tier Session Timeout parameter. Check your web server / servlet engine documentation for more information.

Configuration recommendations:
For the Loadbalancer in front of multiple Mid-Tier servers, the following configurations are recommended:
1.  Session Affinity (Sticky bit) should be enabled and set to Cookie-based    While it may work when is IP based, BMC's recommendation is to use always cookie for better results.

This cookie should be a session cookie, meaning the time out should only occur while there's no activity, if the load balancer is not capable of doing that, the cookie will expire in the middle of doing an activity causing a 9201, if that's the case the session affinity in the load balancer should be set to a really high time, for example 12 hours. This will prevent the cookie to expire in the middle of someone's session.

The Loadbalancer behind Mid-Tier and in front of AR Server should *not* have set the sticky bit, and the Lifespan
setting (on the Mid-Tier Configuration Tool) should be enabled. This will allow us to do the balancing.

2.  As for the timeout values, the Loadbalancer timeout should be set higher than Mid-Tier's. For example, if Mid-Tier's timeout is set to 90 minutes, 95 minutes would be a good value to use on your Loadbalancer.
NOTE:   This rule applies to any device or software that sits in front of the midtier such as proxy servers or SSO solutions.

3.   Enable the OneConnect profile for F5 loadbalancers.   This means that traffic is balanced at the HTTP request level and not the TCP Connection level.

The above are recommendations that will help you solving timeouts and session handling, but the implementation of those would need your Administrator's expertise.    Please note, Loadbalancers are considered
customizations out of BMC Support. We can recommend you general guidelines when working with Loadbalancers, but its eventual troubleshooting may rely on your System Administrator.

Troubleshooting in a loadbalancer scenario:

 Add the following to the midtier's config.properties file and restart the midtier:

arsystem.response.hostip=true.   
This will add a Response header to the responses of the HTTP requests and will tell you which Midtier is responding to the requests.    These can only be seen using a network capture like Fiddler.
How to troubleshoot whether a user is staying bound to a single Mid-tier for the life of the user's session (as required), in a loadbalanced environment?
Alternatively,
Ask the user to access a specific midtier directly, bypassing the loadbalancer.    If the problem does not occur when accessing a specific midtier, the the loadbalancer configuration should be investigated. 
 

Related articles:

ARERR 9201 related to proxy server redirection

How to troubleshoot whether a user is staying bound to a single Mid-tier for the life of the user's session (as required), in a loadbalanced environment?

Related Products:

1. BMC Remedy AR System Server