UnityAPI Session not found - NLB AppServers

[Saulo_Araujo]

Hi,

I'm deploying a custom Web Service that is used to upload documents into OnBase. In summary, this web service has only one function which when called connects to OnBase, performs StoreNewDocument, and then disconnects from the current session. It is a pretty trivial UnityAPI usage and its works perfectly in our Test environment (non-balanced servers) but in the Production environment it has a strange intermittent behavior. Sometimes it performs store documents successfully and sometimes not.

Looking at Diagnostic Console logs in the Production environment I can see cases where the session id was created, performs some steps to store documents, and then disconnects from the current session, finishing the upload successfully. In another hand, there are many cases where the session id was created and the store process doesn't finish successfully. The established session id seems to get lost before completing the process (because I can't see any more logs with this session id) and the custom Web Service returns an exception. When we are faced with this behavior the Diagnostic Console starts to register messages like "Hyland.Services.SessionNotFoundException: Session not found." and "An error occurred creating a connection to data source: LocalSqlServer".

The major difference between Test and Production environments is that Production is a load-balanced environment. There is two AppServers load-balanced in a NLB. The NLB is configured for sticky sessions (Affinity single) as recommended by AppServer MRG.

Additional information, the custom Web Service is hosted by a farm server that there is no sticky session configured. They are stateless servers.

Version: OnBase EP2 19.12.5.1000

Has someone been faced with something like that? Any suggestions?

[Justin_Carlson]

One thing comes to mind - throughout the upload code are you reusing the same Hyland.Unity.Application object? This Application object most likely contains any cookies or other information that allows the sticky sessions to work. If you reconstitute/recreate the Hyland.Unity.Application object from scratch by using the Session ID to connect, this would most likely break your load balancing.

Do you use these App Servers for any other clients, like the Unity Client or Web Client? If you do, it could help test/confirm that load balancing was configured and working properly. If load balancing is properly configured and works when connecting with web clients and/or Unity Client AND the Application object for the Unity API is properly maintained, load balancing should work without too much trouble. (Obviously that is not a guarantee, just a starting point).

[Saulo_Araujo]

Hi Justin!

Thanks for answering.

Every time that my custom UploadDocument function is called a new Application object is created by calling Application.Connect and it is used until finish the upload. I'm sharing a little piece of the code below to clarify.

Yes, I do use these AppServers for Unity Client and Web Client. They are working perfectly and confirm that load balancing was configured properly. My point here is that because there is nothing between the Unity Client workstations and the AppServers NLB the sticky session is maintained for every single workstation. For the Web Client, although the WebServer is balancing too, the session is maintained because the WebServers NLB were configured to affinity single. That is my understanding.

What about the custom Web Service? Well, it is hosted by a cluster. There is a main virtual IP that receives the requests and distributes them for at least six others servers that are stateless. I think that there is an infrastructure problem between AppServer NLB and the custom Web Service cluster. When the "Session is not found" errors happen, I can see that the sessions are created but, for some reason, they are getting lost. I don't know but, as the WebServer NLB, should the Cluster be configured for sticky sessions? What do you think about it?

In a non-balanced like my Test environment, when I do debug with some breakpoints, I can see three main interactions between the custom Web Service and the App Server (looking at Diagnostic Console). The first is in Application.Connect, the second in Application.Core.Storage.StoreNewDocument, and the third in application.Disconnect. Every interaction is made by the same IP address (noticed by HTTPFoward tab on Diagnostic Console). What happens if the IP Address is changed between the three main interactions? For exemple: Application.Connect came from an IP Address (a machine chosen by the cluster) and then when Application.Core.Storage.StoreNewDocument is called the IP Address was changed. Is that make sense?

[Justin_Carlson]

So I think your setup makes sense. Since all of the code to archive is encapsulated in one method, it should always all execute from the same server/IP address, so at least in my initial reading of your setup, I don't think sticky sessions on the cluster would make a difference.

In taking a look at your code, there are some regions that are collapsed and some implicit variable types which make it tough to tell for sure.

At the beginning of your method you are creating a connection (so I assume this means getting an Application object).

In the next lines you get the document type and document type group. However, it is unclear how you are obtaining those. Are you using the application.Core.DocumentTypes collection each time to get the Document Type object? Or are you using some sort of cached object from previous executions?

If you are using cached objects (i.e. Hyland.Unity.DocumentType that was obtained while connected via a different session) this could cause issues.

I am not sure I have any other ideas or suggestions as to why you'd see the behavior that you are. You could also turn on either the Service or Web Service tab of diagnostics (the web service one is pretty chatty, so I wouldn't leave it on indefinitely in production). The service tab will log every time a web service call is made (basically every time the API calls the App Server). The Web Service tab would also log the request and response payloads of those requests. I mention this as a possible additional troubleshooting step because it may capture the actual session ID. Plus I believe you should also see that the application should interact with the App Server more than just three times, but other diagnostics tabs may not show that.  For example, it should have to reach out to the server to get the document type as well. (this would kind of confirm whether a cached object was being used for example).

[Saulo_Araujo]

Hi, @Justin Carlson 

I appreciate your help!

Because my whole code has many lines I tried to share just the main part but, if you wanna take a look, I'm sharing right now the class file here: SdkOnBaseAdapter.cs

So, I have limited access to the entire environment but I'll do the analysis as you suggest the next time.

While we are here discussing the possible reasons for the issue, I found in WebServerAdmin's MRG the following point:

What do you think? Even though we are using these balanced AppServer for Web Client and Unity Client, those clients haven't intensive usage as API access. I'm requesting the environment's ADM to check the anti-virus policies and anything else that may restart or break the communication between the custom Web Service and AppServers.

I keep you updated.

Thanks.