cancel
Showing results for 
Search instead for 
Did you mean: 

WebService Timeout Issue

unknown-user
Champ on-the-rise
Champ on-the-rise
Hi,

We have multiple applications connecting to the repository through Webservice API and upload content. The repository is pretty huge with approximately a million documents. These applications are facing intermittent Timeout issues while uploading the files, but there are no errors in Alfresco/Tomcat logs. The files are not that big and usually, they will be less than 1MB. Its pretty difficult to identify the root cause of this issue as there are no errors or in the logs even after enabling additional logs in Alfresco. We have suggested to increase the timeout at the client side and send the files as attachments using ContentService.writeWithAttachment() method. Here is our client log


AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.net.SocketTimeoutException: Read timed out
faultActor:
faultNode:
faultDetail:
   {http://xml.apache.org/axis/}stackTrace:java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
   at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
   at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
   at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)
   at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
   at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
   at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
   at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
   at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
   at org.apache.axis.client.Call.invoke(Call.java:2767)
   at org.apache.axis.client.Call.invoke(Call.java:2443)
   at org.apache.axis.client.Call.invoke(Call.java:2366)
   at org.apache.axis.client.Call.invoke(Call.java:1812)
   at org.alfresco.webservice.content.ContentServiceSoapBindingStub.write(ContentServiceSoapBindingStub.java:513)
   at com.choicepoint.snac.shared.alfresco.AlfrescoConnection.sendDocument(AlfrescoConnection.java:226)
   at com.choicepoint.snac.ui.service.util.AlfrescoUtils.exportDocuments(AlfrescoUtils.java:287)
   at com.choicepoint.snac.ui.service.util.AlfrescoUtils.exportDocuments(AlfrescoUtils.java:243)
   at com.choicepoint.snac.service.IndexingServiceImpl.exportDocuments(IndexingServiceImpl.java:73)
   at sun.reflect.GeneratedMethodAccessor887.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:36)
   at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:243)
   at java.beans.Statement.invoke(Statement.java:214)
   at java.beans.Expression.getValue(Expression.java:98)
   at com.choicepoint.snac.rpc.RemoteServiceServlet.process(RemoteServiceServlet.java:123)
   at com.choicepoint.snac.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:64)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
   at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
   at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
   at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
   at java.lang.Thread.run(Thread.java:619)

   {http://xml.apache.org/axis/}hostname:<sample.net>
java.net.SocketTimeoutException: Read timed out
   at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
   at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
   at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
   at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
   at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
   at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
   at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
   at org.apache.axis.client.Call.invoke(Call.java:2767)
   at org.apache.axis.client.Call.invoke(Call.java:2443)
   at org.apache.axis.client.Call.invoke(Call.java:2366)
   at org.apache.axis.client.Call.invoke(Call.java:1812)
   at org.alfresco.webservice.content.ContentServiceSoapBindingStub.write(ContentServiceSoapBindingStub.java:513)
   at com.choicepoint.snac.shared.alfresco.AlfrescoConnection.sendDocument(AlfrescoConnection.java:226)
   at com.choicepoint.snac.ui.service.util.AlfrescoUtils.exportDocuments(AlfrescoUtils.java:287)
   at com.choicepoint.snac.ui.service.util.AlfrescoUtils.exportDocuments(AlfrescoUtils.java:243)
   at com.choicepoint.snac.service.IndexingServiceImpl.exportDocuments(IndexingServiceImpl.java:73)
   at sun.reflect.GeneratedMethodAccessor887.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:36)
   at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:243)
   at java.beans.Statement.invoke(Statement.java:214)
   at java.beans.Expression.getValue(Expression.java:98)
   at com.choicepoint.snac.rpc.RemoteServiceServlet.process(RemoteServiceServlet.java:123)
   at com.choicepoint.snac.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:64)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
   at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
   at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
   at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
   at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
   at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
   at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)

We are using Alfresco v3.3. Has anyone faced this kind of a Timeout issue with Webservice API?

Update

Further analyzing the logs shows that RepositoryService.update(cml) is taking a lot of time. Basically, client updates 2 metadata fields (single text values) and upload the document. The actual upload does not take much time (hardly 2-3 secs), but most of the time (40-50secs) is spent on the update(). I checked the source code and found that this method is doing a lot of stuff like adding/removing aspects, versioning etc, but I am not sure if those processings are applicable to our case.

1) Is there any other method that we can use instead of update()? How about we call a custom REST service that just updates the metadata and use the RepositoryService to upload the file.
2) Could indexing slow down the update()? As I mentioned earlier, our content store is pretty big.


Thanks in advance
3 REPLIES 3

soutaga
Champ in-the-making
Champ in-the-making
Hi,

I have the some problem in the same version.

Anyone have an idea How I can pass this problem?

Thanks

Best Regards

unknown-user
Champ on-the-rise
Champ on-the-rise
In our case, file system was the issue (as per the infrastructure team). We were using a shared NAS drive for the content store and we moved to a SAN which resolved the issue. Atleast we havent seen any instances of WebService Timeout issues in the past 4-5 months.

Hope this helps

soutaga
Champ in-the-making
Champ in-the-making
Hi,
Thanks for the information.

We changed timeout to 120 seconds (WebServiceFactory.setTimeoutMilliseconds(120000):smileywink: and our FS have 500GB and we had only 40/50GB free so we increase to 800GB
Timeout problem disappear, but our upload process still get lots of time.

For a file with 30 Kb, 2 aspects, 5 properties (Metadata) it takes like 20/40 seconds and the same operation 1 year ago took like 3/5 seconds.

We use MySql DB and I notice that alf_node_properties have more than 10 millions of lines and 2,6 Gb of space.

We have to upload like 6/7 Gb per week to Alfresco and at this time we have a backlog because this situation.

If you have some advice how to improve upload it will be grateful.

Thanks