Friday, June 9, 2017

DVM Validation on a SOA Suite Domain Causes High CPU Utilization: Analysis and Solution


Problem statement


Once upon a time, my colleagues asked me an interesting question. After each update of the MDS on a large production environment, a significant performance degradation is observed. CPU load on the SOA Suite servers is increased, respectively.

The system automated leveraging Oracle SOA Suite makes extensive use of Domain Value Maps (DVM), which is intended to map from one vocabulary used in a given domain to another vocabulary used in a different domain. For example, one domain may represent Russian rubles as 'RUR', while another domain may represent those as '810' (see ISO 4217). Business users can edit DVMs at runtime using Oracle SOA Composer.

And while DVMs are not so large everything works fine, but if there are some large DVM files (> 200-300 KB), the above strange behaviour may be observed.

Let's have a look at the thread dump collected during the observation.

"[ACTIVE] ExecuteThread: '57' for queue: 'weblogic.kernel.Default (self-tuning)'" #159 daemon prio=9 os_prio=2 tid=0x0000000066bbd800 nid=0x28ac runnable [0x0000000079188000]
java.lang.Thread.State: RUNNABLE
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:600)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.xpath.ComparisonExpr.evaluate(XSLExpr.java:1743)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XPathPredicate.filter(XPathPredicate.java:349)
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:627)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2762)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2722)
at oracle.tip.dvm.sdk.util.XMLUtil.isDVMDocumentValid(XMLUtil.java:211)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:202)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:130)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:217)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:133)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:252)
at sun.reflect.GeneratedMethodAccessor1815.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.xml.xpath.XSLExtFunctions.callStaticMethod(XSLExtFunctions.java:115)
at oracle.xml.xpath.XPathExtFunction.evaluateMethod(XPathExtFunction.java:422)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:347)
at oracle.xml.xslt.XSLValueOf.processAction(XSLValueOf.java:152)
at oracle.xml.xslt.XSLNode.processChildren(XSLNode.java:559)
at oracle.xml.xslt.XSLTemplate.processAction(XSLTemplate.java:278)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:706)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:665)
at oracle.xml.xslt.XSLProcessor.processXSL(XSLProcessor.java:401)
at oracle.xml.jaxp.JXTransformer.transform(JXTransformer.java:578)
at ...


"[ACTIVE] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)'" #158 daemon prio=9 os_prio=2 tid=0x0000000066bbc800 nid=0x4f4 runnable [0x0000000078f78000]
java.lang.Thread.State: RUNNABLE
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:600)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.xpath.ComparisonExpr.evaluate(XSLExpr.java:1743)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XPathPredicate.filter(XPathPredicate.java:349)
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:627)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2762)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2722)
at oracle.tip.dvm.sdk.util.XMLUtil.isDVMDocumentValid(XMLUtil.java:211)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:202)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:130)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:217)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:133)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:252)
at sun.reflect.GeneratedMethodAccessor1815.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.xml.xpath.XSLExtFunctions.callStaticMethod(XSLExtFunctions.java:115)
at oracle.xml.xpath.XPathExtFunction.evaluateMethod(XPathExtFunction.java:422)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:347)
at oracle.xml.xslt.XSLValueOf.processAction(XSLValueOf.java:152)
at oracle.xml.xslt.XSLNode.processChildren(XSLNode.java:559)
at oracle.xml.xslt.XSLTemplate.processAction(XSLTemplate.java:278)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:706)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:665)
at oracle.xml.xslt.XSLProcessor.processXSL(XSLProcessor.java:401)
at oracle.xml.jaxp.JXTransformer.transform(JXTransformer.java:578)
at ...

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" #104 daemon prio=9 os_prio=2 tid=0x000000005edf3800 nid=0x2e30 runnable [0x000000006e059000]
java.lang.Thread.State: RUNNABLE
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XPathPredicate.filter(XPathPredicate.java:349)
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:627)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2762)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2722)
at oracle.tip.dvm.sdk.util.XMLUtil.isDVMDocumentValid(XMLUtil.java:211)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:202)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:130)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:217)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:133)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:252)
at sun.reflect.GeneratedMethodAccessor1815.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.xml.xpath.XSLExtFunctions.callStaticMethod(XSLExtFunctions.java:115)
at oracle.xml.xpath.XPathExtFunction.evaluateMethod(XPathExtFunction.java:422)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:347)
at oracle.xml.xslt.XSLValueOf.processAction(XSLValueOf.java:152)
at oracle.xml.xslt.XSLNode.processChildren(XSLNode.java:559)
at oracle.xml.xslt.XSLTemplate.processAction(XSLTemplate.java:278)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:706)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:665)
at oracle.xml.xslt.XSLProcessor.processXSL(XSLProcessor.java:401)
at oracle.xml.jaxp.JXTransformer.transform(JXTransformer.java:578)
at ...

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" #25 daemon prio=9 os_prio=2 tid=0x000000005bbaa000 nid=0x2acc runnable [0x0000000062268000]
java.lang.Thread.State: RUNNABLE
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:600)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.xpath.ComparisonExpr.evaluate(XSLExpr.java:1743)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:514)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:524)
at oracle.xml.xpath.XPathPredicate.filter(XPathPredicate.java:349)
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:627)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:1102)
at oracle.xml.xpath.PathExpr.evaluate(PathExpr.java:808)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2762)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2722)
at oracle.tip.dvm.sdk.util.XMLUtil.isDVMDocumentValid(XMLUtil.java:211)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:202)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:130)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:217)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:133)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:252)
at sun.reflect.GeneratedMethodAccessor1815.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.xml.xpath.XSLExtFunctions.callStaticMethod(XSLExtFunctions.java:115)
at oracle.xml.xpath.XPathExtFunction.evaluateMethod(XPathExtFunction.java:422)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:347)
at oracle.xml.xslt.XSLValueOf.processAction(XSLValueOf.java:152)
at oracle.xml.xslt.XSLNode.processChildren(XSLNode.java:559)
at oracle.xml.xslt.XSLTemplate.processAction(XSLTemplate.java:278)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:706)
at oracle.xml.xslt.XSLStylesheet.execute(XSLStylesheet.java:665)
at oracle.xml.xslt.XSLProcessor.processXSL(XSLProcessor.java:401)
at oracle.xml.jaxp.JXTransformer.transform(JXTransformer.java:578)
at ...


We can see, the threads are processing some XML data and validating DVMs. When DVM files have been validated, new requests to SOA Suite doesn't fulfill any validation again until a composite application or MDS have been re-deployed.

Analysis


A simple composite for testing purposes only was developed. An asynchronous BPEL-process was included into the composite. The process just makes an invocation of the dvm:lookupValue function over a large (> 500 KB) DVM.


A single DVM is used.


In order to test my hypothesis about the impact of the number of threads on the performance, a JMH-based benchmark was developed. The benchmark sent a hard coded SOAP-request to the server and got responses from.

The following results can be shared with you (the benchmark worked in the Single Shot mode):

One thread:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       35258,785          ms/op

Two threads:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       44261,669          ms/op

Four threads:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       58845,936          ms/op

The composite application was re-deployed on the server before each run.

It seems there is a bug in the code of Oracle SOA Suite: single DVM file is being validated by a number of threads simultaneously, and this is the reason for the observed performance degradation.

Solution


A service request (SR) for Oracle technical support was opened. They rapidly issued a patch for our version of SOA Suite. The patch is available on mysupport.oracle.com. Please, use the following link:

Patch 26177605 - 11.1.1.7.8 - Large DVMs cause high CPU and stuck threads after startup or redeploy

I repeated the experiment.

Interim patches (2) :

Patch  26177605     : applied on Wed Jun 07 16:53:02 MSK 2017
Unique Patch ID:  21330144
   Created on 6 Jun 2017, 11:08:12 hrs PST8PDT
   Bugs fixed:
     26177605
   This patch overlays patches:
     20900797
   This patch needs patches:
     20900797
   as prerequisites


One thread:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       30889.307          ms/op

Two threads:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       35392.957          ms/op

Four threads:
Benchmark                    (port)   (server)  Mode  Cnt      Score   Error  Units
DVMisValidBenchmark.message    8001  localhost    ss       31056.566          ms/op

I think, my post can be completed here, but let's have a look at the thread dump.


"[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1db0095800 nid=0x7673 waiting for monitor entry [0x00007f1d4b7af000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:198)
- waiting to lock <0x00000007eafc5828> (a oracle.fabric.common.dvm.DVM)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:128)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:274)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:175)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.DVMExtFunctions$LookupValue.call(DVMExtFunctions.java:80)
at oracle.fabric.common.xml.xpath.FabricXPathFunctionWrapper.evaluate(FabricXPathFunctionWrapper.java:84)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.;ava:157)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:126)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:264)
at oracle.xml.xpath.JXPathExpression.evaluate(JXPathExpression.java:203)
at ...

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1da0008800 nid=0x5fc5 waiting for monitor entry [0x00007f1d55834000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:198)
- waiting to lock <0x00000007eafc5828> (a oracle.fabric.common.dvm.DVM)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:128)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:274)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:175)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.DVMExtFunctions$LookupValue.call(DVMExtFunctions.java:80)
at oracle.fabric.common.xml.xpath.FabricXPathFunctionWrapper.evaluate(FabricXPathFunctionWrapper.java:84)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:157)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:126)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:264)
at oracle.xml.xpath.JXPathExpression.evaluatge(JXPathExpression.java:203)
at ...

"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1db000f800 nid=0x5193 waiting for monitor entry [0x00007f1d55d39000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:198)
- waiting to lock <0x00000007eafc5828> (a oracle.fabric.common.dvm.DVM)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:128)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:274)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:175)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.DVMExtFunctions$LookupValue.call(DVMExtFunctions.java:80)
at oracle.fabric.common.xml.xpath.FabricXPathFunctionWrapper.evaluate(FabricXPathFunctionWrapper.java:84)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:157)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:126)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:264)
at oracle.xml.xpath.JXPathExpression.evaluate(JXPathExpression.java:203)


"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1d74011000 nid=0x5fc6 runnable [0x00007f1d5512d000]
java.lang.Thread.State: RUNNABLE
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:607)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:564)
at oracle.xml.xpath.PathExpr.evaluate(XSLNodeSetExpr.java:873)
at oracle.xml.xpath.ComparisonExpr.evaluate(XSLExpr.java:1497)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:420)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:473)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:420)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:473)
at oracle.xml.xpath.XSLExprBase.testBooleanExpr(XSLExprBase.java:420)
at oracle.xml.xpath.AndExpr.evaluate(XSLExpr.java:473)
at oracle.xml.xpath.XPathPredicate.filter(XPathPredicate.java:264)
at oracle.xml.xpath.XPathChildAxis.getNodeList(XPathAxis.java:627)
at oracle.xml.xpath.XPathStep.evaluate(XPathStep.java:564)
at oracle.xml.xpath.PathExpr.evaluate(XSLNodeSetExpr.java:873)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2562)
at oracle.xml.parser.v2.XMLNode.selectNodes(XMLNode.java:2522)
at oracle.tip.dvm.sdk.util.XMLUtil.isDVMDocumentValid(XMLUtil.java:197)
at oracle.tip.dvm.entity.DVMRTObject.validateDVM(DVMRTObject.java:200)
- locked <0x00000007eafc5828> (a oracle.fabric.common.dvm.DVM)
at oracle.tip.dvm.entity.DVMRTObject.(DVMRTObject.java:128)
at oracle.tip.dvm.DVMManagerImpl.getDVMRTObject(DVMManagerImpl.java:274)
at oracle.tip.dvm.DVMManagerImpl.lookupValue(DVMManagerImpl.java:175)
at oracle.tip.dvm.LookupValue.lookupValue(LookupValue.java:95)
at oracle.tip.dvm.DVMExtFunctions$LookupValue.call(DVMExtFunctions.java:80)
at oracle.fabric.common.xml.xpath.FabricXPathFunctionWrapper.evaluate(FabricXPathFunctionWrapper.java:84)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:157)
at oracle.xml.xpath.JXPathContext$JXFunction.invoke(JXPathContext.java:126)
at oracle.xml.xpath.XPathExtFunction.evaluate(XPathExtFunction.java:264)
at oracle.xml.xpath.JXPathExpression.evaluate(JXPathExpression.java:203)
at ...


After this information, I started to believe in IBM's talks about some performance degradation of Intel processors under heavy load. My server has a large number of cores (more than handled requests), each thread validating DVM has no contention on others, but when the number of threads increases, the overall performance falls. If you have a chance, please check the situation on your Power or LinuxONE and share with us.

Would you like to give a 'Like'? Please follow me on Twitter!