Overload leads to timeout errors

Sometimes integration scenarios can receive a bulk load from an application, especially when the source application uses batches as the trigger for sending out messages. This load can lead to the following non-retryable SOAP errors in the ESB:
Fault message:
Failed to enqueue deferred event "oracle.tip.esb.server.dispatch.QueueHandlerException:
Context lookup failed "[CONFIGURATION ERROR] Invalid Destination "Topics/ESB_JAVA_DEFERRED" :
javax.jms.InvalidDestinationException: Looking up java:comp/resource/esbRP/Topics/ESB_JAVA_DEFERRED:
javax.naming.NameNotFoundException: No resource named 'esbRP/Topics/ESB_JAVA_DEFERRED'found.
Please verify configuration of adminobject or the lookup string."
Make sure the topic is mapped to a jndi tree"

This behaviour is described in an Oracle note [ref: note ID 1173584.1] with the following solution: increase the timeout settings. Together with two other Oracle notes describing how to avoid BPEL errors due to adapters response time and what timeout settings in SOA can impact AIA [ref: note ID 1074227.1 and 885114.1] this leads to the following timeout settings:

• xa_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 60; recommended setting: 3600
• jms_receive_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 30; recommended setting: 300
• Also according to another Oracle note [ref: note ID 752385.1] you might want to set PingCount and PingInterval to 30 in $SOA_HOME/integration/esb/config/esb_config.ini
• syncMaxWaitTime in $SOA_HOME/bpel/domains/default/config/domain.xml
  Default: 45; recommended setting: 600
• transaction-timeout in $SOA_HOME/j2ee/oc4j_soa/application-deployments/orabpel/ejb_ob_engine/orion-ejb-jar.xml (change this value for all 6 occurences of transaction-timeout in this file)
  Default: up to 3000; recommended setting: 3600
• transaction-timeout in $SOA_HOME/j2ee/[container]/config/transaction-manager.xml (change this value for all containers: e.g. home, designtime and runtime [oc4j_soa])
  Default: from 300 to 3600; recommended setting: 7200
• Timeout in $SOA_HOME/Apache/Apache/conf/httpd.conf
  Default: 300; recommended setting: 300, this option specifies the amount of time Apache will wait for a GET, POST, PUT request and ACKs on transmissions. The default is 300 (seconds) however this may need to be increased.

The settings in the $ORACLE_HOME/integration/esb/config/esb_config.ini file apply to all ESB instances in the Oracle Home. One small bonus feature: if you have many BPEL processes deployed and you experience long waiting times in the ESB Console: put a lazyLoad property in the esb_config.ini file and the waiting time is gone [ref: Bug 7720420].
esb.console.services.lazyLoad.Allowed=true
It's a little confusing regarding the file orion-application.xml, different notes say different things, one note mentions the location in application-deployments which overrides esb_config.ini, another note mentions the location under applications which overrides esb_config.ini. Just to be safe and consistent check the orion-application.xml file in the following locations and if they are present and not commented out apply the values mentioned above for the xa_timeout, jms_receive_timeout, PingCount and PingInterval:
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/application-deployments/esb-rt/orion-application.xml
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/applications/esb-rt/META-INF/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/application-deployments/esb-dt/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/applications/esb-dt/META-INF/orion-application.xml

The above changes also take care of the following errors in the container logfile from $SOA_HOME/opmn/logs:
ORABPEL-05002
ORABPEL-02182
JTA transaction is not present the transaction is not in active state
Message handle error

Instead of increasing the timeout settings for the BPEL processes a more durable solution would be to throtte the inbound flow. There is an activation agent (bpel.xml) property (since 10.1.3.1), which can be used to control the speed at which the adapter posts messages to BPEL [ref: note ID 1178163.1 or Oracle® SOA Suite Best Practices Guide 10g Release 3 (10.1.3.3.0) E10971-01 December 2007]
...
    <activationAgents>
      <activationAgent partnerLink="JmsDequeuePL" ... >
          <property name="minimumDelayBetweenMessages">1000</property>
      </activationAgent>
    </activationAgents>
  </BPELProcess>
</BPELSuitcase>
This setting ensures that there at least will be 1000 milliseconds delay between two consecutive messages being posted to this BPEL process.

Actually this last best practices guide provides many valuable tips, like

• Performance tuning guidelines.
• Answers to frequently asked questions about threading.
• Answers to frequently asked questions about transactions.
• How to optimize the JVM heap. The heap size controls the amount of memory the JVM can use.
• How to relieve the dehydration store by making BPEL processes synchronous.
• Description of performance persistence parameters in bpel.xml.
• WSIF binding (localhost) and EJB binding optimization.
• The relationship among some performance settings
• The objectives and best practices for creating the BPEL cluster.
• Increasing the instanceKeyBlockSize to 100,000. Doing so decreases the frequency at which Oracle BPEL Server visits the dehydration store.

Some best practices should be considered during design and development time, like the BPEL parameters that are configured per BPEL component, whether a BPEL process should be synchronous or asynchronous and whether it should participate in a transaction or not. Other best practices are SOA Suite wide and should be part of the overall configuration, like the start-up memory settings.

Updated June 6, 2011 with LazyLoad, PingCount and PingInterval, another location of the orion-application.xml file and jms_receive_timeout to 300.

Instance wide XML manipulations

If the input file is a plain xml file without namespaces declared the following error occurs (at least when xml validation is set to 'strict'):
<invalidVariables xmlns="http://schemas.oracle.com/bpel/extension"><part name="code">
<code>9710</code></part><part name="summary">
<summary>Invalid xml document.
According to the xml schemas, the xml document is invalid. The reason is: Error::cvc-elt.1: Cannot find the declaration of element 'TestXSD'.
Please make sure that the xml document is valid against your schemas.</summary></part></invalidVariables>

Solution: create in the consumer routing service (the component after the file-reader, postfixed with _RS) the following transformation, first check if the wanted namespace is included in the namespace declarations, otherwise map one field to get all the namespaces declared. Then add the following code in the xsl:
<xsl:stylesheet version="1.0" xmlns:test="http://www.mydomain.com/TestXSD">
<xsl:template match="*">
  <xsl:element name="test:{local-name()}" namespace="http://www.mydomain.com/TestXSD">
    <xsl:apply-templates select="@*|node()"/>
  </xsl:element>
</xl:template>

This code is based on the "identity" template, which was introduced with an example in the XSLT Recommendation itself [ref: www.w3.org]:
<xsl:template match="@*|node()">
  <xsl:copy>
    <xsl:apply-templates select="@*|node()"/>
  </xsl:copy>
</xsl:template>

Some explanation here: "node()" is shorthand for child::node(). That is node() matches all nodes (element, text, comment, processing-instruction) being children of other nodes but not attributes since they are not considered children of their parent element. To find all nodes in order to copy them, we need to match both nodes being children of other nodes, node(), and attributes, @*, that is: match="@*|node()".

This template copies everything recursive from source to target, all the attributes and nodes. Copying everything in itself is no fun but becomes very useful when we add exceptions or conditions to the copying. Here two transformations that copy all the nodes and attributes if:

• They have content: RemoveEmptyNodes.xsl
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="@*|node()">
    <xsl:if test=". != ''">
      <xsl:copy>
        <xsl:apply-templates select="@*|node()"/>
      </xsl:copy>
    </xsl:if>
  </xsl:template>
</xsl:stylesheet>
• They have content or the node contains an attribute with content: RemoveEmptyNodesKeepAttributes.xsl
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="@*|node()">
    <xsl:if test=". != '' or ./@* != ''">
      <xsl:copy>
        <xsl:apply-templates select="@*|node()"/>
      </xsl:copy>
    </xsl:if>
  </xsl:template>
</xsl:stylesheet>

These templates can replace one of Oracle's best practices [ref: Oracle® Application Integration Architecture - Foundation Pack 2.5: Integration Developer's Guide Release 2.5 Part No. E16465-01 December 2009], adding the RemoveEmptyNodes transformation after the main mapping fulfills:

Transformation logic and XPath should be designed with the possibility of missing or empty elements
<xsl:if test="normalize-space(Line3/text())">
  <Line3>
    <xsl:value-of select="Line3/text()"/>
  </Line3>
</xsl:if>

The above templates can save you the time-consuming conditional mapping per element. Just don't forget if the mapping contains empty placeholders (like EBMHeader/EBMTracking/ExecutionUnitID) then fill these first with an assign before calling the RemoveEmptyNodes. Also the element EBMHeader/DataArea/[verb] is a required field. This can be solved with both templates:

• RemoveEmptyNodes.xsl: include in the template an exclusion of this element (the [verb] is Create here):
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:corecom="http://xmlns.oracle.com/EnterpriseObjects/Core/Common/V2">
  <xsl:template match="@*|node()">
    <xsl:if test=". != ''">
      <xsl:copy>
        <xsl:apply-templates select="@*|node()"/>
      </xsl:copy>
    </xsl:if>
  </xsl:template>
  <xsl:template match="corecom:Create">
    <xsl:copy-of select="."/>
  </xsl:template>
</xsl:stylesheet>
• RemoveEmptyNodesKeepAttributes.xsl: fill an optional attribute of this field, although not the most elegant solution...

WSDL artifact errors while deploying

You'll already know that the Oracle JDeveloper contains a nice tool to create the base-mapping (XSL Transformation), unfortunately advanced logic within the mapping get screwed up by this JDeveloper pretty often. Therefor use an advanced text-editor with xml highlighting instead after you finish the base-mapping in JDeveloper. Or stick to the Source-view, no more Design-view.

This leads me to write about a more serious issue with JDeveloper: during development of the integrations all the developers working with ESB components need to have identical workspaces in their JDeveloper. If not, you might run into unwanted errors when deploying. This is known by Oracle and described in a note [ref: note ID 875558.1]:

Applies to: Oracle ESB - Version: 10.1.3.4 to 10.1.3.4.9. This problem can occur on any platform.
Symptoms: When an ESB project is registered using JDeveloper and then moved to a new JDEV Workspace with a different Workspace name, re-registration of the ESB project and its related artifacts (XSL Stylesheets, XML Schemas, WSDL files) might fail.
Cause: ESB artifacts designed with JDeveloper were moved to a different JDeveloper workspace with a new name.
Solution: DO NOT move ESB projects to a different JDEV workspace name. This is not supported by Oracle ESB in the current version.
Explanation: The way ESB identifies a file update in a project at runtime is by using the convention <workspacename>_<projectname>/<filename>. If the user is changing the workspace name, but keeps the project name and file name the same, it will not be possible for JDev to update the related artifacts in the slide repository because the identifier has changed - external references in form of esb:// were created. This is a known limitation.

You might also run into an error while deploying BPEL components that contain a referral to an ESB component, for example in the transformation the target is the ESB component WSDL. This url in the xsl contains the <workspace>, if in development this <workspace> differs from the actual deployed ESB component you might end up seeing the BPEL component in the BPELConsole, but not in the ESB console in BPELSystem. The advice here is: use always the same <workspace> in the JDeveloper, for example let the whole development team develop all ESB components (Projects) under Application AIASystem.

Talking about deployments, there is also a known bug in a jar-file, which is often used by automated deployments, see the note [ref: note ID 1308408.1] which describes the following error:
java.lang.Exception: Programming Error : Adapter Service <WriteData> does not have an associated WSDL artifact
at oracle.tip.esb.jdev.artifacts.model.ESBAdapterServiceArtifactContent.getDeploymentProperties(ESBAdapterServiceArtifactContent.java:159)
For example if you use osbs - Oracle Soa Build Server by Marc Kelderman [ref: http://orasoa.blogspot.com/2009/04/new-oracle-soa-build-server-osbs.html], you'll might receive the error as described in the note above. The solution here is to get from Oracle patch 11871410, extract the file ESBMetadataMigration.jar from the deployments.zip file in this patch and put this jar file in the location mentioned in the obbuild.sh script.