Overload leads to timeout errors

Sometimes integration scenarios can receive a bulk load from an application, especially when the source application uses batches as the trigger for sending out messages. This load can lead to the following non-retryable SOAP errors in the ESB:
Fault message:
Failed to enqueue deferred event "oracle.tip.esb.server.dispatch.QueueHandlerException:
Context lookup failed "[CONFIGURATION ERROR] Invalid Destination "Topics/ESB_JAVA_DEFERRED" :
javax.jms.InvalidDestinationException: Looking up java:comp/resource/esbRP/Topics/ESB_JAVA_DEFERRED:
javax.naming.NameNotFoundException: No resource named 'esbRP/Topics/ESB_JAVA_DEFERRED'found.
Please verify configuration of adminobject or the lookup string."
Make sure the topic is mapped to a jndi tree"

This behaviour is described in an Oracle note [ref: note ID 1173584.1] with the following solution: increase the timeout settings. Together with two other Oracle notes describing how to avoid BPEL errors due to adapters response time and what timeout settings in SOA can impact AIA [ref: note ID 1074227.1 and 885114.1] this leads to the following timeout settings:

• xa_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 60; recommended setting: 3600
• jms_receive_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 30; recommended setting: 300
• Also according to another Oracle note [ref: note ID 752385.1] you might want to set PingCount and PingInterval to 30 in $SOA_HOME/integration/esb/config/esb_config.ini
• syncMaxWaitTime in $SOA_HOME/bpel/domains/default/config/domain.xml
  Default: 45; recommended setting: 600
• transaction-timeout in $SOA_HOME/j2ee/oc4j_soa/application-deployments/orabpel/ejb_ob_engine/orion-ejb-jar.xml (change this value for all 6 occurences of transaction-timeout in this file)
  Default: up to 3000; recommended setting: 3600
• transaction-timeout in $SOA_HOME/j2ee/[container]/config/transaction-manager.xml (change this value for all containers: e.g. home, designtime and runtime [oc4j_soa])
  Default: from 300 to 3600; recommended setting: 7200
• Timeout in $SOA_HOME/Apache/Apache/conf/httpd.conf
  Default: 300; recommended setting: 300, this option specifies the amount of time Apache will wait for a GET, POST, PUT request and ACKs on transmissions. The default is 300 (seconds) however this may need to be increased.

The settings in the $ORACLE_HOME/integration/esb/config/esb_config.ini file apply to all ESB instances in the Oracle Home. One small bonus feature: if you have many BPEL processes deployed and you experience long waiting times in the ESB Console: put a lazyLoad property in the esb_config.ini file and the waiting time is gone [ref: Bug 7720420].
esb.console.services.lazyLoad.Allowed=true
It's a little confusing regarding the file orion-application.xml, different notes say different things, one note mentions the location in application-deployments which overrides esb_config.ini, another note mentions the location under applications which overrides esb_config.ini. Just to be safe and consistent check the orion-application.xml file in the following locations and if they are present and not commented out apply the values mentioned above for the xa_timeout, jms_receive_timeout, PingCount and PingInterval:
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/application-deployments/esb-rt/orion-application.xml
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/applications/esb-rt/META-INF/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/application-deployments/esb-dt/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/applications/esb-dt/META-INF/orion-application.xml

The above changes also take care of the following errors in the container logfile from $SOA_HOME/opmn/logs:
ORABPEL-05002
ORABPEL-02182
JTA transaction is not present the transaction is not in active state
Message handle error

Instead of increasing the timeout settings for the BPEL processes a more durable solution would be to throtte the inbound flow. There is an activation agent (bpel.xml) property (since 10.1.3.1), which can be used to control the speed at which the adapter posts messages to BPEL [ref: note ID 1178163.1 or Oracle® SOA Suite Best Practices Guide 10g Release 3 (10.1.3.3.0) E10971-01 December 2007]
...
    <activationAgents>
      <activationAgent partnerLink="JmsDequeuePL" ... >
          <property name="minimumDelayBetweenMessages">1000</property>
      </activationAgent>
    </activationAgents>
  </BPELProcess>
</BPELSuitcase>
This setting ensures that there at least will be 1000 milliseconds delay between two consecutive messages being posted to this BPEL process.

Actually this last best practices guide provides many valuable tips, like

• Performance tuning guidelines.
• Answers to frequently asked questions about threading.
• Answers to frequently asked questions about transactions.
• How to optimize the JVM heap. The heap size controls the amount of memory the JVM can use.
• How to relieve the dehydration store by making BPEL processes synchronous.
• Description of performance persistence parameters in bpel.xml.
• WSIF binding (localhost) and EJB binding optimization.
• The relationship among some performance settings
• The objectives and best practices for creating the BPEL cluster.
• Increasing the instanceKeyBlockSize to 100,000. Doing so decreases the frequency at which Oracle BPEL Server visits the dehydration store.

Some best practices should be considered during design and development time, like the BPEL parameters that are configured per BPEL component, whether a BPEL process should be synchronous or asynchronous and whether it should participate in a transaction or not. Other best practices are SOA Suite wide and should be part of the overall configuration, like the start-up memory settings.

Updated June 6, 2011 with LazyLoad, PingCount and PingInterval, another location of the orion-application.xml file and jms_receive_timeout to 300.