Overload leads to timeout errors

Sometimes integration scenarios can receive a bulk load from an application, especially when the source application uses batches as the trigger for sending out messages. This load can lead to the following non-retryable SOAP errors in the ESB:
Fault message:
Failed to enqueue deferred event "oracle.tip.esb.server.dispatch.QueueHandlerException:
Context lookup failed "[CONFIGURATION ERROR] Invalid Destination "Topics/ESB_JAVA_DEFERRED" :
javax.jms.InvalidDestinationException: Looking up java:comp/resource/esbRP/Topics/ESB_JAVA_DEFERRED:
javax.naming.NameNotFoundException: No resource named 'esbRP/Topics/ESB_JAVA_DEFERRED'found.
Please verify configuration of adminobject or the lookup string."
Make sure the topic is mapped to a jndi tree"

This behaviour is described in an Oracle note [ref: note ID 1173584.1] with the following solution: increase the timeout settings. Together with two other Oracle notes describing how to avoid BPEL errors due to adapters response time and what timeout settings in SOA can impact AIA [ref: note ID 1074227.1 and 885114.1] this leads to the following timeout settings:

• xa_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 60; recommended setting: 3600
• jms_receive_timeout in $SOA_HOME/integration/esb/config/esb_config.ini
  Default: 30; recommended setting: 300
• Also according to another Oracle note [ref: note ID 752385.1] you might want to set PingCount and PingInterval to 30 in $SOA_HOME/integration/esb/config/esb_config.ini
• syncMaxWaitTime in $SOA_HOME/bpel/domains/default/config/domain.xml
  Default: 45; recommended setting: 600
• transaction-timeout in $SOA_HOME/j2ee/oc4j_soa/application-deployments/orabpel/ejb_ob_engine/orion-ejb-jar.xml (change this value for all 6 occurences of transaction-timeout in this file)
  Default: up to 3000; recommended setting: 3600
• transaction-timeout in $SOA_HOME/j2ee/[container]/config/transaction-manager.xml (change this value for all containers: e.g. home, designtime and runtime [oc4j_soa])
  Default: from 300 to 3600; recommended setting: 7200
• Timeout in $SOA_HOME/Apache/Apache/conf/httpd.conf
  Default: 300; recommended setting: 300, this option specifies the amount of time Apache will wait for a GET, POST, PUT request and ACKs on transmissions. The default is 300 (seconds) however this may need to be increased.

The settings in the $ORACLE_HOME/integration/esb/config/esb_config.ini file apply to all ESB instances in the Oracle Home. One small bonus feature: if you have many BPEL processes deployed and you experience long waiting times in the ESB Console: put a lazyLoad property in the esb_config.ini file and the waiting time is gone [ref: Bug 7720420].
esb.console.services.lazyLoad.Allowed=true
It's a little confusing regarding the file orion-application.xml, different notes say different things, one note mentions the location in application-deployments which overrides esb_config.ini, another note mentions the location under applications which overrides esb_config.ini. Just to be safe and consistent check the orion-application.xml file in the following locations and if they are present and not commented out apply the values mentioned above for the xa_timeout, jms_receive_timeout, PingCount and PingInterval:
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/application-deployments/esb-rt/orion-application.xml
$SOA_HOME/j2ee/[ESB_RUNTIME_CONTAINER]/applications/esb-rt/META-INF/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/application-deployments/esb-dt/orion-application.xml
$SOA_HOME/j2ee/[ESB_DESIGNTIME_CONTAINER]/applications/esb-dt/META-INF/orion-application.xml

The above changes also take care of the following errors in the container logfile from $SOA_HOME/opmn/logs:
ORABPEL-05002
ORABPEL-02182
JTA transaction is not present the transaction is not in active state
Message handle error

Instead of increasing the timeout settings for the BPEL processes a more durable solution would be to throtte the inbound flow. There is an activation agent (bpel.xml) property (since 10.1.3.1), which can be used to control the speed at which the adapter posts messages to BPEL [ref: note ID 1178163.1 or Oracle® SOA Suite Best Practices Guide 10g Release 3 (10.1.3.3.0) E10971-01 December 2007]
...
    <activationAgents>
      <activationAgent partnerLink="JmsDequeuePL" ... >
          <property name="minimumDelayBetweenMessages">1000</property>
      </activationAgent>
    </activationAgents>
  </BPELProcess>
</BPELSuitcase>
This setting ensures that there at least will be 1000 milliseconds delay between two consecutive messages being posted to this BPEL process.

Actually this last best practices guide provides many valuable tips, like

• Performance tuning guidelines.
• Answers to frequently asked questions about threading.
• Answers to frequently asked questions about transactions.
• How to optimize the JVM heap. The heap size controls the amount of memory the JVM can use.
• How to relieve the dehydration store by making BPEL processes synchronous.
• Description of performance persistence parameters in bpel.xml.
• WSIF binding (localhost) and EJB binding optimization.
• The relationship among some performance settings
• The objectives and best practices for creating the BPEL cluster.
• Increasing the instanceKeyBlockSize to 100,000. Doing so decreases the frequency at which Oracle BPEL Server visits the dehydration store.

Some best practices should be considered during design and development time, like the BPEL parameters that are configured per BPEL component, whether a BPEL process should be synchronous or asynchronous and whether it should participate in a transaction or not. Other best practices are SOA Suite wide and should be part of the overall configuration, like the start-up memory settings.

Updated June 6, 2011 with LazyLoad, PingCount and PingInterval, another location of the orion-application.xml file and jms_receive_timeout to 300.

JNDI locations jms database

In the source code it's good practice to use Enterprise Information Systems (EIS) connection factories. In the Enterprise Manager this eis JNDI (Java Naming and Directory Interface) location points to the datasource JNDI location. Therefor the physical endpoint information can be kept outside the sourcecode, either in the connection factory (quick and dirty) or in a connection pool (multi-layered) to which the connection factory points to, via a datasource.

Here two examples using connection pools, one for a database connection and one JMS connection with the database as datastore.

Database connection, the source JNDI location is eis/db/XREF with the following connection factory $SOA_HOME/j2ee/oc4j_soa/application-deployments/default/DbAdapter/oc4j-ra.xml:
<connector-factory location="eis/db/XREF" connector-name="DbAdapter">
  <config-property name="xADataSourceName" value="jdbc/XREFDataSource"/>
  <config-property name="dataSourceName" value=""/>
  <config-property name="platformClassName" value="oracle.toplink.platform.database.Oracle9Platform"/>
  <config-property name="usesNativeSequencing" value="true"/>
  <config-property name="sequencePreallocationSize" value="50"/>
  <config-property name="defaultNChar" value="false"/>
  <config-property name="usesBatchWriting" value="true"/>
  <config-property name="usesSkipLocking" value="false"/>
  <connection-pooling use="none">
  </connection-pooling>
  <security-config use="none">
  </security-config>
  <connectionfactory-interface>javax.resource.cci.ConnectionFactory</connectionfactory-interface>
</connector-factory>
There is also a GUI available in the Enterprise Manager, goto instance [oc4j_soa], tab Applications and click on Default application. Then click module DbAdapter and tab Connection Factories.

The connection factory points to the JNDI location jdbc/XREFDataSource in $SOA_HOME/j2ee/oc4j_soa/config/data-sources.xml:
<managed-data-source connection-pool-name="XREFConnectionPool" jndi-name="jdbc/XREFDataSource" name="XREFDataSource"/>
<connection-pool name="XREFConnectionPool">
  <connection-factory factory-class="oracle.jdbc.xa.client.OracleXADataSource" user="aia" password="[password]" url="jdbc:oracle:thin:@[DB-HOST]:[DB-PORT]:[DB-SID]" commit-record-table-name=""/>
</connection-pool>
There is also a GUI available in the Enterprise Manager, goto instance [oc4j_soa], tab Administration and under Services go to task JDBC Resources.

That's it. Now the jms connection, the source JNDI location is eis/jms/JMSProducer with the following connection factory $SOA_HOME/j2ee/oc4j_soa/application-deployments/default/JmsAdapter/oc4j-ra.xml:
<connector-factory location="eis/jms/JMSProducer" connector-name="JmsAdapter">
  <config-property name="connectionFactoryLocation" value="java:comp/resource/JMSProducer/QueueConnectionFactories/QCF"/>
  <config-property name="factoryProperties" value=""/>
  <config-property name="acknowledgeMode" value="AUTO_ACKNOWLEDGE"/>
  <config-property name="isTopic" value="false"/>
  <config-property name="isTransacted" value="false"/>
  <config-property name="username" value="jmsuser"/>
  <config-property name="password" value="[password]"/>
  <connection-pooling use="none">
  </connection-pooling>
  <security-config use="none">
  </security-config>
  <connectionfactory-interface>oracle.tip.adapter.jms.IJmsConnectionFactory</connectionfactory-interface>
</connector-factory>
There is also a GUI available in the Enterprise Manager, goto instance [oc4j_soa], tab Applications and click on Default application. Then click module JmsAdapter and tab Connection Factories.

As you can see there is no JNDI location the connection factory refers to. This is done in the resource provider (you can see the name inside the config property connectionFactoryLocation, between the slashes after resource), the resource provider offers no Enterprise Manager GUI, only the xml file $SOA_HOME/j2ee/oc4j_soa/config/application.xml:
<resource-provider name="JMSProducer" class="oracle.jms.OjmsContext">
  <description>oc4j-jms AIASystem Queue Producer resource provider</description>
  <property name="datasource" value="jdbc/JMSDataSource" />
</resource-provider>

This resource provider points to the JNDI location jdbc/JMSDataSource in $SOA_HOME/j2ee/oc4j_soa/config/data-sources.xml:
<managed-data-source connection-pool-name="JMSConnectionPool" jndi-name="jdbc/JMSDataSource" name="JMSDataSource"/>
<connection-pool name="JMSConnectionPool">
  <connection-factory factory-class="oracle.jdbc.xa.client.OracleXADataSource" user="jmsuser" password="[password]" url="jdbc:oracle:thin:@[DB-HOST]:[DB-PORT]:[DB-SID]" commit-record-table-name=""/>
</connection-pool>
There is also a GUI available in the Enterprise Manager, goto instance [oc4j_soa], tab Administration and under Services go to task JDBC Resources.

BPELConsole session timeout

Want to disable the continuous and annoying relogins into the BPELConsole?
The solution is simple, just change the session timeout so you'll not logout automatically after some time anymore:

Login to the Enterprise Manager (em), open the oc4j container oc4j_soa and click on application orabpel, then click on module 'console' and go to tab Administration. Next click on the configuration properties - Go to Task to View/edit configuration properties for this web module. Here you can set the value of the Session Timeout (seconds): a value of -1 will disable the timeout, the default is 1200 (20 min.), 12 hours will be 43200.

As usual: the change will become effective after a restart.

This setting can be changed for all sites (em, esb, AIA), just find the right module by clicking the oc4j container oc4j_soa and click on the tab Administration. Next click on the J2EE Websites - Go to Task to Manage the J2EE websites in this OC4J instance. Click on the default-web-site and you see the list of all Web Modules, one per Application and their web-address (root-context). For the root-context BPELConsole you see application orabpel and module console, you can change the timeout for every application (em - ascontrol - ascontrol, esb - esb-dt - esbconsole, AIA - AIAApplication - AIAUserInterface) the same way as described above.

Don't forget to restart, once more:

1. Login to the Enterprise Manager (em).
2. Click the oc4j container [oc4j_soa].
3. Click on the tab Administration.
4. Click on the J2EE Websites - Go to Task to Manage the J2EE websites in this OC4J instance.
5. Click on the default-web-site.
6. You see the list of all Web Modules, click on th web module of the application you want to change the timeout of. Example click on the web module [console] of Application orabpel.
7. Click on the tab Administration.
8. Click on the configuration properties - Go to Task to View/edit configuration properties for this web module.
9. Set a new value of the Session Timeout (seconds): a value of -1 will disable the timeout, the default is 1200 (20 min.), 12 hours will be 43200.
10. Restart.

How to determine the SOA Suite MLR version?

There are many ways to determine the MLR (SOA Bundle Patchsets - Merge Label Request) version of the SOA Suite, here are three ways to find your servers MLR version:
1. Go to $SOA_HOME/bpel/bin and run obversion.sh
./obversion.sh
*****************************************************************************
Oracle BPEL Server version 10.1.3.4.0
  Build: 0
  Build time: Sun Mar 07 17:51:49 PST 2010
  Build type: release
  Source tag: PCBPEL_10.1.3.4.0MLR10_GENERIC_RELEASE
2. Similarly, you can see this same information in the $SOA_HOME/bpel/domains/<domain_name>/logs/domain.log OR $SOA_HOME/opmn/logs/default_group~<container_name>~default_group~1.log file. Upon startup, the log file will contain the build information and MLR number entries similar to the output above.
3. Third way, look for the patches in opatch lsinventory: OPatch/opatch lsinventory |egrep '8372150|8589928|9317334'
Patch 8372150: TRACKING BUG FOR CUMULATIVE MLR#8 ON TOP OF 10.1.3.4.0
Patch 8589928: TRACKING BUG FOR CUMULATIVE MLR#9 ON TOP OF 10.1.3.4.0
Patch 9317334: TRACKING BUG FOR CUMULATIVE MLR#10 ON TOP OF 10.1.3.4.0

Tips to start SOA Suite developments

In order to have all the interfaces in Oracle SOA Suite developed in an uniform way there is a need for coding standards and naming conventions. Oracle delivers the Application Integration Architecture, which includes a canonical model, a test-framework, coding standards, naming conventions and more.

Some tips when starting with Oracle SOA Suite:

• Stick to one version or patch-level for all environments. Only if a new version or MLR (patch-bundle) is needed, for example to fix a showstopper, start a project to upgrade/patch all your environments.
• Developers must have read the AIA Development Guide [ref: Oracle® Application Integration Architecture - Foundation Pack 2.5: Integration Developer's Guide Release 2.5 Part No. E16465-01 December 2009] or taken an AIA course. As for some the AIA Development Guide is not an easy and fast read, a good practice is to create a summary for those developers :) Add to this summary company-specific coding standards and naming conventions.
• Implement the components in either BPEL or ESB technology based on their purpose. All ABCSs in BPEL, all EBFs in BPEL, all EBSs in ESB, all non-soap communication with applications use ESB adapters.
• Use the AIA Service Generator for the BPEL components (ABCSs). This takes care of code for AIA ErrorHandling, CAVS test framework and sets instance title etc.
• Automate the deployments using scripts or e.g. the osbs - Oracle Soa Build Server by Marc Kelderman [ref: http://orasoa.blogspot.com/2009/04/new-oracle-soa-build-server-osbs.html]
• Set xml validation to strict on the development and test environments (except while executing stress-tests).
• Define ownership for all the EBO customizations, DVMs, XRefs, AIA ConfigurationProperties and system-wide parameters.
• When DVM or XRef technology is used for other purposes then intended, prefix the name with for example Z_.
• Define ownership of all the applications within the SOA Suite, especially naming (name per application or per module within, name the middleware communicating with or the application behind), ApplicationObjectLibrary and connection details.
• Define appropriate Service Groups in the ESB System AIASystem, for example one service group per application.
• Transformations from ABM --> EBM should be comprehensive mapping of all available attributes (All ABM Attributes supported in EBM should be mapped). Instance identifying information needs to be populated with as much information as possible in every Identification/IdentificationType element. EBM header needs to be populated fully.
• Use the EBM verbs/EBS operations with care and as intended. For example master data from the System of Record uses the sync, messages to the SoR use create/update.
• The same for EBM List messages, bundling messages lowers the number of instances, but if one message fails all the objects within this message are blocked.

Hope this helps you...