Tuesday, October 6, 2015

ORDS Java heap space OutOfMemoryError

I recently ran into a problem with an Apex application running on ORDS on Tomcat. The application has a page with a custom tabular form (built using the apex_item package). When this page is submitted, the form values are stored in the "g_fxx" arrays (g_f01, g_f02, etc).

The problem was that when the number rows (and therefore the number of elements in the arrays) got too big, the server would respond with a HTTP 500 - Internal Server Error message.

The first thing to do in this case is to check the Tomcat logs at /usr/share/tomcat7/latest/logs to see what the real error message is. Tomcat generates one log file per day, and the log files have the naming format catalina.yyyy-mm-dd.log.

Opening the relevant log file, I found several occurrences of this error message (truncated for brevity):

oracle.dbtools.rt.web.WebErrorResponse internalError
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
at oracle.jdbc.driver.T4CCallableStatement.doOall8(...)



So, Tomcat/ORDS was basically running out of memory when processing a page submission with dozens or hundreds of values (even though in my case the values were all contained in a few g_fxx arrays).

I googled a bit and found a recent post from the Apex forums that deals with the same issue. In my case I'm using ORDS 2.0.10, but it appears that the same issue can be experienced with ORDS 3.0.1.

Given that in this particular case my server only has 1 GB of memory, perhaps it's not that strange that the Java process would run out of memory (after all, the server is also running Oracle XE, Tomcat and Apache...).

To increase the memory available to Tomcat, you need to create a file called setenv.sh in the Tomcat bin directory, and make it executable.

nano /usr/share/tomcat7/latest/bin/setenv.sh
# paste the below text and save
chmod +x /usr/share/tomcat7/latest/bin/setenv.sh
service tomcat restart

Here is the content I put in the setenv.sh file (based on this):

#! /bin/sh

export CATALINA_OPTS="$CATALINA_OPTS -Xms128m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx512m"
export CATALINA_OPTS="$CATALINA_OPTS -server"


This sets the initial memory to 128MB and the maximum memory to 512MB (more info). The -server flag is to optimize execution when Java runs as a server (more info).

Remember to restart Tomcat for the new settings to take effect.

4 comments:

Mark Lancaster said...

Hi Morten

I had the same issue with ORDS 2.x about 18 months ago, at a clients site.

Similar issue, with a tabular form with a large number of editable fields.
Like you we increased the memory, but even that wasn't enough.

Tested reducing the number of editable fields (rows), which made the issue go away, but wasn't an viable solution for the particular page.
Logged a SR with Oracle Support, who said the issue was with Java memory allocation and no solution was available at the time.

Ended up having to switch back to Oracle HTTP server with mod_plsql, which resolved the issue.

We did limited testing due to time constraints, I would have liked to see if making the max length for input items smaller affected memory allocation or not.

I had hoped the issue was resolved in ORDS 3.x, but your blog indicates it's still an issue.

Regards

Mark

Anonymous said...

I can also advise that I have run into the problem too, in my case it would fail on page where I would post base64 encoded files via f0x arrays. I came across the following post at the time which may have suggested it was a memory deallocation issue with associative arrays with the flavour of jdk in use: https://community.oracle.com/thread/1776427?start=0

Matt

Gianluigi Trento said...

Hi Morten,
we use BIRT as report engine for our Apex application and we had many problems with "Java heap space".
It seems that today we finally found the solution: the 12c version of Oracle JDBC.
With Java JDK 7 must be used ojdbc7.jar not ojdbc6.jar.
No more problems with BIRT Designer, next week will test Tomcat 7 and 8.

Regards,
Gianluigi

Manuel said...

Hi Morton,
thanks for your Tip.

You suggest the -Xms128m and -Xmx512m values per ORDS or per connectet database?
We use one ORDS for conntecd with two PDBs.
And we run 3 different ORDS on one tomcat

Regards,
Manu