Monday

ORA-01000: maximum open cursors exceeded



     You get this error when a user of a host program attempts to open more cursors than they are allowed. 

     The number of cursors allowed is dictated by the OPEN_CURSORS initialization parameter, and this quota can be eaten up by both implicit and explicit cursors. If you run into this error, there is a possibility that there is a bug in your application. Perhaps you’ve got an open cursor statement within a loop and you do not have a matching close cursor, and as a result your code is bleeding cursors all over the place.

     However, it is possible that the OPEN_CURSORS number is just too low for the needs of your application and has to be upped. The default value is 50; however, the only factor limiting how high this number can go – 300, 1000, 2000 even – is what the operating system can take. However, it may be unwise to choose to change the OPEN_CURSORS parameter too steeply, rather than examine your code for leaks. 

     ORA-01000, the maximum-open-cursors error, is an extremely common error in Oracle database development. In the context of Java, it happens when the application attempts to open more ResultSets than there are configured cursors on a database instance.

Common causes are:

Configuration mistake

You have more threads in your application querying the database than cursors on the DB. One case is where you have a connection and thread pool larger than the number of cursors on the database.

You have many developers or applications connected to the same DB instance (which will probably include many schemas) and together you are using too many connections.

Solution:

Increasing the number of cursors on the database (if resources allow) or
Decreasing the number of threads in the application.

Cursor leak

The applications is not closing ResultSets (in JDBC) or cursors (in stored procedures on the database)

Solution:

Cursor leaks are bugs; increasing the number of cursors on the DB simply delays the inevitable failure. Leaks can be found using static code analysis, JDBC or application-level logging, and database monitoring.
Background

This section describes some of the theory behind cursors and how JDBC should be used. If you don't need to know the background, you can skip this and go straight to 'Eliminating Leaks'.

What is a cursor?

A cursor is a resource on the database that holds the state of a query, specifically the position where a reader is in a ResultSet. Each SELECT statement has a cursor, and PL/SQL stored procedures can open and use as many cursors as they require. You can find out more about cursors on Orafaq.

A database instance typically serves several different schemas, many different users each with multiple sessions. To do this, it has a fixed number of cursors available for all schemas, users and sessions. When all cursors are open (in use) and request comes in that requires a new cursor, the request fails with an ORA-010000 error.

Finding and setting the number of cursors

The number is normally configured by the DBA on installation. The number of cursors currently in use, the maximum number and the configuration can be accessed in the Administrator functions in Oracle SQL Developer. From SQL it can be set with:

ALTER SYSTEM SET OPEN_CURSORS=1337 SID='*' SCOPE=BOTH;
Relating JDBC in the JVM to cursors on the DB

The JDBC objects below are tightly coupled to the following database concepts:

JDBC Connection is the client representation of a database session and provides database transactions. A connection can have only a single transaction open at any one time (but transactions can be nested)
A JDBC ResultSet is supported by a single cursor on the database. When close() is called on the ResultSet, the cursor is released.
A JDBC CallableStatement invokes a stored procedure on the database, often written in PL/SQL. The stored procedure can create zero or more cursors, and can return a cursor as a JDBC ResultSet.
JDBC is thread safe: It is quite OK to pass the various JDBC objects between threads.

For example, you can create the connection in one thread; another thread can use this connection to create a PreparedStatement and a third thread can process the result set. The single major restriction is that you cannot have more than one ResultSet open on a single PreparedStatement at any time. See Does Oracle DB support multiple (parallel) operations per connection?

Note that a database commit occurs on a Connection, and so all DML (INSERT, UPDATE and DELETE's) on that connection will commit together. Therefore, if you want to support multiple transactions at the same time, you must have at least one Connection for each concurrent Transaction.

Closing JDBC objects

A typical example of executing a ResultSet is:

Statement stmt = conn.createStatement();
try {
    ResultSet rs = stmt.executeQuery( "SELECT FULL_NAME FROM EMP" );
    try {
        while ( rs.next() ) {
            System.out.println( "Name: " + rs.getString("FULL_NAME") );
        }
    } finally {
        try { rs.close(); } catch (Exception ignore) { }
    }
} finally {
    try { stmt.close(); } catch (Exception ignore) { }
}


*Note how the finally clause ignores any exception raised by the close():

If you simply close the ResultSet without the try {} catch {}, it might fail and prevent the Statement being closed
We want to allow any exception raised in the body of the try to propagate to the caller. If you have a loop over, for example, creating and executing Statements, remember to close each Statement within the loop.
In Java 7, Oracle has introduced the AutoCloseable interface which replaces most of the Java 6 boilerplate with some nice syntactic sugar.

Holding JDBC objects

JDBC objects can be safely held in local variables, object instance and class members. It is generally better practice to:

Use object instance or class members to hold JDBC objects that are reused multiple times over a longer period, such as Connections and PreparedStatements
Use local variables for ResultSets since these are obtained, looped over and then closed typically within the scope of a single function.
There is, however, one exception: If you are using EJBs, or a Servlet/JSP container, you have to follow a strict threading model:

Only the Application Server creates threads (with which it handles incoming requests)
Only the Application Server creates connections (which you obtain from the connection pool)
When saving values (state) between calls, you have to be very careful. Never store values in your own caches or static members - this is not safe across clusters and other weird conditions, and the Application Server may do terrible things to your data. Instead use stateful beans or a database.
In particular, never hold JDBC objects (Connections, ResultSets, PreparedStatements, etc) over different remote invocations - let the Application Server manage this. The Application Server not only provides a connection pool, it also caches your PreparedStatements.
Eliminating leaks

There are a number of processes and tools available for helping detect and eliminating JDBC leaks:

During development - catching bugs early is by far the best approach:

Development practices

Good development practices should reduce the number of bugs in your software before it leaves the developer's desk. Specific practices include:
Pair programming, to educate those without sufficient experience
Code reviews because many eyes are better than one
Unit testing which means you can exercise any and all of your code base from a test tool which makes reproducing leaks trivial
Use existing libraries for connection pooling rather than building your own
Static Code Analysis: Use a tool like the excellent Findbugs to perform a static code analysis. This picks up many places where the close() has not been correctly handled. Findbugs has a plugin for Eclipse, but it also runs standalone for one-offs, has integrations into Jenkins CI and other build tools

At runtime:

Holdability and commit

If the ResultSet holdability is ResultSet.CLOSE_CURSORS_OVER_COMMIT, then the ResultSet is closed when the Connection.commit() method is called. This can be set using Connection.setHoldability() or by using the overloaded Connection.createStatement() method.

Logging at runtime

Put good log statements in your code. These should be clear and understandable so the customer, support staff and teammates can understand without training. They should be terse and include printing the state/internal values of key variables and attributes so that you can trace processing logic. Good logging is fundamental to debugging applications, especially those that have been deployed.
You can add a debugging JDBC driver to your project (for debugging - don't actually deploy it). One example (I have not used it) is log4jdbc. You then need to do some simple analysis on this file to see which executes don't have a corresponding close. Counting the open and closes should highlight if there is a potential problem
Monitoring the database. Monitor your running application using the tools such as the SQL Developer 'Monitor SQL' function or Quest's TOAD. Monitoring is described in this article. During monitoring, you query the open cursors (eg from table v$sesstat) and review their SQL. If the number of cursors is increasing, and (most importantly) becoming dominated by one identical SQL statement, you know you have a leak with that SQL. Search your code and review.

1 comment:

  1. To find out who opened many cursors on instance use following query:

    SELECT SID, count(*) ses_cursors, sum(count(*)) OVER() total_cursors
    FROM v$open_cursor
    GROUP BY SID
    ORDER BY 2 DESC;

    ReplyDelete