Roadmap for Zope Database Adapters

Overview

The past few weeks, having brought the Zope 2 release to a beta level, have brought to the attention of everyone that the current design and architecture of various DAs is incompatible with use in a fully threaded environment. This document hopefully will clarify some of what the various level of compatibilities is, as well as comments on individual DAs, and their future.

DA Thread Safety Levels

In order to quantify in a uniform manner the compatibility of various DAs with Zope2, we have developed the following "levels" for database adapters to implement.

Level 1

All database adapters which implement this level of compatibility must run in a single thread as they have no level of thread safety. This currently applies to nearly all DAs. This means in ZServer that Zope runs in a single thread.

Level 2

Level 2 database adapters must be safe to use in a multi-threaded Zope instance. This can take multiple behaviour paterns:

• Internal locking/mutexes for single DA access
• Global locking through the transaction manager if appropriate

While a sub-optimal solution, it should be usable on any application that has minimal, or infrequent, database access. This means that all access paterns will be serialized.

In addition, all DAs at this level must release the Python global interpreter lock.

Level 3

A database adapter implementing Level 3 compliance must be fully safe running in a multi-threaded Zope instance. In addition, it must support:

• Multiple connections in the same thread
• Multiple connections in different threads running simultaneously

Note that there are instances (see the CLI information below) under which the DA might implement Level 3, but the underlying client libraries are somewhat different.

Transaction Manager Issues

Zope has the concept of a Transaction Manager which deals with the integration of Zope transactions and transactions in an external data source.

New Symantics

With the introduction of Zope 2, there are new Transaction Manager symantics, described as below. If you wish to bring a DA into line with the new specification, these methods are what you must implement.

• self._begin - This is called at the beginning of every transaction. Depending on your database, and its implementation of "implicit transactions", you may or may not have to do anything here besides a pass.
• self._finish - This is called when a transaction is finished, or committed.
• self._abort - Obviously this is called when the transaction involved has been aborted. It should always roll back the transaction in the database.

Integration Requirements

Not all DAs talk to data stores which have transactional symantics. These Database Adapters should not implement integration with the Transaction Manager unless it makes complete sense. However, all DAs which deal with data stores which do have a transactional symantic should integrate completely with the Transaction Manager.

Relational Database Adapters Available

OracleDA (v2.1.0)

The current OracleDA is completely L3 compliant and works with both multi-threading and full concurrency with the new transaction machienery.

Near term progress. None.

Long term progress. Plans have been made to split the ZOracleDA into two seperate DAs:

• ZOracle7DA - Compatible with Oracle 7 OCI interfaces.
• ZOracle8DA - Comaptible with Oracle 8 OCI interfaces. This will also introduce some level of support for LOBs.

Support availability. Commercial support is available. Note that this is support for the DA, Digital Creations will not be selling general support for dcoracle.

ZSybaseDA (v2.0.0)

The current internal Digital Creations implementation of a Sybase DA (using the ctsybase Python level interface) is operating at Level 1, with some work having been made at making it Level 2. Currently it has a Sybase-level memory leak when at Level 2.

Near term progress. Before the final release of Zope 2.0, the ZSybaseDA will be brought up to Level 2 DA. This is being forced by customer requirements.

Long term progress. While we desire to bring the ZSybaseDA up to full Level 3 compiance exists, it will not happen until customer requirements have driven it.

Support availability. Commercial support is available.

MySQL Database Adapter (v1.1.3)

The current MySQL adapter is at least Level 1 compliant, however, it does not currently implement the new transaction manager symantics.

Near term progress. We would very much like to transfer ownership of the MySQL DA to someone who has a desire to keep it maintained.

Long term progress. None.

Support availability. None.

SOLID Database Adapter (v2.0.0)

Due to the recent change in SOLID Technologies marketing direction and a clarification from their staff that they are no longer interested in the "stand alone" database market, we have deprecated support for this module.

Near term progress. None.

Long term progress. We think if desired, this could be implemented in the upcoming replacement rather than in a seperate module.

Support availability. None.

ODBC Database Adatper (v3.0.3)

The current ODBC DA is not compliant with any level of Zope 2 Database Adapter thread safety. In addition, it is not integrated with the new transaction amanger symantics. This is a serious problem.

Near term progress. By the final release of Zope 2.0, the ODBC DA will be modified to be at a minimum Level 2 compliant, based on the available "thunk" mechanism as well as transaction manager integration.

Long term progress. See information below on the future of a X/Open CLI interface DA.

Support availability. Commercial support is available.

SQL/CLI Database Adapter

In the process of reviewing the desirability of an IBM DB2 database adapter, it has come to our attention that both ODBC and DB2 are derived from the same X/Open and ISO specification called the "Call Level Interface", and are both compatible at that level. Because of this, we will be transitioning to a new design for the ODBC DA, as well as integrating support for IBM DB2 into the same DA.

Background on SQL/CLI

The X/Open Company and the SQL Access Group jointly defined a specification for a callable SQL interface reffered to as the X/Open Call Level Interface. The goal of this interface is to increase the portability of applications by enabling them to become independent of any one database vendor's programming interface. Most of the X/Open Call Level Interface specification has been accepted as part of the ISO Call Level Interface International Specification (ISO/IEF 9075-3:1995 SQL/CLI).

Microsoft developed a callable SQL interface called Open Database Connectivity (ODBC) for Microsoft operating systems based on a preliminary draft of the X/Open CLI. Version 3 of ODBC conforms to almost all of the ISO SQL/CLI. ODBC 3.0 does contain considerable functionality that is not part of the International Standard; much of this is being added to the next draft of that standard.

A Merged DA

This new DA will be called the CLIDAzope :-) Actually, we've not come up with a name, but it will be based purely on the CLI specification intially, with small subclasses to deal with ODBC and DB2 individually. This will allow us to leverage two seperate environments at the same time. Eventually we will integrate with DB2s excellent (Oracle are you listening? :-) LOB support, and work to integrate with the support in Microsoft SQLServer 7 wherever possible.

Relation to UNIX Platforms

Many people have asked about ODBC support on UNIX, and this is the solution. What this provides is the ability to link directly with an SQL/CLI compatible library without having to deal with the issues of an ODBC Driver Manager. In addition, we hope that this will also reduce the performance impact of the driver manager.

Non-Relational Database Adapters

In addition to integration with relational databases, there are also various DAs which do not talk to "relational" data stores. These include:

• LDAP
• Ultraseek
• IMAP

These databases are covered by the same thread-safety levels, but may or may not be integrated with the transaction manager depending on the paradigm that they are constructed inside.

Deprecated Database Adapter Issues

As you will note, we have deprecated several database adapters which we have handled the distribution of, and therefore they are now orphaned. In order to help people who currently depend on them, we are offering them to the community to take up the torch of support. Toward that end we are willing to:

• Transfer code ownership to a new "champion," so long as the license remains Open Source.
• Provide CVS repository space for the management of the project.

Please let us know if you have the time and desire to take up the banner of these DAs.

The Kludge of the Century

While we understand that it is desirable to have everything run beautifiully in a fully threaded environment, it is also unrealistic to assume that this will all be done within the next 48 hours, and so we have provided a klduge to get around this issue. This does not escape the requirement for TM integration, it simply fixes the problem with threading in DAs that don't support threading yet, but do use the Transaction Manager.

In order to bring your DA into line, you will need to subclass your DB class as follows:

      import Shared.DB.ZRDB.THUNK

      class DB(Shared.DB.ZRDB.THUNK):

      ...

The result of this "thunk" is to create a DA that is single-threaded through the transaction manager. More so, all DAs which are "thunked" are single threaded with each other. This means that a MySQL DA will wait on a Sybase DA, and vice versa. While this is sub-optimal, it is at least functional, which is more than many DAs currently can claim under Zope 2.

Support Issues

Where support has been discussed, please understand that we will try and provide support on an as available basis to all comers, however we can not guarantee support response times and effectiveness to any non-paying customers. This should be taken into account when selecting a DA.

When commercial support is offered it is offered in multiple fashions:

• Time & materials
• Per-incident basis
• Yearly contract

Please contact Digital Creations for more information about this option.