You are not logged in Log in Join
You are here: Home » Download Zope Products » Zope Object Database » NEWS » View Document

Log in
Name

Password

 

NEWS

What's new in ZODB3 3.2 alpha 1

Release date: 17-Jan-2003

Most of the changes in this release are performance and stability improvements to ZEO. A major packaging change is that there won't be a separate ZEO release. The new ZConfig is a noteworthy addtion (see below).

ZODB

An experimental new transaction API was added. The Connection class has a new method, setLocalTransaction(). ZODB applications can call this method to bind transactions to connections rather than threads. This is especially useful for GUI applications, which often have only one thread but multiple independent activities within that thread (generally one per window). Thanks to Christian Reis for championing this feature.

Applications that take advantage of this feature should not use the get_transaction() function. Until now, ZODB itself sometimes assumed get_transaction() was the only way to get the transaction. Minor corrections have been added. The ZODB test suite, on the other hand, can continue to use get_transaction(), since it is free to assume that transactions are bound to threads.

ZEO

There is a new recommended script for starting a storage server. We recommend using ZEO/runzeo.py instead of ZEO/start.py. The start.py script is still available in this release, but it will no longer be maintained and will eventually be removed.

There is a new zdaemon implementation. This version is a separate script that runs an arbitrary daemon. To run the ZEO server as a daemon, you would run "zdaemon.py runzeo.py". There is also a simple shell, zdctl.py, that can be used to manage a daemon. Try "zdctl.py -p runzeo.py".

There is a new version of the ZEO protocol in this release and a first stab at protocol negotiation. (It's a first stab because the protocol checking supporting in ZODB 3.1 was too primitive to support anything better.) A ZODB 3.2 ZEO client can talk to an old server, but a ZODB 3.2 server can't talk to an old client. It's safe to upgrade all the clients first and upgrade the server last. The ZEO client cache format changed, so you'll need to delete persistent caches before restarting clients.

The ZEO cache verification protocol was revised to require many fewer messages in cases where a client or server restarts quickly.

The performance of full cache verification has improved dramatically. XXX Get measurements from Jim -- somewhere in 2x-5x recall. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely.

A bug was fixed in the ZEO protocol that allowed clients to read stale cache data while cache verification was being performed. The fixed version prevents the client from using the storage until after verification completes.

The ZEO server has an experimental monitoring interface that reports usage statistics for the storage server including number of connected clients and number of transactions active and committed. It can be enabled by passing the -m flag to runsvr.py.

The ZEO ClientStorage no longer supports the environment variables CLIENT_HOME, INSTANCE_HOME, or ZEO_CLIENT.

The ZEO1 package is still included with this release, but there is no longer an option to install it.

BTrees

The BTrees package now has a check module that inspects a BTree to check internal invariants. Bugs in older versions of the code code leave a BTree in an inconsistent state. Calling BTrees.check.check() on a BTree object should verify its consistency. (See the NEWS section for 3.1 beta 1 below to for the old BTrees bugs.)

Fixed a rare conflict resolution problem in the BTrees that could cause an segfault when the conflict resolution resulted in any empty bucket.

Installation

The distutils setup now installs several Python scripts. The runzeo.py and zdaemon.py scripts mentioned above and several fsXXX.py scripts from the Tools directory.

The test.py script does not run all the ZEO tests by default, because the ZEO tests take a long time to run. Use --all to run all the tests. Otherwise a subset of the tests, mostly using MappingStorage, are run.

Storages

There are two new storages based on Sleepycat's BerkeleyDB in the BDBStorage package. Barry will have to write more here, because I don't know how different they are from the old bsddb3Storage storages. See Doc/BDBStorage.txt for more information.

It now takes less time to open an existing FileStorage. The FileStorage uses a BTree-based index that is faster to pickle and unpickle. It also saves the index periodically so that subsequent opens will go fast even if the storage was not closed cleanly.

Misc

The new ZConfig package, which will be used by Zope and ZODB, is included. ZConfig provides a configuration syntax, similar to Apache's syntax. The package can be used to configure the ZEO server and ZODB databases. See the module ZODB.config for functions to open the database from configuration. See ZConfig/doc for more info.

The zLOG package now uses the logging package by Vinay Sajip, which will be included in Python 2.3.

The Sync extension was removed from ExtensionClass, because it was not used by ZODB.

What's new in ZODB3 3.1.1?

Release date: 10-Dev-2002

It was possible for earlier versions of ZODB to deadlock when using multiple storages. If multiple transactions committed concurrently and both transactions involved two or more shared storages, deadlock was possible. This problem has been fixed by introducing a sortKey() method to the transaction and storage APIs that is used to define an ordering on transaction participants. This solution will prevent deadlocks provided that all transaction participants that use locks define a valid sortKey() method. A warning is raised if a participant does not define sortKey(). For backwards compatibility, BaseStorage provides a sortKey() that uses __name__.

Added code to ThreadedAsync/LoopCallback.py to work around a bug in asyncore.py: a handled signal can cause unwanted reads to happen.

A bug in FileStorage related to object uncreation was fixed. If an a transaction that created an object was undone, FileStorage could write a bogus data record header that could lead to strange errors if the object was loaded. An attempt to load an uncreated object now raises KeyError, as expected.

The restore() implementation in FileStorage wrote incorrect backpointers for a few corner cases involving versions and undo. It also failed if the backpointer pointed to a record that was before the pack time. These specific bugs have been fixed and new test cases were added to cover them.

A bug was fixed in conflict resolution that raised a NameError when a class involved in a conflict could not be loaded. The bug did not affect correctness, but prevent ZODB from caching the fact that the class was unloadable. A related bug prevented spurious AttributeErrors when a class could not be loaded. It was also fixed.

The script Tools/zeopack.py was fixed to work with ZEO 2. It was untested and had two silly bugs.

Some C extensions included standard header files before including Python.h, which is not allowed. They now include Python.h first, which eliminates compiler warnings in certain configurations.

The BerkeleyDB based storages have been merged from the trunk, providing a much more robust version of the storages. They are not backwards compatible with the old storages, but the decision was made to update them in this micro release because the old storages did not work for all practical purposes. For details, see Doc/BDBStorage.txt.

What's new in ZODB3 3.1 final?

Release date: 28-Oct-2002

If an error occurs during conflict resolution, the store will silently catch the error, log it, and continue as if the conflict was unresolvable. ZODB used to behave this way, and the change to catch only ConflictError was causing problems in deployed systems. There are a lot of legitimate errors that should be caught, but it's too close to the final release to make the substantial changes needed to correct this.

What's new in ZODB3 3.1 beta 3?

Release date: 21-Oct-2002

A small extension was made to the iterator protocol. The Record objects, which are returned by the per-transaction iterators, contain a new data_txn attribute. It is None, unless the data contained in the record is a logical copy of an earlier transaction's data. For example, when transactional undo modifies an object, it creates a logical copy of the earlier transaction's data. Note that this provide a stronger statement about consistency than whether the data in two records is the same; it's possible for two different updates to an object to coincidentally have the same data.

The restore() method was extended to take the data_txn attribute mentioned above as an argument. FileStorage uses the new argument to write a backpointer if possible.

A few bugs were fixed.

The setattr slot of the cPersistence C API was being initialized to NULL. The proper initialization was restored, preventing crashes in some applications with C extensions that used persistence.

The return value of TimeStamp's __cmp__ method was clipped to return only 1, 0, -1.

The restore() method was fixed to write a valid backpointer if the update being restored is in a version.

Several bugs and improvements were made to zdaemon, which can be used to run the ZEO server. The parent now forwards signals to the child as intended. Pidfile handling was improved and the trailing newline was omitted.

What's new in ZODB3 3.1 beta 2?

Release date: 4-Oct-2002

A few bugs have been fixed, some that were found with the help of Neal Norwitz's PyChecker.

The zeoup.py tool has been fixed to allow connecting to a read-only storage, when the --nowrite option is given.

Casey Duncan fixed a few bugs in the recent changes to undoLog().

The fstest.py script no longer checks that each object modified in a transaction has a serial number that matches the transaction id. This invariant is no longer maintained; several new features in the 3.1 release depend on it.

The ZopeUndo package was added. If ZODB3 is being used to run a ZEO server that will be used with Zope, it is usually best if the server and the Zope client don't share any software. The Zope undo framework, however, requires that a Prefix object be passed between client and server. To support this use, ZopeUndo was created to hold the Prefix object.

Many bugs were fixed in ZEO, and a couple of features added. See ZEO-NEWS.txt for details.

The ZODB guide included in the Doc directory has been updated. It is still incomplete, but most of the references to old ZODB packages have been removed. There is a new section that briefly explains how to use BTrees.

The zeoup.py tool connects using a read-only connection when --nowrite is specifified. This feature is useful for checking on read-only ZEO servers.

What's new in ZODB3 3.1 beta 1?

Release date: 12-Sep-2002

We've changed the name and version number of the project, but it's still the same old ZODB. There have been a lot of changes since the last release.

New ZODB cache

Toby Dickenson implemented a new Connection cache for ZODB. The cache is responsible for pointer swizzling (translating between oids and Python objects) and for keeping recently used objects in memory. The new cache is a big improvement over the old cache. It strictly honors its size limit, where size is specified in number of objects, and it evicts objects in least recently used (LRU) order.

Users should take care when setting the cache size, which has a default value of 400 objects. The old version of the cache often held many more objects than its specified size. An application may not perform as well with a small cache size, because the cache no longer exceeds the limit.

Storages

The index used by FileStorage was reimplemented using a custom BTrees object. The index maps oids to file offsets, and is kept in memory at all times. The new index uses about 1/4 the memory of the old, dictionary-based index. See the module ZODB.fsIndex for details.

A security flaw was corrected in transactionalUndo(). The transaction ids returned by undoLog() and used for transactionalUndo() contained a file offset. An attacker could construct a pickle with a bogus transaction record in its binary data, deduce the position of the pickle in the file from the undo log, then submit an undo with a bogus file position that caused the pickle to get written as a regular data record. The implementation was fixed so that file offsets are not included in the transaction ids.

Several storages now have an explicit read-only mode. For example, passing the keyword argument read_only=1 to FileStorage will make it read-only. If a write operation is performed on a read-only storage, a ReadOnlyError will be raised.

The storage API was extended with new methods that support the Zope Replication Service (ZRS), a proprietary Zope Corp product. We expect these methods to be generally useful. The methods are:

  • restore(oid, serialno, data, version, transaction)

    Perform a store without doing consistency checks. A client can use this method to provide a new current revision of an object. The serialno argument is the new serialno to use for the object, not the serialno of the previous revision.

  • lastTransaction()

    Returns the transaction id of the last committed transaction.

  • lastSerial(oid)

    Return the current serialno for oid or None.

  • iterator(start=None, stop=None)

    The iterator method isn't new, but the optional start and stop arguments are. These arguments can be used to specify the range of the iterator -- an inclusive range [start, stop].

FileStorage is now more cautious about creating a new file when it believes a file does not exist. This change is a workaround for bug in Python versions upto and including 2.1.3. If the interpreter was builtin without large file support but the platform had it, os.path.exists() would return false for large files. The fix is to try to open the file first, and decide whether to create a new file based on errno.

The undoLog() and undoInfo() methods of FileStorage can run concurrently with other methods. The internal storage lock is released periodically to give other threads a chance to run. This should increase responsiveness of ZEO clients when used with ZEO 2.

New serial numbers are assigned consistently for abortVersion() and commitVersion(). When a version is committed, the non-version data gets a new serial number. When a version is aborted, the serial number for non-version data does not change. This means that the abortVersion() transaction record has the unique property that its transaction id is not the serial number of the data records.

Berkeley Storages

Berkeley storage constructors now take an optional config argument, which is an instance whose attributes can be used to configure such BerkeleyDB policies as an automatic checkpointing interval, lock table sizing, and the log directory. See bsddb3Storage/BerkeleyBase.py for details.

A getSize() method has been added to all Berkeley storages.

Berkeley storages open their environments with the DB_THREAD flag.

Some performance optimizations have been implemented in Full storage, including the addition of a helper C extension when used with Python 2.2. More performance improvements will be added for the ZODB 3.1 final release.

A new experimental Autopack storage was added which keeps only a certain amount of old revision information. The concepts in this storage will be folded into Full and Autopack will likely go away in ZODB 3.1 final. ZODB 3.1 final will also have much improved Minimal and Full storages, which eliminate Berkeley lock exhaustion problems, reduce memory use, and improve performance.

It is recommended that you use BerkeleyDB 4.0.14 and PyBSDDB 3.4.0 with the Berkeley storages. See bsddb3Storage/README.txt for details.

BTrees

BTrees no longer ignore exceptions raised when two keys are compared.

Tim Peters fixed several endcase bugs in the BTrees code. Most importantly, after a mix of inserts and deletes in a BTree or TreeSet, it was possible (but unlikely) for the internal state of the object to become inconsistent. Symptoms then varied; most often this manifested as a mysterious failure to find a key that you knew was present, or that tree.keys() would yield an object that disgreed with the tree about how many keys there were.

If you suspect such a problem, BTrees and TreeSets now support a ._check() method, which does a thorough job of examining the internal tree pointers for consistency. It raises AssertionError if it finds any problems, else returns None. If ._check() raises an exception, the object is damaged, and rebuilding the object is the best solution. All known ways for a BTree or TreeSet object to become internally inconsistent have been repaired.

Other fixes include:

  • Many fixes for range search endcases, including the "range search bug:" If the smallest key S in a bucket in a BTree was deleted, doing a range search on the BTree with S on the high end could claim that the range was empty even when it wasn't.
  • Zope Collector #419: repaired off-by-1 errors and IndexErrors when slicing BTree-based data structures. For example, an_IIBTree.items()[0:0] had length 1 (should be empty) if the tree wasn't empty.
  • The BTree module functions weightedIntersection() and weightedUnion() now treat negative weights as documented. It's hard to explain what their effects were before this fix, as the sign bits were getting confused with an internal distinction between whether the result should be a set or a mapping.

ZEO

For news about ZEO2, see the file ZEO-NEWS.txt.

This version of ZODB ships with two different versions of ZEO. It includes ZEO 2.0 beta 1, the recommended new version. (ZEO 2 will reach final release before ZODB3.) The ZEO 2.0 protocol is not compatible with ZEO 1.0, so we have also included ZEO 1.0 to support people already using ZEO 1.0.

Other features

When a ConflictError is raised, the exception object now has a sensible structure, thanks to a patch from Greg Ward. The exception now uses the following standard attributes: oid, class_name, message, serials. See the ZODB.POSException.ConflictError doc string for details.

It is now easier to customize the registration of persistent objects with a transaction. The low-level persistence mechanism in cPersistence.c registers with the object's jar instead of with the current transaction. The jar (Connection) then registers with the transaction. This redirection would allow specialized Connections to change the default policy on how the transaction manager is selected without hacking the Transaction module.

Empty transactions can be committed without interacting with the storage. It is possible for registration to occur unintentionally and for a persistent object to compensate by making itself as unchanged. When this happens, it's possible to commit a transaction with no modified objects. The change allows such transactions to finish even on a read-only storage.

Two new tools were added to the Tools directory. The analyze.py script, based on a tool by Matt Kromer, prints a summary of space usage in a FileStorage Data.fs. The checkbtrees.py script scans a FileStorage Data.fs. When it finds a BTrees object, it loads the object and calls the _check method. It prints warning messages for any corrupt BTrees objects found.

Documentation

The user's guide included with this release is still woefully out of date.

Other bugs fixed

If an exception occurs inside an _p_deactivate() method, a traceback is printed on stderr. Previous versions of ZODB silently cleared the exception.

ExtensionClass and ZODB now work correctly with a Python debug build.

All C code has been fixed to use a consistent set of functions from the Python memory API. This allows ZODB to be used in conjunction with pymalloc, the default allocator in Python 2.3.

zdaemon, which can be used to run a ZEO server, more clearly reports the exit status of its child processes.

The ZEO server will reinitialize zLOG when it receives a SIGHUP. This allows log file rotation without restarting the server.

What's new in StandaloneZODB 1.0 final?

Release date: 08-Feb-2002

All copyright notices have been updated to reflect the fact that the ZPL 2.0 covers this release.

Added a cleanroom PersistentList.py implementation, which multiply inherits from UserDict and Persistent.

Some improvements in setup.py and test.py for sites that don't have the Berkeley libraries installed.

A new program, zeoup.py was added which simply verifies that a ZEO server is reachable. Also, a new program zeopack.py was added which connects to a ZEO server and packs it.

What's new in StandaloneZODB 1.0 c1?

Release Date: 25-Jan-2002

This was the first public release of the StandaloneZODB from Zope Corporation. Everything's new! :)