Revision History, Zope Enterprise Objects, version 2

ZEO 2.0.1 (2002/12/10)

The changes since the ZEO 2.0 release are:

• The logging code for client connection was improved in several ways. In particular, client connection and disconnection messages include the server's address.
• ClientCache.py: make some functions into methods; include the storage name in all log messages; append the storage name to the trace filename.
• zrpc/trigger.py: in handle_read() (both versions!), catch socket errors from the recv() call and return early in that case.

ZEO 2.0 (2002/10/28)

There were no changes made after beta 3.

ZEO 2.0 beta 3 (2002/10/21)

The changes since the ZEO 2.0b2 release are:

• Changed signal handling in start.py script. The start script was changed to respond to signals in the same manner as Zope. See Doc/ZEO/SIGNALS.txt (new).
• Changed pidfile policy in start.py script. The start script only writes a single pid to the pidfile; when using zdaemon, this is the zdaemon process's pid (which passes signals on to the actual storage server). Also, the pidfile is deleted upon completion of the server's main loop. (Only this latter change was also applied to ZEO1/start.py.)
• The start script now chdir's into the var directory, to give a predictable place for core dumps.
• The stats.py and simul.py script now skip suspect broken records; this makes for a more robust trace analysis.
• Some files had their RCS Id removed from docstrings; this was used only sporadically.

ZEO 2.0 beta 2 (2002/10/04)

The changes since the ZEO 2.0b1 release are:

• Added doc strings

  A start was made with adding doc strings to the major modules. ClientStorage.py was completed, a start was made with StorageServer.py. The rest will follow before the final release.

• Fixed race conditions

  Potential race conditions in many places were fixed by adding appropriate locking, including zrpc/smac.py and TransactionBuffer.py. Picklers are no longer shared and reused, as sharing a pickler is not re-entrant.

• Fixed trigger on Windows 98

  The code in zrpc/trigger.py to create a trigger instance for Windows could cause mysterious hangs on Windows 98 when a port was reused. Fixed by changing the port allocation algorithm.

• pack() waits by default

  The default value for the wait flag argument to pack() is changed to 1, making pack() wait by default (just like for the ClientStorage constructor.)

• Read-only fallback option for ClientStorage

  The ClientStorage class now has a new optional argument, read_only_fallback, defaulting to false. If set to true, the client will accept either read-only servers or read-write servers. If multiple server addresses are given, an available read-write server will be preferred over available read-only servers. When connected to a read-only server, a background thread continues to attempt to open a read-write connection, and when successful, will disconnect an existing read-only connection and reconnect to the read-write server. If this option is false, the original behavior is retained: when the read_only option is false, only read-write servers are acceptable; when the read_only option is true, all servers (read-only or read-write) are acceptable.

• Connect code refactored

  The client-side ZRPC code that connects to the server was rewritten almost from scratch (to allow adding the fallback option described above). This code should now do the right thing on Windows systems, at least on Win98 and Win2k (previously, the connect_ex() status wasn't always interpreted correctly).

• Overlapping calls allowed

  The limitation that ZRPC calls are serialized in the client is lifted. A multi-threaded client may now make overlapping calls to ClientStorage methods. Three notoriously slow calls (undoInfo(), undoLog(), pack()) are now executed in a separate thread in the server. The limitation to a single outstanding transaction remains though.

• zrpc made robust against pickling errors

  It was possible for the arguments to a zrpc call to include objects that were unpicklable on the server. (Example: The code for an object is not available on the server.) There were a variety of failure conditions that could occur, some of which would leave the client hanging forever. Fixed the error handling code to always close the connection when a message can't be unpickled. This approach allows the client and server to recover.

• Locking bug in new_oid() fixed

  The client-side code in new_oid() would not release its lock if its call to the server's new_oids() raised an exception. This would cause a hard hang upon a subsequent new_oid() call.

• Fix handling of mid-transaction disconnect

  When a connection was disconnected while a transaction was in progress, the server-side code attempting to abort the transaction could run into various rare traps. Amongst others, this frequently caused the checkCommitLock2On{Abort,Commit} tests to hang on Win98.

• Logging cleanup

  Various log messages were added, removed, or reformatted, or had their priority adjusted. The subsystem string in log messages is now more consistent and never contains spaces. When the subsystem string contains a process id, it is the correct process id (there were cases where the parent's process id continued to be used after a fork() call).

• Code cleanup

  Many parts of the code have been cleaned up. The changes include whitespace normalization, spelling fixes, correcting out-of-date comments and docstrings, removing unused imports, attributes and variables, renaming attributes and methods for consistency, and so on. Some refactoring of the test suite was also done.

• Fixed possible deadlock in storage server

  The locking used in the low-level zrpc mechanism was changed to prevent asynchronous callback messages from using the call lock. This prevents a possible deadlock when cache verification callbacks are processed quickly.

• Fixed performance problem differently

  Some low-level socket code was rewritten again to batch short messages and chunk long ones even more efficiently, as suggested by Toby Dickenson.

• Cache performance simulation

  A simulation program (ZEO/simul.py) was added that performs a simulation of the ZEO client cache, given a cache trace file produced by the client cache trace option (see below) and a cache size. The cache simulation will show the effectiveness of a particular cache size given the traffic measured at a site. More information is in the ZEO/README.txt file. (The simulation program also has options to simulate various alternative cache designs. We are looking for volunteers to send us a cache trace file for their site, so we can evaluate the effectiveness of various cache designs given their site's load. Send email to [email protected] if you are interested in participating.)

• Client cache bugs fixed

  Fixed several bugs in the client cache code that caused the cache to be less effective than it should. In particular:

  = the code to select the most recent current persistent cache file was broken due to a wrong seek offset;

  = the code to write a byte to a record to invalidate it was broken due to a wrong seek call;

  = on a cache file flip, the index data for the cache file that's being overwritten wasn't cleared out.

• Fix for abortVersion()

  Different versions of ZODB do different things with the serialnos for abortVersion(). To be safe and avoid giving non-version data with an invalid serialno after an abort version, both the version and non-version data are invalidated.

  This hurts cache effectiveness when a version is aborted, but we expect this will have little practical impact.

• Add sync() method back to ClientStorage

  The sync() method may be called by the ZODB Connection class on the storage. It can also be called by applications that do not have a separate asyncore mainloop when the application wants to handle pending object invalidations.

• undoLog() and undoInfo() can run in background

  These methods can take a long time to run if the client has specified a filter or specification. ZEO 2 will run these operations in a separate thread, allowing other client requests to be handled as long the underlying storage allows them.

• Client cache code cleanup

  The doc string was updated to provide more and more accurate information; other minor coding style improvements include logging a message at the INFO level whenever a broken cache record is detected. Cache flips and non-trivial read_index() statistics are also logged.

• Client cache trace option

  Setting the environment variable ZEO_CACHE_TRACE to point to a (writable) file writes a detailed trace of cache requests to that file. The stats.py program can be used to analyze this trace file. More information is in the ZEO/README.txt file.

• Fixed handling of closed connections in the server

  When a client disconnected, the storage server wasn't properly calling the strategy's tpc_abort, so it wasn't properly restarting another transaction that might possibly be waiting.

• Other code cleanup and small bugfixes

  Renamed some methods. Removed some unused variables, functions, methods and files. Added some explanatory comments. Added some asserts. Whitespace cleanup. ClientStorage.close() is now idempotent. Fixed a bug in undo(). ImmediateCommitStrategy.store() only catches exceptions that inherit from Exception. Added missing import of DecodingError in zrpc/marshal.py.

• Optimized cache opening

  ClientStorage opens the cache later to avoid "opening" it twice when a connection is available right away.

• Added a modest unit test suite for the client cache

  The basic operations of the client cache are now tested.

• Wait for connection by default

  The default for the ClientStorage class has changed to wait until a connection is established by default. To return immediately without waiting for a connection, use ClientStorage(..., wait=0). (This only makes sense when a persistent cache is also specified, by using the client keyword argument to the ClientStorage constructor, or by setting the ZEO_CLIENT environment variable.

• Arguments debug and wait_for_server_on_startup are back

  The ZEO1 arguments debug and wait_for_server_on_startup were added to the ClientStorage constructor signature, for backwards compatibility (these are used in many a custom_zodb.py file). Setting debug has no effect except issuing a log message. The wait_for_server_on_startup is an alias for the wait argument; using wait_for_server_on_startup issues a log message.

• ClientStorage and StorageServer subclassing hooks

  The classes instantiated within the client and server are now defined as class variables. This approach allows subclasses to substitute different classes.

ZEO 2.0 beta 1 (2002/08/23)

The changes since the ZEO 2.0a1 release are:

• Added version identification.

  The variable ZEO.version is a string identifying the version in the style of sys.version. The same string is also available in ZEO/version.txt for people who want to check without loading the code.

• Fixed performance problem.

  The low-level socket code was often calling send() with very small strings instead of joining them together before calling send. The small string variant ran afoul of the Nagle algorithm and delayed TCP acks.

• Fix bugs in server handling of queued transactions

  Several bugs in 2.0a1 could cause the storage to lock up when it attempted to restart a transaction that was queued waiting for an earlier transaction to finish. The locking code in the two-phase commit methods was revised to properly use the condition variable protocol. Thanks to Tim Peters for pointing this out.

• Avoid locking up server on unexpected client exit

  If a client disconnected unexpectedly in the middle of a two- phase commit, the server would never release the commit lock.

• Very long log messages avoided.

  Some of the logging messages at BLATHER and lower were producing very long log messages because they included all the argument of method calls, even when the argument was a list with thousands of elements. These log messages are now truncated at a reasonable length.

• Fix log rotation

  The SIGHUP behavior was fixed. If the ZEO server receives a SIGHUP, it will re-initialize its logging code. Thus, it is possible to rotate the log files and then send a HUP.

• Fixed problem handling signals.

  The asyncore and zrpc error handlers would trap SystemExit if it was raised by a signal handler. They now ignore it.

• pack() can be called on a read-only connection

  This ends up being very convenient for packing a read-only server, even though it isn't obvious that pack() is a read- only operation.

• pack(wait=1) now waits

  A bug caused pack(wait=1) to return immediately, even though the pack was still running.

• Fix bug in history().

  The history() implementation prevent the client from returning more than one item.

• Enhanced logging for blocked transactions.

  If a transaction had to be blocked because another transaction was executing, the server would log an info message. Now it also logs a message when the transaction is resumed, so that it doesn't look like the transaction was blocked forever.

• Fixed bug in logging call for store().

  The logging call could raise NameError.

• Removed debug argument to ClientStorage constructor.

  It had no effect.

• Removed ZEO.trigger

  There were two copies of the trigger module in the last release. The one in ZEO.trigger was removed, because it is only needed in ZEO.zrpc.trigger.

• Added minimal tests of start.py.

  There are only run on Unix.

ZEO 2.0 alpha 1 (2002/05/31)

Brief overview of the differences between ZEO 1.0 and 2.0.

• New protocol.

  ZEO 2 uses a different wire protocol and a different API to make RPC calls. The new protocol was designed to be flexible and simple. It includes an initial handshake to set the version number, which should allow future changes to the protocol while reducing the difficulty of upgrades.

• Better handling of concurrent commits.

  The ZEO server serializes concurrent commits to guarantee consistency; the mechanism is often called the distributed commit lock. ZEO 2 improves the efficiency of concurrent commits by allowing data to be transferred to the server before entering the commit lock.

• The ZEO client and server can be configured to operate in read-only mode.
• A ZEO client can be configured with multiple server addresses. It uses the first server it can connect to.
• The wait_for_server_on_startup keyword argument to ClientStorage has been renamed wait.