Upgrade Your Existing ZCatalog to ZCatalog in Zope 2.3.1b2 or Better

Created by . Last modified on 2003/08/01.

Upgrading A ZCatalog From Zope Versions Prior to 2.3.1b2 to Zope 2.3.1b2+

Overview

Zope version prior to 2.3.1b2 have a ZCatalog implementation which uses a previous generation of Zope BTree machinery. With Zope 2.3.1b2+, the ZCatalog uses a newer BTree implementation which is capable of improving performance under heavy write load. The ZCatalog uses BTrees to efficiently service queries and index requests.

You should read this document carefully before upgrading to a Zope version later than 2.3.1b2 from an earlier version if you make use of ZCatalog in your Zope instance.

Q & A

Q

What do I need to do to take advantage of the new catalog data structures?

A

Once you upgrade an existing Zope to Zope equal to or later than 2.3.1b2+, your ZCatalog(s) will continue to work indefinitely using the "old" BTree data structures. Objects that you add to the catalog will continue to be added to the "old" data structures. You will likely notice nothing much has changed. You need to perform manual steps to convert your data structures.

Before doing any conversion, it's recommended that you MAKE A BACKUP of your Data.fs (or other ZODB datastore). We expect there to be little problem during the conversion process, but it's always safe to have a backup.

Before converting your catalog data structures using the steps outlined below, make sure to disable write access to the catalog instance you're converting. This is most easily accomplished by taking your site down and doing the conversion "offline", but if you cannot do this for uptime reasons, make sure to disable portions of your site which allow your users to make catalog changes for the duration of the upgrade process. This will help prevent the possibility of losing user data written to a ZCatalog during the upgrade.

To convert your ZCatalog BTree data structures, visit the manage_convertBTrees method of your catalog instance(s) as a user with management privileges through a browser.. For example, if your catalog is named "catalog" and it's in a folder named "afolder", you would visit the conversion method by using the URL:

       http://yourserver.com/afolder/catalog/manage_convertBTrees

You will see a message returned to your browser like:

       Finished conversion in 0.027897003127 seconds (0.01 cpu)

Once you see this message, your existing catalog has been converted to use the new BTree data structures. Perform this step for each of your ZCatalog instances.

Additionally, once you "update" or "clear" a ZCatalog instance (via the "Advanced -> Update Catalog" or "Advanced -> Clear Catalog" management screen options respectively), the ZCatalog and the index objects which relate to it will automatically be converted to make use of the new BTree data structures. However, the Vocabulary object used by the catalog instance will not be converted. Thus, it's best to use the manage_convertBTrees method to convert your catalog instances instead of relying on a recatalog via "update catalog", because the manage_convertBTrees mechanism converts all catalog-related data structures, including the Vocabulary objects. Using update catalog also takes much longer than manage_convertBTrees.

New ZCatalog, Vocabulary, and Index instances created after the installation of Zope 2.3.1b2 or better will be created using the new BTree data structures and need not be converted.

Q

I've never used the catalog before, and I have no data to convert from an old catalog, but I'm using Zope 2.3.1b2 or better. Do I need to pay attention to any of this?

A

No. Only folks who have data in "old" catalogs that they wish to upgrade to the new catalog data structures need to pay any attention to this whatsosever.

Q

How long will the conversion take?

A

This depends on how much data you have in your Catalog, as well as how many indexes you have, and what types of indexes they are. On a Pentium III 800 with 256MB RAM, with a catalog consisting of 40,000 cataloged objects against 9 indexes (5 text, 4 field), the conversion via manage_convertBTrees takes about a second and a half. Not very long, in other words.

Q

With an "old" ZCatalog (shipped with 2.3.0 or less), it seems that some of my documents aren't indexed properly or that I can't unindex documents properly.

A

Consider rebuilding the catalog via the "Advanced ->Update Catalog" button. There were some bugs in Catalogs prior to the one which ships with 2.3.1b2. Rebuilding the catalog will allow you to index and unindex documents properly.

Q

What sort of "real world" benefits will I see in catalog operations when I start using a catalog with the new BTree data structures as opposed to a catalog with the "old" BTree data structures?

A

Sites with high write traffic will likely see fewer ZODB ConflictErrors due to the new BTree design. This will cause fewer transaction retries, speeding read and write performance overall. Incremental indexing operations (such as those performed by "CatalogAware" objects) will cause less egregious "database bloat" in undoing storages like FileStorage. Queries which use AND, OR, OR NOT, or NEAR operations will likely be faster. Bulk reindexing operations may use less RAM.

Q

I'm using ZEO, and I make heavy use of the ZCatalog. I'd like to convert all of my ZEO clients individually, while keep my production site "up and running" during the upgrade. Do I need to do anything special?

A

If you want to perform an "in-place" upgrade of all your ZEO clients, it's highly recommended that you disallow any writes to all ZCatalog instances in your database for the duration of the upgrade process (possibly by disallowing user access to code which does catalog writes) as outlined in the first "A" above. Once you have finished upgrading all of your clients' Zope files, you should restart all of the clients and initiate an manage_convertBTrees against each of your ZCatalog instances from a single ZEO client only. Subsequently, all of the clients will use the new data structures. You may then reenable writes to your ZCatalog(s).

The downtime caused by the inability to do writes to a heavily used Catalog for any period of time may be painful for some sites, but we expect that the benefit of performing the upgrade will outweigh the detriment of losing functionality for a period of time, especially for users of ZEO.

Q

I've updated my catalog and nothing seems different. Why?

A

This is actually a good sign! The changes we've made to the ZCatalog are not "user-visible". There have been no significant changes to the API or the management interface. Only the underlying data structures have changed.

Q

What should I do if I have a problem after I convert my ZCatalog?

A

First of all, try to undo the transaction for manage_convertBTrees by visiting the the Catalog's Undo tab. If that doesn't seem to help, restore your "old" ZODB datastore from a backup. Then post a message to the Zope mail list ([email protected]) with the following information:

• a "traceback" if there is an error message during operation with the new data structures (usually available via the HTML source of the error page)
• the version of Zope you're upgrading from, and the version you're upgrading to.
• the symptoms of the problem.