Upgrade Your Existing ZCatalog to ZCatalog in Zope 2.3.1b2 or Better
Upgrading A ZCatalog From Zope Versions Prior to 2.3.1b2 to Zope 2.3.1b2+
Zope version prior to 2.3.1b2 have a ZCatalog implementation which
uses a previous generation of Zope BTree machinery. With
Zope 2.3.1b2+, the ZCatalog uses a newer BTree implementation
which is capable of improving performance under heavy write load.
The ZCatalog uses BTrees to efficiently service queries and index
You should read this document carefully before upgrading to a Zope
version later than 2.3.1b2 from an earlier version if you make use
of ZCatalog in your Zope instance.
Q & A
- What do I need to do to take advantage of the new catalog
- Once you upgrade an existing Zope to Zope equal to or later
than 2.3.1b2+, your ZCatalog(s) will continue to work indefinitely
using the "old" BTree data structures. Objects that you add to
the catalog will continue to be added to the "old" data
structures. You will likely notice nothing much has changed. You
need to perform manual steps to convert your data structures.
Before doing any conversion, it's recommended that you MAKE A
BACKUP of your Data.fs (or other ZODB datastore). We expect
there to be little problem during the conversion process, but
it's always safe to have a backup.
Before converting your catalog data structures using the
steps outlined below, make sure to disable write access to
the catalog instance you're converting. This is most easily
accomplished by taking your site down and doing the conversion
"offline", but if you cannot do this for uptime reasons,
make sure to disable portions of your site which allow
your users to make catalog changes for the duration of the
upgrade process. This will help prevent the possibility of
losing user data written to a ZCatalog during the upgrade.
To convert your ZCatalog BTree data structures, visit the
manage_convertBTrees method of your catalog instance(s) as a
user with management privileges through a browser.. For
example, if your catalog is named "catalog" and it's in a folder
named "afolder", you would visit the conversion method by
using the URL:
You will see a message returned to your browser like:
Finished conversion in 0.027897003127 seconds (0.01 cpu)
Once you see this message, your existing catalog has been
converted to use the new BTree data structures. Perform this
step for each of your ZCatalog instances.
Additionally, once you "update" or "clear" a ZCatalog instance
(via the "Advanced -> Update Catalog" or "Advanced -> Clear
Catalog" management screen options respectively), the ZCatalog
and the index objects which relate to it will automatically be
converted to make use of the new BTree data structures.
However, the Vocabulary object used by the catalog instance will
not be converted. Thus, it's best to use the
manage_convertBTrees method to convert your catalog instances
instead of relying on a recatalog via "update catalog", because
manage_convertBTrees mechanism converts all
catalog-related data structures, including the Vocabulary
update catalog also takes much longer than
New ZCatalog, Vocabulary, and Index instances created after the
installation of Zope 2.3.1b2 or better will be created using the
new BTree data structures and need not be converted.
- I've never used the catalog before, and I have no data to
convert from an old catalog, but I'm using Zope 2.3.1b2 or better.
Do I need to pay attention to any of this?
- No. Only folks who have data in "old" catalogs that they
wish to upgrade to the new catalog data structures need to pay any
attention to this whatsosever.
- How long will the conversion take?
- This depends on how much data you have in your Catalog, as
well as how many indexes you have, and what types of indexes they
are. On a Pentium III 800 with 256MB RAM, with a catalog
consisting of 40,000 cataloged objects against 9 indexes (5 text,
4 field), the conversion via
manage_convertBTrees takes about a
second and a half. Not very long, in other words.
- With an "old" ZCatalog (shipped with 2.3.0 or less), it
seems that some of my documents aren't indexed properly
or that I can't unindex documents properly.
- Consider rebuilding the catalog via the
"Advanced ->Update Catalog" button. There were some bugs
in Catalogs prior to the one which ships with 2.3.1b2.
Rebuilding the catalog will allow you to index and unindex
- What sort of "real world" benefits will I see in catalog
operations when I start using a catalog with the new BTree data
structures as opposed to a catalog with the "old" BTree data
- Sites with high write traffic will likely see fewer ZODB
ConflictErrors due to the new BTree design. This will cause fewer
transaction retries, speeding read and write performance overall.
Incremental indexing operations (such as those performed by
"CatalogAware" objects) will cause less egregious "database bloat"
in undoing storages like FileStorage. Queries which use AND, OR,
OR NOT, or NEAR operations will likely be faster. Bulk reindexing
operations may use less RAM.
- I'm using ZEO, and I make heavy use of the ZCatalog. I'd
like to convert all of my ZEO clients individually, while keep my
production site "up and running" during the upgrade. Do I need to
do anything special?
- If you want to perform an "in-place" upgrade of all your ZEO
clients, it's highly recommended that you disallow any writes to
all ZCatalog instances in your database for the duration of the
upgrade process (possibly by disallowing user access to code which
does catalog writes) as outlined in the first "A" above. Once
you have finished upgrading all of your clients' Zope files, you
should restart all of the clients and initiate an
manage_convertBTrees against each of your ZCatalog instances
from a single ZEO client only. Subsequently, all of the clients
will use the new data structures. You may then reenable writes to
The downtime caused by the inability to do writes to a heavily
used Catalog for any period of time may be painful for some
sites, but we expect that the benefit of performing the
upgrade will outweigh the detriment of losing functionality
for a period of time, especially for users of ZEO.
- I've updated my catalog and nothing seems different. Why?
- This is actually a good sign! The changes we've made to the
ZCatalog are not "user-visible". There have been no significant
changes to the API or the management interface. Only the
underlying data structures have changed.
- What should I do if I have a problem after I convert my
- First of all, try to undo the transaction for
manage_convertBTrees by visiting the the Catalog's Undo tab. If
that doesn't seem to help, restore your "old" ZODB datastore from
a backup. Then post a message to the Zope mail list
(firstname.lastname@example.org) with the following information:
- a "traceback" if there is an error message during operation
with the new data structures (usually available via the HTML
source of the error page)
- the version of Zope you're upgrading from, and the version
you're upgrading to.
- the symptoms of the problem.