Why?

The stock lexicon deals well with 8 bit strings if you get the locale setting in zope.conf right; it does not work with Unicode and UTF-8. UnicodeLexicon fills this gap.

What?

This product adds a ZCTextIndex Unicode Lexicon type to Zope. The lexicon comes with word splitters, stop word removers, and a case normalizer.

If you have GenericSetup installed, you can use the provided extension profile to create Unicode lexicons in your portal_catalog and update the Title, Description, and SearchableText ZCTextIndexes.

Limitations

The lexicon assumes either Unicode or UTF-8. If your site employs e.g. UTF-16 (aka UCS-2) or UTF-32 (aka UCS-4) you will have to change the enc constant in UnicodeSplitter.py accordingly.

The extension profile installs lexicons without stop word removers. This is because only English language stop words are supported, and it is safe to assume you are using Unicode precisely because you need to handle non-English text.