################################################################

$RCSfile: README.txt,v $

Author: Craeg Strong

$Date: 2003/04/21 01:52:42 $

################################################################

Contents

1. Description
2. Advantages, Disadvantages, and Alternative Products
3. External Dependencies
4. CVS Sandbox Registry
5. Which Sandbox?
6. How are files served up from sandboxes?
7. Development Workflow
8. Creating Instances Programmatically
9. Contributions
10. Schema Migration
11. Notes
12. Unit Testing on Win32

Description

The CVSFile package is a Zope product. CVSFile allows one to use CVS as a version control system for Zope content. The developer creates a CVSFile, pointing to a file in the developer's CVS Sandbox on the Zope server. Other Zope objects are able to access the content, but it is actually stored on the disk, rather than in the ZODB. CVSFile enables the user to modify the file and do simple CVS commands. CVSFile inherits from ExternalFile, a Zope product that behaves like a standard Zope object like File, Page Template,or DTMLMethod, but points to external content in the filesystem. Please read the ExternalFile Product README before proceeding.

The reason we are creating this new product is to support our usage of CVS for revision control. We would like to be able to continue to store website content within CVS as we did in our pre-Zope days, while at the same time utilize the full power of Zope.

With CVSFile, every developer can use the same, single Zope server instance and yet they each see their own versions of files. This is a crucially important point and a significant advantage for CVSFile. It is done through browser cookies. See below for a more detailed explanation.

In this way, CVS is used for what it is best at: revision control, support for parallel development, branching, merging, etc. Similarly, Zope is used for what it is best at: support for acquisition, editing through the web, DTML, integrated security, undo, etc.

Of course, CVSFile does not store everything in CVS. The central idea behind CVSFile is to store files in CVS and store metadata in ZODB. It is therefore most useful for those who already have significant content stored in CVS, but want to use Zope as a publishing environment. You might even have multiple CVSFile instances in the ZODB all pointing to the same file in a CVS sandbox, but with different Zope metadata (properties, etc.).

Note: CVSFile is not really intended for use in a production setting, (although we happen to use it that way :-). Larger organizations tend to want tighter controls in production settings; for them, highly granular versioning and collaborative development are more important during the development and testing phases of a project. In such organizations, in order to move into production, the contents of a website should go through a release process, part of which usually involves "freezing" or "snapshotting" the content. We hope to enhance CVSFile to better support such a "macro" development process in future. For additional discussion, refer to the "Development Workflow" section.

There are two ways to create CVSFiles from the Zope Management Interface (ZMI): one at a time or in a batch. To use the batch option, simply select "CVS File Batch" from the standard Zope Add menu. It provides a way to automatically add an instance of CVSFile for every file within a directory. Because CVSFile instances are merely links or aliases, they can be deleted without affecting the underlying file. Therefore, an easy way to create CVSFile instances is to create them a batch at a time and then delete the unwanted instances (for example, those pointing to junk files, "tilde files", etc.).

CVSFile is a subclass of ExternalFile, therefore it inherits all of its behaviors. Please refer to the ExternalFile README file for a important additional information.

Advantages, Disadvantages, and Alternative Products

Advantages: Files stored in CVS are directly usable by other tools besides Zope. Zope stores the Zope-specific metadata, CVS stores the non-Zope specific data.

Disadvantages: Metadata is not versioned in CVS, and remains only in the ZODB. This means that the Zope environment is not reproducible from the information in CVS alone.

If:

a) it is an absolute requirement to version all metadata, as well as all data, and

b) you are starting from scratch and don't already have a large number of files in CVS,

perhaps it would be better to use a product that "pickles" Zope products and stores their complete contents in CVS (data and metadata). See ZCVSFolder . The major disadvantage with ZCVSFolder, and the reason we did not use it, is that everything stored in CVS as "pickles," and is therefore not directly useable by other tools.

External Dependencies

CVSFile calls out to CVS via standard Python system calls, and therefore requires the "cvs" commandline program to be installed and available in the PATH -- that is, the PATH used when starting up Zope (e.g. something like "python z2.py"). Fortunately, CVS is available for nearly all platforms from cvshome . CVSFile doesn't require any particular version of CVS, and you should be able to use it on UNIX and windoze using either the regular or client/server flavor.

NOTE

CVSFile does not currently provide any support for doing "cvs login" through the web. Therefore, if you are using the client/server flavor of CVS, you will have to do a "cvs login" on the Zope server machine before executing any CVSFile commands.

CVS Sandbox Registry

In order to enable parallel development as much as possible, we want to avoid strict locking such as required by various CVS+Web products like CVSwebClient. They require strict locking in order to deal with possible race conditions as they provide direct access to the CVS repository itself. Instead, we would like to point at a CVS sandbox. What does this mean? In order to avoid these locking issues, we would like each developer to access content from his own CVS sandbox, while using the central Zope server as per usual for navigation and non-CVS Zope objects.

We have therefore created an object called a CVSSandboxRegistry. This object represents a logically related group of CVS sandboxes. Examples of logically related groups might be: the "development" group, or the "QA" group, but any grouping is fine. Information about each of the sandboxes within the group is held in the registry object. Every registry object must have one and only one sandbox designated as the "default sandbox." A registry object contains information about at least one, and at most arbitrarily many sandboxes.

In order to create an instance of CVSFile, or to perform CVS commands on an existing instance of CVSFile, or to view the contents of a CVSFile (either in the second view (ZMI) or third view (user web site)), there must exist an instance of CVSSandboxRegistry that is available via acquisition. The CVSFile instance obtains the following information from the registry:

• which sandbox to use (the "current" sandbox), and
• the "current" sandbox base directory.

How does a CVSFile instance find a CVSSandboxRegistry?

The current lookup policy is as follows: use the first registry found by acquisition. Alternatively, a more complex policy could be used such as

• looking in some predefined place, where the place could be specified in some property (a la Zope3 explicit acquisition), or
• finding all registries in the ZODB and presenting a list for selection by the user.

Feel free to adopt either policy by reimplementing the function findCVSSandboxRegistry() in the CVSFile.py module. YMMV

NOTE Because of the way it is currently implemented, you cannot create a CVSSandboxRegistry in the Zope Root Folder. Instead, it must be in a subfolder of ZODB.

Which Sandbox?

The registry object enables a user to select a particular sandbox in which to work. All CVSFile objects within the scope of this registry object will then operate with respect to the selected sandbox. The information that identifies which sandbox is selected is held in a browser cookie. If the cookie is not present for whatever reason, the "default" sandbox is automatically used. This is the reason that there must always be exactly one sandbox within a CVSSandboxRegistry that is labelled the "default."

The default sandbox is very important. It is the one that content will be served out of for anyone who has not made a sandbox selection, either because they decided not to, or because they do not have access to the ZMI. For example, the general public!

To create a registry object, the initial default CVS sandbox must be registered. However, new CVS sandboxes may be added at any time, and the selection of which is the default can be changed at any time.

The registry object allows the user to select a CVS sandbox in which to work, to change that selection at any time, and to perform CVS operations across the user's currently selected sandbox as a whole or, (eventually, when we implement it!) across all sandboxes within the group.

When the user selects a sandbox in which to work, a browser cookie is automatically created that records this choice. The cookie is given a very long "lifetime" by default, because developers or content authors usually want to work in the same sandbox every day. In this way, a developer may only have to select a sandbox once per project, or even less frequently.

Confusion may result, however, if a developer changes to a different browser. For example, say a developer normally uses Internet Explorer, and has made his sandbox selection within the ZMI using IE. This means that IE is holding a cookie that records this selection. If the developer fires up Opera and looks at the content, without visiting the registry object to select his sandbox, he will potentially see different content in the two browsers. This is because the Zope "view" in IE is serving up CVSFile content from his selected sandbox, while the Zope "view" in Opera is serving up CVSFile content from the "default" sandbox.

We are currently looking into ways to solve this problem. Please feel free to forward any helpful suggestions. The main issue is that we need a better understanding of typical usage patterns for developers and content authors.

How are files served up from sandboxes?

CVSFile is a subclass of ExternalFile, so every instance of CVSFile is, of course, an ExternalFile. As such, it records the pathname of the file to which it is a reference in two parts: basedir and filepath.

CVSFile exploits this fact by equating sandbox location with basedir. Each CVSFile resolves its file reference relative to the base directory of the selected sandbox. The name of the selected sandbox is stored in a browser cookie, and the base directory of that sandbox is stored in a CVSSandboxRegistry.

In this way, we could have a CVSFile that refered to:

$basedir/com/arielpartners/knowledgebase/journal/j-20011017.xml

Which might resolve to:

/home/cstrong/ade/com/arielpartners/knowledgebase/journal/j-20011017.xml

for me and:

/home/zope/website-cvs/com/arielpartners/knowlegebase/journal/j-20011017.xml

for a QA tester.

[ assuming $basedir means the sandbox base directory ]

In this way, developers can use their local CVS sandboxes with their local Zope servers, or CVS sandboxes on a central Zope server, or both-- simply by pointing at different Zope servers and specifying different values for the sandbox name in their cookies.

Local versions of the Zope database (ZODB) can be cloned from the server using the ZSyncer automated synchronization product. See:

ZSyncer

for more details.

Development Workflow

CVSFile is intended for use in the development and testing phases of a project, for example to produce a corporate portal or website. This is when versioning and collaboration are especially important. Once a website has passed QA tests and is deemed ready for production, it should go through a release process. That release process usually involves "freezing" or "snapshotting" the content. The versions of content files used are given "official versions" (e.g. cvs tags) so that they may be retrieved later. In this way, the release could be recreated from scratch at a later time.

In the future, CVSFile could support the release workflow process. For example: do a CVS tag of a set of versioned files, archive them, and perhaps import them into the ZODB for production (so there are no production dependencies on external filesystems-- beyond that of the Zope server itself). The production versions would be "frozen," so that edit functionality would be disabled.

Creating Instances Programmatically

Normally instances of CVSFile are created by hand in the Zope Management Interface. There are a number of screens (a mini-workflow) that are traversed in order to ensure that the instance is created properly, to deal with all of the things that can go wrong (creating new files, overwriting existing files, permission issues, etc.) and to confirm exactly what was done.

However, there are times when you don't want that. You simply want to run a Python script and create an instance of CVSFile. Do not despair! This is easy to do. Check out the extras subdirectory. You will find the source code for an example: uploadCreate. It is commented and should be easy to adapt to your uses. Basically it calls the addCVSFile() method in CVSFile.py' directly, bypassing the GUI creation workflow code.

Contributions

We hope others find this code useful. If you have extended or improved this product, please feel free to submit your changes to us.

If there is enough interest, we would certainly consider setting up an open-source project on SourceForge.

Schema Migration

From time to time, you may find yourself with a new version of this product, either because we have released a "new improved" version or from some changes you may have made on your own. How do you deal with all of the existing instances in your ZODB that were created with the old definition? Here is our preferred technique:

1. Find the repair() method (we usually put it at the bottom of the respective python source file)
2. Change the repair() method so that it updates the object from the old version to the new version
3. Create a python script in the root folder that recursively calls repair() on each of the object with a given metatype
4. Execute the python script (the easy way is via the "test" tab)

Here is our python script. We call it "repairAll" and put it in the ZODB root folder:

    objects = context.ZopeFind(context, obj_metatypes=[metaType], search_sub=1)
    map(lambda x: x[1].repair(), objects)
    return map(lambda x: x[0], objects)

Notes

This document is written in structured text. For a quickie intro to structured text, look here.

Unit Testing on Win32

If you don't plan to run the automated unit test suite, this section is irrelevant.

For some reason, the automated unit tests don't run properly for me on win32 using the FAT file system under Zopes earlier than 2.5.1. I found the following workaround. In the file name:

    [ZOPE]\lib\python\Testing\custom_zodb.py

Where ZOPE stands for the directory in which you installed Zope, Change line number seven:

    Storage = DemoStorage(base=FileStorage(dfi,read_only=1), quota=(1<<20))

to instead read:

    Storage = DemoStorage(base=FileStorage(dfi, read_only=0), quota=(1<<20))

My guess is that FAT does not support the read_only attribute as required.