CMFDeployment

Introduction

This tool allows for static deployment of a CMF site to multiple target systems akin to the deployment features of advanced commerical CMS systems like interwoven or vignette.

Features

• Allows for custom skins for deployment separate then content management interface.
• allows for business logic to filter deployed content using TALES expressions and Python Scripts
• Pluggable Deployment Strategies (Incremental, Full)
• Pluggable Deployment Protocols, (currently RsyncSSH)
• retraction of deployed content (using RsyncSSH protocol)
• Deployment to multiple servers
• Merging of DirectoryView (skin) resources into merged content.
• Provides for Mime and Extension mapping to content
• Provides for configurable URI Resolving of internal links within rendered content.
• extensive statistics gathering and logging frameworks
• and many more.

Todo Before 1.0

• speed/memory optimization of incremental strategy
• more documentation, online help
• management ui cleanup
• extreme load testing

Installation

Developed and Tested on Linux (2.4), Solaris 9, with Python 2.1.3, Zope 2.5.1 and CMF1.3

• Decompress distribution file and move to Products directory.
• restart zope
• in the root of the cmf site, add an external method:

        Id: install_deployment_tool
        Module: CMFDeployment.Install
        Function: install
  

• click on the test tab to complete installation

Optionally, you can install iuveno's OrderedFolder product to allow for explict ordering of filters, python scripts, and mime mappings. One warning, if your using OrderedFolder in conjunction with exUserFolder, you need to patch OrderedFolder to disable the dummy zbabel tags.

Debian users will need to install the pyxml package.

Configuration

The use of this tool requires configuration of several sections, outlined below. further documentation can be found in the docs directory.

Identification

Identifies Content that should be deployed. The Portal Catalog is used as a content source, TALES Filters and Python Script Filters can be used to filter the available content set, and restrict what content objects will be deployed.

Organization

Organizes the content in a target directory structure. The default structure and organization implementation, utilizes the existing cmf structure, although it does allow for mounting from a non portal root folder.

Mastering

The process of rendering/cooking the content.

• skin selection
• mime mapping configuration for content
• render method selection for content

URIS Link Normalization

During the mastering process the rendered view of content has extensions attached, and intra object references in rendered content need to be updated, as well as target url host information.

Directory View (skin) Merging

Skins .. Directory Views

certain resources of directory views often need to be deployed alongside content.

Deployment

The actual deployment of content to the target server.

Targets

Protocols

Libraries

This Product is distributed with the following third party modules which are included here under their original license terms, and are distributed here for convience (all such files can be found under CMFDeployment/lib)

• pypect:"http://pyexpect.sf.net" - Noah Spurrier (Python License)
• policycaller:"http://cvs.zope.org/Products/Scheduler/methodcaller.py" (ZPL)
• linuxproc:"http://www.zope.org/Members/mcdonc" (ZPL)
• LockFile:"http://www.list.org" (GPL)

License and Copyright

• @ kapil thangavelu 2002
• GPL

Credits

Kapil Thangavelu - Author

I'd like to thank alan runyaga for much dicussion and feedback on this tool. It has benefited greatly from his insights and wisdom.

I'd like to thank my employer for allowing me to opensource this tool that was developed at their cost. Without companies and clients such as them the open source world would be a much poorer place.

Introduction

A CMF/Plone browser for local subversion repositories. features basic directory and file browsing of any revision, and line by line file annotation for text files.

Pictures

worth a thousand words :-)

• DirectoryView
• Annotations
• FileRevisions

ExtIndex and ExtSearch

Two products to facilitate searching the file system from Zope.

ExtSearch

What does it do

Offers access file system searches via a swish search daemon. results returned are wrapped in brainz and offer access to search ranking, size,path and document title.

ExtIndex

What does it do?

Integrates with ZCatalog (as a Pluggable Index) and allows you to search file system content that an object might proxy for in conjunction with other ZCatalog maintained indexes on an object. Currently fs search is implemented through swish++.

swish++

Free Transactions

This is an attempt to associate zodb transactions on a per connection basis. the major benefits is by disassociating the transaction to thread association we can have multiple connections in a single thread and hence multiple dbs open while maintaining per connection transaction integrity.

functional changes to application code

Application code can continue to use get_transaction().register(ob) as long as ob is a persistent object or a zodb connection. Application code that tries to integrate non-persistent objects for transactional control needs to call get_transaction() with a persistent object or connection as an arg so that it can be registered with an appropiate connection.

Transitioning between transaction states cannot be done via get_transaction(), a persistent object or connection must be passed in so the proper transaction is returned.

what doesn't work

• versions... i've never really used versions, so i'm not sure about their implementation, other than it looks strange;)
• mounted storages. this is an alternative to this approach for multi db systems, but requires? treating multiple dbs as one logical unit. it might work i have no idea... it seems more a zopish thing at the moment.
• probably most advanced usages... some things i've seen like db.open with a transaction.

basically use of the full range of ZODB features (ie Zope) will not work.

otoh, most basic and common usages as a POS will probably work.

other caveats

multi-threaded code shouldn't have share connections, its probably better for such code to use and return conns to the db as needed.

installation

import this module and call the install_free_txn function, which will hotpatch the connection and transaction classes and install the get_transaction accessor in globals.

• transaction.py abort free transaction - connection.py sync method, pass in self

tests

i did some basic tests. if it breaks you get to keep both pieces

license

Zope Public License (ZPL) ZPL 1.0 or a later version of the ZPL at your discretion.

author: kapil thangavelu

What is it: A search interface to mailman archives. The python product is basically a retrieval script for mailman archives. (www.list.org) It splits them into individual emails and presents a glorified grep wrapper to Zope for searching them. Its disk intensive and resource intensive depending on the size of the archive. although its mainly proof of concept, it is functional. Top of the TODO list are some speed optimizations.

This is pre-alpha. I wouldn't recommend playing with it unless you're interested in developing it.

Page-Contract is a python product that allows for easy form/input validation from python scripts, dtml, and python products.

it adds a new type of dtml-tag to your zope installation. dtml-contract is a multi-purpose tag that is designed to provide documentation, marshalling and filtering of request data, and custom error handling of the request data. the documentation comes from a read only section of the contract which should describe the purpose of the object, its author, and some information about its parameters. the marshalling and filtering is done by naming variables expected in the request and associating one or more filters with those variables. PageContract comes with a set of prebuilt filters although it can easily be extended by writing new filters. lastly it allows association of different error handlers to different variables. an example of using it from a python script lives at http://www.zopelabs.com/cookbook/1008972251

RDBTools

Provides for pythonscript accessible functions to perform set operations on zsql method results.

Operations provided

• union_results
• join_results
• difference_results

The algorithms are based off Aaron Watters kjbuckets package and his early 1995 news posting to comp.lang.python.

Example Use Case

You have correlatable data in multiple dbs. Lets say employee contact info is in postgres, and payment info is in oracle. You want to aggregate this info into a single result set for a report display. a contrived example but a useful illustration.

contact table:

      create table corp_employee_contacts (
         employee_id   integer unique not null,
         email         varchar(255),
         first_name    varchar(30),
         last_name     varchar(30),
         middle_name   varchar(30)
      );

payment table:

      create table corp_employee_accounts (
         employee_id   integer unique not null,
         payment_period integer not null,
         payment_amount numeric default 0 not null
      )

zsql method: getEmployeeContacts:

      select * from corp_employee_contacts

zsql method: getEmployeeAccounts:

      select employee_id, sum(payment_amount) as total_pay
      from corp_employee_accounts
      group by employee_id

python script getEmployeeTotalPay:

      from Product.RDBTools import join_results

      contacts = context.getEmployeeContacts()
      accounts = context.getEmployeeAccounts()

      return join_results(accounts, contacts)

the resultant result set contains properly merged information from both results

Advanced Usage Note

Acquisition Contexts

ZSQL Results sets are bound to a context, the set operations exposed here preserve the context of the first result set argument.

Brains

brains are not preserved.

Speed

additional speed can be had by using installing the kjbuckets extension module, if its not installed the python implementation found in zope will be used.

Cross Products

if no common columns are found a cross product is returned.

Author

kapil thangavelu

License

X11, see LICENSE.txt for more details

SQLEngine

SQLEngine provides a method for loading SQLMethods from the filesystem

Methods are associated with a symbolic name by which they are invoked.

The goal of SQLEngine was to provide support for , work integrating Zope with the OpenACS, as such the default format supported by SQLEngine for filesystem SQL is OpenACS's query file xml syntax.

A second distinct goal of SQLEngine was to provide an abstraction to allow further work on changes to Zope's relational integration.

Why do i want to use it

If you want to keep your sql methods on the fs, either for source control or because they're generated from an external source.

Definitions

Sources

sources are locations where one or more resources files maybe found. Generally sources are either directories or jar files jars unimplemented.

Resources

Resources constitute files which contain sql statements. Resources may delimit sql statements in a variety of fashions, how a particular resource is handled is dependent on the config file within the SQLEngine product. The default handling is for xml resource files

Queries

Queries represent SQLMethods gathered from Resources. they have an associated name that is used for calling them.

Architecture

SQLEngine is divided under the hood into four parts. The SQLEngine itself a Zope object that provides managment interfaces for adding and removing sources/resources/queries and one interface for calling a query which takes a symbolic name for the query and args executes it and returns the results.

The other components are the Finder which searches the file system for files with given file extensions. The Loader extracts the raw query information from these files. The Builder takes this raw information and turns it into a QueryObject that gets stored with associated name in the SQLEngine.

The parts were designed to be interchangable, replacements for any of these components can be specified in the EngineConfig File.

Usage

Add a SQLEngine specifying a connection. Add a source, for the sake of example you can point it to the data directory of the SQLEngine distribution which contains some sample xml query files from the OpenACS. Load the source. The engine is now populated. It can be called via EngineName.execute(method_name, args)

Management

functionality is provided for reloading methods from the filesystem.

Futures

This is currently prealpha ware in the sense that it only serves my purpose of usage with openacs's xml syntax and does not provide alot of additonal capabilities beyond that. in either management or usage.

Author

Kapil Thangavelu License: GPL CVS: $Id: README,v 1.1 2001/09/28 05:00:56 ender Exp $

TaskEngine

Description

a oo framework for composition and execution of tasks.

designed for composing graphs, trees, or sets of functions, with multiple execution modes (pipeline, return registries, custom) and definition specified in various text formats.

its main design and product use was for installers and system maintainance. an example of which is the zope_builder.

this is a premature release

Install

via standard distutils command:

      python setup.py install

note this also installs several 3rd party modules including

• logging a logging facility (BSD)
• optik an command line argument parsing lib (BSD)
• mapply (ZPL2)

task engine has been tested on python versions 2.1 and 2.2

License

GPL

Author

kapil thangavelu

Transactional File System

Integrates with the zodb transaction system in attempt to present a simple file system interface that is transactional.

usage outside of zope requires copying zope's lib/python/Shared/DC/ZRDB/TM.py file and editing the import line for it in TransactionalFileSystem.py

to run the unit tests the TxnFs directory should be on your python path, as well as the ZODB, and TM.py.

License:

GPL

Author:

kapil thangavelu