MountedDatabases

Mounted databases

Problem: It would be desirable to mix multiple storages or multiple databases in the same of Zope object space. For example one might want to mix storages with difference semantics. Objects for which undo or histories were important would be stored in storages that supports undo and kept revision information. Objects that change often or for which undo would be inappropriate might be stored in a different storage that didn't keep multiple revisions. We must try to approach this by allowing multiple storages to be combined in a single storage however accommodating multiple semantics within the same storage seems problematic. An alternative approach is to allow multiples of databases with the road semantics to be combined in a single object system.

Allowing multiple databases within a single object system has been considered from the beginning of the development of the Zope object database. From the beginning the transaction system assumed that there might be multiple databases present within the application. This has been used to integrate relational transactional semantics within the larger transaction system. It was envisioned that multiple object databases would be present and that objects in one database could refer directly to objects in other databases.

This presents some significant problems however. Maintaining the necessary database connections when traversing from one object to another is rather complicated, although doable. Garbage collection in the presence of multiple databases with arbitrary connections across databases is extremely difficult. In fact it may be an unsolved problem.

It was suggested by Philip Eby that a simpler approach might be taken. Some application-level object might provide a bridge to a second database in much the same way that SQL methods and SQL database connections provide a bridge to relational databases.

Recent experiments along these lines have been very promising. On the basis of these experiments we are moving forward to provide both the infrastructure support and example products that allow multiple Zope databases to be used within a single cell process. This document describes the basic assumptions and requirements of such a system.

Basic model

The model of the mounted database approach on the mounting of file systems in Unix system. We will allow an object from one database to be mounted or inserted into the object space of another database. We will allow out name references to span multiple databases in much the same way that symbolic links can span multiple file systems in a Unix system. We will disallow a direct object references across databases in much the same way that's hard links are disallowed across file systems.

Basic approach

We will use mount objects. These small objects will know how to retrieve an object from a database. They will provide an __of__ method that returns the object retrieved it from the database in the context of their container:

      def __of__(self, parent):
          data=getattr(self, '_v_data', None)
          if data is None:
              db=getattr(self, '_v_db', None)
              if db is None: db=self._connect()
              j=db.open(version=self._p_jar.getVersion())
              self._p_jar.onCloseCallback(self._close)
              app=j.root()['Application']
              data=self._v_data=Acquisition.aq_base(
                  app.unrestrictedTraverse(self._path))

          return data.__of__(parent)

      def _close(self):
          self._v_data._p_jar.close()
          del self._v_data

MartijnF?: wikis sometimes are a bit tricky with source code; when I read it as web page, app=jroot()['Application'] has the square brackets stripped off.

MindLace?: corrected

When a mount object retrieves an object from its database it must open a database connection. It needs to assure that the database connection is closed at the end of the current request. It does this by registering a callback with the parent object's database connection. When the parent object's connection is closest the callback will be called and the mount object's database connection will be closed.

Issues

We need a callback registration mechanism in database connections. (Update by Shane: Completed.)
We need to modify the copy and paste mechanisms so that they use object paths rather than monikers. Monikers use object IDs? from a single object database. They could be extended to record object database information in addition to object IDs? however this would require a database registry which would be desirable to avoid. By relying on object paths instead, we leverage the traversal capabilities of Zope and the ability of mount object's to participate in traversal. (Update by Shane: Completed.)
We need to decide what the semantics of ZClasses? should be in a mounted database environment. ZClasses? are stored in object database. If we allow references from an instance in one database to ZClasses? in another database we would be almost violating the cross-database reference restriction. ZClasses? are a little bit different than other objects in that they don't use direct references between instances and ZClasses?. Rather indirection is used. A ZClass?' instance records a global ID for the ZClass? the global ID is used to look up the ZClasses? when the instance is loaded from the database.
At least initially we will impose a restriction that a ZClass? can be created in a mounted storage only if the mounted storage has a registration for the ZClass?. The easiest way to accomplish this, would be to use another Zope database as a mounted storage and to define the ZClass? in the control panel for the database that is mounted.

Update

With the above restriction, mounted databases work with ZClasses?.
We need to modify the ZODB logic to prevent cross database references. While we may allow cross-database references some day, some protocol should be used at that time to prevent invalid references. In the short term we will simply disallowed references across databases. (Update by Shane: Jim has been setting this up and it may be working already.)
Someone needs to come up with the UI for managing mounted databases. Some people want to be able to be able to create and mount arbitrary databases through the web. This is controvercial for security reasons.

anthony - 1 Jun 2000: I'd be curious to know about the security issues here. The only one I can think of right now is that you could mount an object from a ZODB underneath a different acl_users where the username would have a different meaning. But I think that this is handled by the newer security model, where there's a link back to the actual ZODB user folder(?). It could also be addressed by not allowing mounts to anything but the root of the new ZODB.

Progress (Shane) - 22 May 2000

A mounted database is now running on zope.org. The updated copy/paste is not yet installed on zope.org but the patch is in the CVS repository. Once we got it going, we have not had any problems that we know of.

The mount work is on a branch in CVS, but the changes required are so small that I wouldn't be surprised if the changes went into the next release.

The UI issue is quite significant. We might consider the possibility of managing databases from the control panel, in a centralized way, rather than the normal Zope decentralized way.

Advantages to centralized database management:

One place to go for packing.
System administrators would always know the status of the various databases.

Disadvantages:

(Add your own comments)

Advantages to decentralized management:

Creating a separate database for each user is more straightforward. (Excellent option for ISP's.)
anthony: it allows you to roll out different chunks of a large ZODB in pieces; currently you have to push the entire ZODB in one large update (if you're doing a staged development).
anthony: I was wondering about how you could mount/unmount various ZODB's, so I hacked around in the code a bit and figured out the magic. If you delete the mountitem, and it's the last reference to the FileStorage (or whatever) it just quietly unmounts it. Very neat. It means you can push out different pieces on the fly. You can also make multiple mountpoints into the same FileStorage (symlinks :).

Disadvantages:

Currently the mounted database on Zope.org is installed in a decentralized way. I didn't realize this problem until I saw it on the Zope management screen: there is no hint at all that an object is a mounted object. It just looks like any other object. That can be confusing!

MountedFileStorage

See the MountedFileStorage for a product that allows you to mount multiple file stores.

MountTab

anthony, 2000-06-05. I'm trying to put together some code to produce a page which lists the mounts in the current database in a sane-ish way. There's a number of issues with this that I've put on a seperate page.