Plone with a clean development environment on Leopard

I still have a G5 PowerMac. It runs Mac OS X 10.5 Leopard. I know it’s old, but since it’s a PowerPC I can’t upgrade the MacOS. It’s still very relevant for doing Plone development, as Apple has a long history of installing old libraries anyway, so managing that stuff is just part of life with Apple. Here’s how I set it up to do development for Plone in a clean way:

When you install Plone using the installers with default options you get a virtual environment. Changes you make to your buildout in there generally don’t affect your operating system. When you want to develop a Plone product or do core development, however, you’ll be working from a Subversion working copy, not a normally-installed Plone setup. The development environment instructions gets you set up with some nice tools, but there are a couple of additional steps you can do to keep your development environment nice and tidy.

Things you’ll need to have a sane development situation on OS X:

  • XCode Tools – Version 3.1.4 seems to be the last PPC-compatible version. It’s there, but you’ll need to register and dig for it. XCode 3.1.4 includes…
  • Subversion – version 1.4.4, [update: Plone now uses git] which seems to be just new enough to work. If you are reading this later on and 1.4.4 doesn’t work anymore, install a new version using…
  • Homebrew – or another package manager of some kind. I’ve used MacPorts, Fink, Gentoo Prefix, and now I’m on the PowerPC branch of Homebrew. I got this idea from David Glick. However, I almost gave up when complications arose from removing MacPorts. I’m sure you can adapt these instructions to whatever package manager you want. I like Homebrew because it doesn’t duplicate what the OS provides, such as…
  • Python – but Plone doesn’t work with Python 2.5, so you’ll need to install Python manually, because Homebrew doesn’t duplicate what the OS provides. You’ll want Python 2.6 for Plone 4 and Python 2.4 for Plone 3. For 2.4 you’ll want to compile it with a special MACOSX_DEPLOYMENT_TARGET=10.5 flag. Whatever Python you have, you’ll want…
  • PIP – You can get by with easy_install, but despite OS X having surprisingly good segregation for site-packages, who wants to live without uninstall? Just easy_install pip, and never use easy_install again. Then use pip to install…
  • virtualenv – Creates silos in which to put Plone setups so they don’t contaminate your system and your system doesn’t contaminate your Plone. You can pip install this with the system Python and point it to a specific Python version later, when you use it.

How to set up a virtualized Plone development environment:

Set up a virtual environment.
$ virtualenv -v -p /usr/local/bin/python2.6 --no-site-packages --distribute ./py26env
Check out the Plone development buildout.
$ svn co  https://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.1/ ./plone41devel

[Update: Plone now uses git.]

Change directories into your working copy.
$ cd ./plone41devel
Invoke bootstrap.py with the virtualenv’s python.
$ ../py26env/bin/python bootstrap.py && bin/buildout

Set up a local buildout configuration

The buildout configuration files for Plone are in Subversion. If you are a core developer and make a change you run the risk of accidentally committing that change to the core. A file named ‘local.cfg’ should be in svn:ignore, which means you can create it and edit it to your heart’s content without worrying about it getting caught up in your feverish fixing. It needs to hook back into the real buildout.cfg though, so it should look something like this:

[buildout]
extends =
	buildout.cfg

Then you just run bin/buildout -c local.cfg and after that you can use bin/develop rb.

Fixing PIL and lxml for OS X

Plone 4.1 needs PIL and lxml to be truly happy, but OS X makes it particularly challenging to install these. Fortunately, others have solved this problem, so I’m just going to give you my local.cfg that takes care of it. PIL is easy: Alex Clark’s Pillow does all the work for us. The real trick is the redefinition of the [instance] part with an arbitrary dependency on [lxml]. That’s necessary to force the lxml part to be run before anything else, which is the only way it will work.

[buildout]
extends =
	buildout.cfg
eggs +=
	Pillow
parts +=
	lxml

[lxml]
recipe = z3c.recipe.staticlxml
egg = lxml
build-libxslt = true
build-libxml2 = true
static-build = true
libxml2-url = ftp://xmlsoft.org/libxslt/LATEST_LIBXML2
libxslt-url = ftp://xmlsoft.org/libxslt/LATEST_LIBXSLT

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs = ${buildout:eggs}
environment-vars =
	zope_i18n_compile_mo_files true
	_dummy ${lxml:egg}

Finally, if you want to check out a package, you can modify [sources] in local.cfg. Just add a sources part:

[buildout]
extends =
…
[sources]
my.product = svn svn://uri

and run bin/develop co my.product && bin/develop rb to check out and install most eggifiable repositories.

Plone 4.1 with Apache and mod_wsgi (sorta)

Update 2011-06-01: Still not working perfectly, but I did manage to clean up a couple steps by using more of mod_wsgi’s bells and whistles:

  • Using a .pth file and the WSGIPythonPath directive you don’t need to manually load all the eggs into sys.path
  • Using Daemon Mode per Graham Dumpleton’s comment. To be honest I haven’t noticed any difference caused by this change…

Issues:

  • The site is still slow (despite using Daemon Mode)
  • I get signal errors. This is a known problem when trying to serve signal-dependent python stuff via mod_wsgi, but I’m not sure it causes any real problems.
    [Wed Jun 01 17:59:40 2011] [warn] mod_wsgi (pid=12739): Callback registration for signal 10 ignored.
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/zeocluster/zope2.wsgi", line 3, in 
    [Wed Jun 01 17:59:40 2011] [warn]     application = make_wsgi_app(None, '/var/www/Plone-ZEO-4.1rc2/zeocluster/parts/client1/etc/zope.conf')
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/Zope2-2.13.7-py2.6.egg/Zope2/Startup/run.py", line 68, in make_wsgi_app
    [Wed Jun 01 17:59:40 2011] [warn]     starter.prepare()
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/Zope2-2.13.7-py2.6.egg/Zope2/Startup/__init__.py", line 90, in prepare
    [Wed Jun 01 17:59:40 2011] [warn]     self.registerSignals()
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/Zope2-2.13.7-py2.6.egg/Zope2/Startup/__init__.py", line 340, in registerSignals
    [Wed Jun 01 17:59:40 2011] [warn]     self.cfg.trace])
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/Zope2-2.13.7-py2.6.egg/Signals/Signals.py", line 115, in registerZopeSignals
    [Wed Jun 01 17:59:40 2011] [warn]     SignalHandler.registerHandler(SIGUSR1, showStacks)
    [Wed Jun 01 17:59:40 2011] [warn]   File "/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/Zope2-2.13.7-py2.6.egg/Signals/SignalHandler.py", line 37, in registerHandler
    [Wed Jun 01 17:59:40 2011] [warn]     signal.signal(signum, self.signalHandler)
  • In many cases I can’t POST to the site. I can log in as long as it’s not HTTP Basic, but I can’t edit pages or make site-setup configuration changes. Not sure why.

Notes:

  • For the following instructions I did every step as the user apache, so Plone runs as that user, as does zeo. This may not be the absolute best practice, but it made things a bit simpler.
  • You can substitute instance in probably every case I used client1.
  1. Install Apache and mod_wsgi. Make sure to specify Python 2.6 for mod_wsgi.
  2. Grab a Plone 4.1 release candidate and install it as the user apache. Use the same Python 2.6 as you did for mod_wsgi.
  3. Create a path configuration file:
    $ ( # Do this in a subshell so we don't contaminate the IFS
    >   # variable in our normal shell.
    > cd /var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs
    > eggs=( *.egg )
    > IFS=$'\n'
    > echo "${eggs[*]}" > mod_wsgi.pth
    > )
  4. In your buildout directory create an empty file called ‘zope2.wsgi.in’ as a collective.recipe.template template.copy the bin/client1 file into zope2.wsgi.in (because you need all the egg paths)
  5. Put these two lines in it:
    from Zope2.Startup.run import make_wsgi_app
    application = make_wsgi_app(None, '${zope-conf}')
  6. Add a section called wsgi to your buildout.cfg file:
    [buildout]
    …
    parts =
            …
            wsgi
            …
    …
    [wsgi]
    recipe = collective.recipe.template
    input = zope2.wsgi.in
    output = zope2.wsgi
    zope-conf = ${client1:location}/etc/zope.conf
  7. In your apache config:
    …
    WSGIPythonPath /var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/
    WSGIDaemonProcess neon processes=1 threads=1 python-path=/var/www/Plone-ZEO-4.1rc2/buildout-cache/eggs/
    WSGIProcessGroup neon
    WSGIScriptAlias / /var/www/Plone-ZEO-4.1rc2/zeocluster/zope2.wsgi
    
    <Directory "/var/www/Plone-ZEO-4.1rc2/zeocluster">
            Order allow,deny
            Allow from all
    </Directory>
    …
  8. Run buildout and start the zeoserver.
  9. I had some problems with HTTP Basic Authentication through WSGI, so I avoided it by starting the client without WSGI for the purpose of logging in to the ZMI to create the Plone site. bin/client1 fg
  10. Log in to http://localhost:8080 and create the Plone site.
  11. Once I had actually created the Plone site, I killed the client1 instance and removed the <http-server> section from zope.conf
  12. Start apache, and observe the logs as you navigate around your Plone site on port 80.

As I said, my site is slow and a lot of errors appear in the logs, but it’s functional, which is better than I’ve seen it so far. More work on this later.

How to build a simple shared zeoclient-only setup for Plone 3

Note: The following is probably very adaptable to Plone 4. In fact, I suspect it’s even easier in Plone 4. But I’m working on Plone 3 right now, so there you go.

This is the second in a two-part guide on how to separate the Plone database server from the client instances. Even with load-balancing proxies, convenient high-availability in Plone always remains just out of reach. As long as the zeoserver and its clients are on the same machine, you have to consider both server and clients in your change control processes. Separation is simplification. I’ve given other reasons to care in my previous post, but as an update, here is the approximate storage size of a single client-only site with a lot of eggs:

$ du -hs *
53M	Python-2.4
73M	Zope-2.10.11-final-py2.4
149M	testcase

Getting started with the clients

  1. Decide how you will structure your filesystem:
    • Shared Python 2.4: /opt/plone/Python-2.4
    • Shared Zope 2.10: /opt/plone/Zope-2.10.11-final-py2.4
    • Buildout for zeo: /opt/plone/testcase
  2. Create the user that the clients will run as. Create the parents of the above paths and give that user ownership. Become that user for the rest of these instructions.
    useradd plone
    mkdir /opt/plone && chown plone /opt/plone
    su - plone || sudo -u plone bash
  3. Download and unpack the Universal Installer (UI) from Plone.org. Use the UI to install a shared Zope and Python, but nuke the rest of what it installs for now.
    ./install.sh --target=/opt/plone zeo && rm -rf /opt/plone/{buildout-cache,zeocluster}
  4. Use the UI to install an actual zeoserver using that shared Python and Zope:
    ./install.sh --target=/opt/plone/testcase --with-python=/opt/plone/Python-2.4/bin/python --with-zope=/opt/plone/Zope-2.10.11-final-py2.4 zeo
  5. Go to the zeocluster directory and clean out the buildout stuff that is only needed for Plone clients. You really only need bin/buildout, buildout.cfg. If you’re using bash you can do this quickly with extglob:
    cd /opt/plone/testcase/zeocluster && shopt -s extglob
    rm -rf ../buildout-cache/ bin/!(buildout) !(bin|buildout.cfg|versions.cfg) .installed.cfg
  6. Create a backup of the buildout file and begin working on it. You can delete almost everything in there, just keep the zope2, zopepy and client parts and fix any broken references. Here’s what mine looks like:
    [buildout]
    extends = versions.cfg
    versions = versions
    extensions = buildout.dumppickedversions
    zeo-address = 10.0.0.87:8100
    client1-address = 8080
    client2-address = 8081
    eggs = Plone
    zcml =
    develop =
    debug-mode = off
    parts =
            zope2
            client1
            client2
            zopepy
    
    [zope2]
    recipe = plone.recipe.zope2install
    fake-zope-eggs = true
    additional-fake-eggs =
            ZConfig
            pytz
    location = /opt/plone/Zope-2.10.11-final-py2.4
    
    [client1]
    recipe = plone.recipe.zope2instance
    zeo-client = true
    zeo-address = ${buildout:zeo-address}
    effective-user = plone
    user = admin:abc
    http-address = ${buildout:client1-address}
    debug-mode = ${buildout:debug-mode}
    verbose-security = ${buildout:debug-mode}
    deprecation-warnings = ${buildout:debug-mode}
    eggs = ${buildout:eggs}
    zcml = ${buildout:zcml}
    environment-vars = PYTHON_EGG_CACHE ${buildout:directory}/var/.python-eggs
    zope2-location = ${zope2:location}
    
    [client2]
    < = client1
    http-address = ${buildout:client2-address}
    
    [zopepy]
    recipe = zc.recipe.egg
    eggs = ${buildout:eggs}
    interpreter = zopepy
    extra-paths = ${zope2:location}/lib/python
    scripts = zopepy
    
    [versions]
    Cheetah = 2.0.1
    Paste = 1.7.2
    PasteScript = 1.7.3
    ZopeSkel = 2.11.1
    collective.recipe.backup = 1.1
    plone.recipe.command = 1.0
    plone.recipe.distros = 1.5
    plone.recipe.osxcontroller = 0.3
    plone.recipe.precompiler = 0.3
    plone.recipe.unifiedinstaller = 0.9
    collective.recipe.zope2cluster = 1.0
    PasteDeploy = 1.3.3
    zc.recipe.egg = 1.2.2
  7. Once you’re satisfied with your buildout configuration, run buildout, fix permissions and try to start the instance. (Note: buildout always crashes the first time I do this because of a setuptools incompatibility. Just run it again if it fails.):
    bin/buildout || bin/buildout && bin/client1 start && bin/client2 start
  8. That’s it! Now to create a new client setup, just repeat steps 5-8, updating the paths and port numbers (zeo-address, clientN-address) to suit.

That’s really all there is to it.

How to build a simple shared zeoserver-only setup for Plone 3

Note: The following is probably very adaptable to Plone 4. In fact, I suspect it’s even easier in Plone 4. But I’m working on Plone 3 right now, so there you go.

When you first install Plone, you get everything together: server, client, whatever, it just works. I’m sure not everybody knows, or cares, that Zope can be run as a zeoserver distinct from the zeoclient. But there are some reasons to care:

  1. Simplicity: Plone can seem complex, but zeoserver is dead simple. Separating it from the clients is the easiest way to see this.
  2. Flexibility: Need to reboot or migrate a server without site downtime? You can’t, unless you’re using load-balancing proxies. But with zeoserver on a separate machine from the clients, the clients can withstand short outages of the zeoserver without apparent downtime, even if you don’t have a load balancer! (YMMV depending on demand, zeoserver’s reboot time and the clients’ caches.) Moving the clients gets easier, too.
  3. Backups: I don’t need to backup all of Plone. The only important part is the Data.fs (and in Plone 4, the blobs, but that’s for another day). Now my backup scenario can effectively ignore the client servers, and just back up what’s on the server with the ZODBs on it.
  4. Scalability: I’ll let these results speak for themselves:
    Before
    $ du -hs --exclude '*.log' plone
    526M     site1
    407M     site2
    676M     site3
    After:
    $ du -hs *
    53M	Python-2.4
    73M	Zope-2.10.11-final-py2.4
    11M	site1
    11M	site2
    11M	site3

    Well, that’s not quite fair – the client code isn’t listed there, but it’s no longer an issue for this server.

Hopefully that’s enough to get you interested.

Getting started

  1. Decide how you will structure your filesystem:
    • Where the Data.fs will be stored: /srv/plone/testcase/Data.fs
    • Shared Python 2.4: /opt/plone/Python-2.4
    • Shared Zope 2.10: /opt/plone/Zope-2.10.11-final-py2.4
    • Buildout for zeo: /opt/plone/testcase
  2. Create the user that the zeoservers will run as. Create the parents of the above paths and give that user ownership. Become that user for the rest of these instructions.
    useradd zeo
    mkdir /srv/plone && chown zeo /srv/plone
    mkdir /opt/plone && chown zeo /opt/plone
    su - zeo || sudo -u zeo bash
  3. Download and unpack the Universal Installer (UI) from Plone.org. Use the UI to install a shared Zope and Python, but nuke the rest of what it installs for now.
    ./install.sh --target=/opt/plone zeo && rm -rf /opt/plone/{buildout-cache,zeocluster}
  4. Use the UI to install an actual zeoserver using that shared Python and Zope:
    ./install.sh --target=/opt/plone/testcase --with-python=/opt/plone/Python-2.4/bin/python --with-zope=/opt/plone/Zope-2.10.11-final-py2.4 zeo
  5. Go to the zeocluster directory and clean out the buildout stuff that is only needed for Plone clients. You really only need bin/buildout, buildout.cfg. If you’re using bash you can do this quickly with extglob:
    cd /opt/plone/testcase/zeocluster && shopt -s extglob
    rm -rf ../buildout-cache/ bin/!(buildout) !(bin|buildout.cfg|versions.cfg) .installed.cfg
  6. Create a backup of the buildout file and begin working on it. You can delete almost everything in there, just keep the zope2 and zeoserver parts and fix any broken references. Here’s what mine looks like:
    [buildout]
    extends = versions.cfg
    versions = versions
    extensions = buildout.dumppickedversions
    dump-picked-versions-file = picked-versions.cfg
    zeo-address = 8100
    file-storage = /srv/plone/testcase/Data.fs
    parts =
    	zope2
    	zeoserver
    
    [zope2]
    recipe = plone.recipe.zope2install
    url = ${versions:zope2-url}
    fake-zope-eggs = true
    additional-fake-eggs =
    	ZConfig
    	pytz
    
    [zeoserver]
    recipe = plone.recipe.zope2zeoserver
    zope2-location = ${zope2:location}
    zeo-address = ${buildout:zeo-address}
    effective-user = zeo
    file-storage = ${buildout:file-storage}
  7. Once you’re satisfied with your buildout configuration, run buildout, fix permissions and try to start the instance. (Note: buildout always crashes the first time I do this because of a setuptools incompatibility. Just run it again if it fails.):
    bin/buildout || bin/buildout && chown -R zeo:zeo . && bin/zeoserver start
  8. That’s it! Now to create a new zeoserver, just repeat steps 5-8, updating the paths and port number (zeo-address) to suit.

That’s really all there is to it. Tomorrow I’ll post how to put together the client side.