My latest experimental web server needs a task queue to do asynchronous processing. Here’s how I deploy Celery on my Ubuntu server.
First Install Your RabbitMQ
Celery is a task queue, oroginally for Django, and then for any Python app, and now in principle usable from any programming language. It needs to queue tasks reliably and one way to do that is with an Advanced Message Queuing Protocol (AMQP) server such as RabbitMQ.
I followed the procedure recommended in the RabbitMQ site and installed RabbitMQ from the Bintray repository run by JFrog. The main gotchas were
rabbitmqctlprogram used to configure the server is installed in a mysterious location
/usr/lib/rabbitmq/bin/rabbitmqctl, which took me a little while to find; and
For some reason I had an ‘Erlang cookie’ that did not match the server’s—perhaps because of previous attemots at installing RabbitMQ—but this was easily fixed by copying the correct one:
rm ~/.erlang.cookie sudo cat /var/lib/rabbitmq/.erlang.cookie > ~/.erlang.cookie chmod 600 ~/.erlang.cookie
On my development system I have already installed Celery in to the app and created tasks and tested things locally.
I repeated the setup steps from instructions on the Celery site to create an AMQP user and vhost, and set up the environment variable for the site in the usual way:
echo amqp://ooble:correct-horse-battery-staple@localhost:5672/ooble | sudo tee /service/ooble/env/CELERY_BROKER_URL
The next step was pushing the code to the server and trying it out extemporaneously:
. ~/virtualenvs/ooble/bin/activate envdir /service/ooble/env celery -A linotak.celery worker --loglevel=info
Finally, I wrapped up the startup script as a Daemontools service by creating a
directory with a
run script like this:
#! /bin/sh SITE=ooble PACKAGE=linotak SITE_HOME=/home/$SITE exec 2>&1 cd $SITE_HOME/Sites/$PACKAGE exec envdir /service/$SITE/env \ setuidgid $SITE \ $SITE_HOME/virtualenvs/$SITE/bin/celery \ -A $PACKAGE.celery \ worker --loglevel=info
Rather than having its own env dir it uses the env dir of the web server (hence has identical environment variables).
This site deviates from my usual system in that I called the Python package
notes taker) but the site it powers is ooble.uk.