GNU Social Queue Setup

Wed 22 February 2017

How to replace the default MySQL queue with one that sucks less.

By default, GNU Social will use MySQL as a work queue for handling various tasks. This works for a site with few users, but begins to drag down performance when your traffic increases. Here's how to get it working fast again.

This article will assume the OS is Ubuntu 16.04, but should be easy enough to modify for your situation.

1. Install RabbitMQ

RabbitMQ is a powerful and lightweight queue application that is easy to set up. Other queues exist and may be usable, but this one worked with few modifications where others just silently failed to work (for example, ArtemisMQ.) This is most likely due to GNU Social rather than the queues, but that's irrelevant for our purposes.

You can download it for your platform here: https://www.rabbitmq.com/download.html

Install using your package manager if possible so that it properly configures init script/systemd service file, so that you don't have to start it manually:

wget https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
gpg --import rabbitmq-release-signing-key.asc
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server_3.6.6-1_all.deb
https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server_3.6.6-1_all.deb.asc
gpg --verify rabbitmq-server_3.6.6-1_all.deb.asc rabbitmq-server_3.6.6-1_all.deb
dpkg -i rabbitmq-server_3.6.6-1_all.deb #ok if this errors
apt-get -f install #get all dependencies

Assuming it is now installed, we next need to enable STOMP plugin. STOMP is a queue API that GNU Social expects and RabbitMQ has a plugin for. (GS only supports STOMP 1.0, in case you try to use a different queue this may cause you problems.)

So, here's how we activate STOMP:

rabbitmq-plugins enable rabbitmq_stomp

STOMP is now enabled and listening on port 61613.

WARNING: at this point our STOMP queue is set up, but it is listening on ALL interfaces, and with a default user/pass of guest / guest

Firewall this or anyone can access your queue. Instructions here: https://www.rabbitmq.com/stomp.html detail how to restrict this to localhost and change the password, but for my purposes it wasn't necessary.

2. Slight Edit to STOMP Libraries

A change has to be made to the included STOMP libraries, or they will fail most of the time. The messages that GNU Social feeds to STOMP contains RabbitMQ control characters. So we'll munge the data into compliance by BASE64-encoding it. Because it was easy and it was the first thing I thought of.

Change thus, ${GS_ROOT}/lib/stompqueuemanager.php on line 154:

$msg = serialize($envelope);

to:

$msg = base64_encode(serialize($envelope));

and on line 482:

$message = unserialize($frame->body);

to:

$message = unserialize(base64_decode($frame->body));

Change on line 493:

if ($this->isDeadletter($frame, $message)) {
$this->stats('deadletter', $queue);
            return false;
}

to:

if ($this->isDeadLetter($frame, $message)) {
$this->stats('deadletter', $queue);
            return false;
}

(we are just fixing a case bug.)

Finally, we must add an additional header to STOMP connection in ${GS_ROOT}/extlib/Stomp.php on line 326:

$headers['activemq.prefetchSize'] = $this->prefetchSize;

add:

$headers['activemq.prefetchSize'] = $this->prefetchSize;
$headers['prefetch-count'] = '1';

The existing header is only for activemq. The additional header is only for RabbitMQ, and is hardcoded to a value of 1 because this is the only value that works.

These modifications are an absolutely necessary step. And yes, the changes will be submitted upstream, eventually.

3. Configure GNU Social

Now we will edit our config.php file, which is probably in your GS installation root. There are many different ways your system could be configured, so try to use common sense to not configure a mess.

$config['queue']['daemon'] = true;
$config['queue']['enabled'] = true;
$config['queue']['subsystem'] = 'stomp'; //if this was 'db' then you need to change it.
$config['queue']['stomp_server'] = 'tcp://127.0.0.1:61613';
$config['queue']['stomp_username'] = 'guest';
$config['queue']['stomp_password'] = 'guest';
$config['queue']['control_channel'] = 'gs-control';
$config['queue']['queue_basename'] = 'gs-queue';
//$config['queue']['stomp_persistent'] = true;
//$config['queue']['stomp_transactions'] = true;
//$config['queue']['stomp_acks'] = true;

By default, GNU Social puts slashes in control_channel and queue_basename , which is valid for STOMP queues, but not for RabbitMQ; so we rename them to something plainer or everything will fail.

NOTE: No persistence is happening, so if the queue dies or is shut off, the queue content will be lost. There are a set of commented-out config items in the code above that should make the queue safer, I have not tested them.

4. Start Queue Daemons

I am not going to go into how to set up Queue Daemons with SystemD or Monit or anything else. Just telling you now you must start them.

su - www-data bash -c "script/startdaemons.sh"

They like to die for no reason, so I do recommend monit to watch them and restart them when they fail.

5. Enjoy

Check out your logs and watch the queuedaemons processing your notices. Enjoy your much, MUCH faster GNU Social server.

blogroll

social