larlet-fr-david-cache

A place to cache linked articles (think custom and personal wayback machine)

title: Getting Takahē to run on Piku url: https://taoofmac.com/space/blog/2022/12/21/0900 hash_url: a889fa6d4e

Last night after work I decided to see how easy it would be to run a Takahē ActivityPub instance under Piku, my tiny Python-oriented PaaS.

Self-hosting Mastodon is all the rage, but having to deal with a full-blown installation of Ruby (which is always a pain to install properly, even if you use rbenv), plus the abomination that is Sidekiq and the overall Rube Goldberg-esque architectural approach that is almost mandatory to deal with the complexities of ActivityPub is just something I don’t want to maintain. Ever. Even inside Docker.

Which is why I have been developing my own ActivityPub server using Sanic and a very lightweight asyncio-based approach at handling all the transactional aspects of ActivityPub atop SQLite. And let me tell you, I honestly wish the protocol was less about doing what boils down to P2P webhooks with PEM signatures embedded in requests.

Enter Takahē

logo — Blue birds are cool. Well, at least flightless ones.

But Takahē is now aiming to support client apps as of version 0.6, is built on Django (which I have always loved as a framework), and it saves me from the trouble of building everything from scratch, so… I had to try it out.

More to the point, Django is exactly what Piku was originally designed to run.

Besides running as a WSGI app, Takahē uses an async stator to handle all the background tasks (which is also exactly the pattern I aim for and designed Piku to support), so I just had to see how easy it was to get it running under Piku on very low-end hardware.

The Hardware

I have a 4GB Raspberry Pi 4s set up as an SSD-backed Proxomox server, hosting several different arm64 LXC containers I use for developing stuff. I love it because I can use LXC CPU allocations to throttle things and make sure they run fast enough on very low-end hardware, plus I can just snapshot, mess up and restore entire environments.

So I set up an Ubuntu 22.04 container with 1GB of RAM and access to 2 CPU cores, capped to 50% overall usage–which is roughly the performance of a Raspberry Pi 2 give or take, albeit with a fully 64-bit CPU.

I deployed Piku, set up a CloudFlare tunnel, and then went to town.

Zero Code Changes Required

In short, what I needed to get Takahē up and running under Piku was to:

Clone the repository.
Create a production remote pointing to Piku.
Edit the supplied ENV and Procfile.
Do a git push production main.

It was that simple.

Here’s the configuration I used, annotated. First the ENV file:

# Yes, I went and got it to use SQLite, and it nearly worked 100%
TAKAHE_DATABASE_SERVER=sqlite:////home/piku/takahe.db
# This is what I eventually migrated to (more below)
# TAKAHE_DATABASE_SERVER=postgres://piku:<password>@localhost/takahe
# I actually love Django debugging, and with it on I can see the inner workings
TAKAHE_DEBUG=true
# You know who uses this password, don't you? 
TAKAHE_SECRET_KEY=pepsicola
# No, it's not the one I'm actually using.
# Anyway, this next one breaks a little on Piku, so I need to revise parsing for this case.
TAKAHE_CSRF_TRUSTED_ORIGINS=["http://127.0.0.1:8000", "https://127.0.0.1:8000"]
TAKAHE_USE_PROXY_HEADERS=true
TAKAHE_EMAIL_SERVER=console://console
TAKAHE_MAIN_DOMAIN=insightful.systems
TAKAHE_ENVIRONMENT=development
TAKAHE_MEDIA_BACKEND=local://
TAKAHE_MEDIA_ROOT=/home/piku/media
TAKAHE_MEDIA_URL=https://insightful.systems/media/
TAKAHE_AUTO_ADMIN_EMAIL=<my e-mail>
SERVER_NAME=insightful.systems

# This is all Piku config from here on down
# I need IPv6 off for sanity inside Proxmox
DISABLE_IPV6=true
LC_ALL=en_US.UTF-8
LANG=$LC_ALL
# This ensures nginx only accepts requests from CloudFlare, plus a few extra tweaks
NGINX_CLOUDFLARE_ACL=True
NGINX_SERVER_NAME=$SERVER_NAME

# These are caching settings for my dev branch of Piku
NGINX_CACHE_SIZE=2
NGINX_CACHE_TIME=28800
NGINX_CACHE_DAYS=12
# This has nginx cache these prefixes
NGINX_CACHE_PREFIXES=/media,/proxy 

# This maps static user media directly to an nginx route
NGINX_STATIC_PATHS=/media:/home/piku/media,/static:static,/robots.txt:static/robots.txt
PORT=8000
# You want to set these, trust me. I should make them defaults in Piku.
PYTHONIOENCODING=UTF_8:replace
PYTHONUNBUFFERED=1
TZ=Europe/Lisbon
# This tells uWSGI to shut down idle HTTP workers
# Saves RAM, but startup from idle is a bit more expensive CPU-wise
UWSGI_IDLE=60
# We need to run at least 2 uWSGI workers for Takahe
UWSGI_PROCESSES=2
# Each worker will have this many threads 
# (even though I'm only giving this 2 cores)
# to match the original gunicorn config.
UWSGI_THREADS=4

…and only very minor changes to the Procfile:

wsgi: takahe.wsgi:application
worker: python manage.py runstator
release: python manage.py migrate

In essence, I removed gunicorn (which I could use anyway) to let uWSGI handle HTTP requests and scale down to zero (saving RAM). And yes, Piku also supports release activities, thanks to Chris McCormick.

And that was it. Zero code changes. None. Nada. And I can use exactly the same setup on any VPS on the planet, thanks to Piku.

After a little faffing about with the media storage settings (which I got wrong the first time around, since Takahē also uses /static for its own assets), I had a fully working ActivityPub instance, and, well… John Mastodon just happened to sign up:

John Mastodon, with fail whale by Matthew Inman — Every ActivityPub server ought to make this their demo account.

Teething Issues

Takahē nearly works with SQLite, but sadly it relies on JSON_CONTAINS, which is an unsupported feature in SQLite (but one which PostgreSQL excels at).

The upshot of this was that the stator worker was very sad and bombed out when trying to handle hashtags–but all critical stuff worked, so there might well be a workaroud.

But I took some time after breakfast to migrate the database, and since my Django skills are rusty, here are my notes:

# Open a shell to Piku
ssh -t [email protected] run takahe bash 
sudo apt install postgresql
python manage.py dumpdata > /tmp/dump.json
sudo su - postgres
psql

-- Set up the database
create user piku;
create database takahe;
alter role piku with password '<mysecret>';
grant all privileges on database takahe to piku;
alter database takahe owner to piku;

# Reset all the migrations, just in case
find . -path “*/migrations/*.py” -not -name “__init__.py” -delete
find . -path “*/migrations/*.pyc” -delete
# Reapply them
python manage.py makemigrations
python manage.py migrate
# Wipe all default entities
python manage.py shell
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
# Load everything back
python manage.py loaddata /tmp/dump.json

Performance

Overall, I’m quite impressed with the whole thing. Even with such measly resources and Linux’s tendency to take up RAM with buffers, Takahē under Piku is taking up around 100MB per active worker (2 web handlers, plus the stator worker), plus less than 50MB for PostgreSQL and nginx together.

So I’m seeing less than 512MB of RAM in actual use, and a steady <10% CPU load inside the container as the stator keeps picking up inbound updates, handling them (including any outbound requests) and doing all the messy housekeeping associated with ActivityPub:

My Pimox console, showing CPU and RAM — These are the stats a few hours later in the day, after publishing this post.

But here’s the kicker: Since this is being capped inside LXC, that is actually around 5% overall CPU load on the hardware–which should translate to something like 2% of CPU usage on any kind of “real” hardware.

With only one active user for now (but following a few accounts already), this is very, very promising.

I have no real plans to leave mastodon.social for my own domain, but using Takahē to host a small group of people (or a company) with nothing more than a tiny VPS seems entirely feasible, and is certainly in my future.

Next Steps

Right now, I’m going to try to contribute by testing various iOS clients (I will be using the Takahē public test instance as well) and do some minor tweaks to my install, namely:

Setting up nginx caching. Cloudflare is already caching one third of the data, but I want to bulk up this setup so that I can eventually move it to Azure, and I’ve been meaning to add that to Piku anyway.
Fine-tuning the stator to see how it scales up or down (I might want to try to scale it down further).
Trying gunicorn to see if it makes any difference in overall RAM and CPU.
Seeing if I can get it to work on Azure Functions (that is sure to be fun, although the current SDK failed to install on my M1 and I haven’t tried since).
Look at how media assets are handled and see if I can add a patch to support Azure Storage via my own aioazstorage library.
Deploy on my k3s cluster, to get a feel for how much it would cost to run on spot instances.

There goes my holiday break, I guess…

Update: A Few Days Later

I’ve since sorted out nginx caching in Piku (and will soon be merging it to main), which makes things significantly snappier. I’ve also filed #287 to improve caching via Cloudflare and #288 to have nginx immediately cache assets (which works for me, at least).

Before that, I had some fun tuning stator pauses and filed #232, which resulted in a tweak that lowered idle CPU consumption to a pretty amazing 3% in my test instance.

With the caching tweaks, gunicorn doesn’t have any real advantage against uWSGI workers, although I suspect that may be different in higher-load instances.

I’ve also tossed the source tree into an Azure Function and got it to “work”, but not fully. Right now I’m not sure that is worth pursuing given I still need an external database, but I’m really curious to try again in a few months’ time.

index.md 20KB Raw Blame History