A place to cache linked articles (think custom and personal wayback machine)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

пре 1 година
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146
  1. title: Getting Takahē to run on Piku
  2. url: https://taoofmac.com/space/blog/2022/12/21/0900
  3. hash_url: a889fa6d4e07bdc390d44461ed6dce21
  4. <p class="lead">Last night after work I decided to see how easy it would be to run a <a href="https://jointakahe.org" rel="external">Takahē</a> <a href="/space/protocols/activitypub" rel="next">ActivityPub</a> instance under <a href="https://github.com/piku/piku" rel="external">Piku</a>, my tiny <a href="/space/dev/python" rel="next">Python</a>-oriented PaaS.</p>
  5. <p>Self-hosting <a href="https://joinmastodon.org" rel="external">Mastodon</a> is all the rage, but having to deal with a full-blown installation of <a href="/space/dev/ruby" rel="next">Ruby</a> (which is always a pain to install properly, even if you use <code>rbenv</code>), plus the abomination that is <a href="https://sidekiq.org" rel="external">Sidekiq</a> and the overall Rube Goldberg-esque architectural approach that is almost mandatory to deal with the complexities of <a href="/space/protocols/activitypub" rel="next">ActivityPub</a> is just something I don’t want to maintain. Ever. Even inside Docker.</p>
  6. <p>Which is why I have been developing my own <a href="/space/protocols/activitypub" rel="next">ActivityPub</a> server using <a href="https://sanic.dev" rel="external">Sanic</a> and a very lightweight <code>asyncio</code>-based approach at handling all the transactional aspects of <a href="/space/protocols/activitypub" rel="next">ActivityPub</a> atop <a href="/space/db/sqlite" rel="next">SQLite</a>. And let me tell you, I honestly wish the protocol was less about doing what boils down to P2P webhooks with PEM signatures embedded in requests.</p>
  7. <a id="anchor-enter-takahe" class="anchor" href="/space/blog/2022/12/21/0900#enter-takahe" rel="anchor"><h2 id="enter-takahe">Enter Takahē</h2></a><figure>
  8. <img src="/media/blog/2022/12/21/0900/bqAzQDKitQgUZ3Uu9MByVxvt8r4=/takahe.png" title="logo">
  9. <figcaption>Blue birds are cool. Well, at least flightless ones.</figcaption>
  10. </figure>
  11. <p>But <a href="https://jointakahe.org" rel="external">Takahē</a> is now aiming to support client apps as of version <code>0.6</code>, is built on <a href="dev/python/django" rel="nofollow">Django</a> (which I have always loved as a framework), and it saves me from the trouble of building <em>everything</em> from scratch, so… I had to try it out.</p>
  12. <p>More to the point, <a href="dev/python/django" rel="nofollow">Django</a> is <em>exactly</em> what <a href="https://github.com/piku/piku" rel="external">Piku</a> was originally designed to run.</p>
  13. <p>Besides running as a <code>WSGI</code> app, <a href="https://jointakahe.org" rel="external">Takahē</a> uses an async stator to handle all the background tasks (which is also <em>exactly</em> the pattern I aim for and designed <a href="https://github.com/piku/piku" rel="external">Piku</a> to support), so I just had to see how easy it was to get it running under <a href="https://github.com/piku/piku" rel="external">Piku</a> on very low-end hardware.</p>
  14. <a id="anchor-the-hardware" class="anchor" href="/space/blog/2022/12/21/0900#the-hardware" rel="anchor"><h2 id="the-hardware">The Hardware</h2></a><p>I have a 4GB <a href="/space/hw/raspberry_pi" rel="next">Raspberry Pi</a> 4s set up as an SSD-backed <a href="https://github.com/pimox/pimox7" rel="external">Proxomox</a> server, hosting several different <code>arm64</code> LXC containers I use for developing stuff. I love it because I can use LXC CPU allocations to throttle things and make sure they run fast enough on very low-end hardware, plus I can just snapshot, mess up and restore entire environments.</p>
  15. <p>So I set up an Ubuntu 22.04 container with 1GB of RAM and access to 2 CPU cores, capped to 50% overall usage–which is <em>roughly</em> the performance of a <a href="/space/hw/raspberry_pi" rel="next">Raspberry Pi</a> 2 give or take, albeit with a fully 64-bit CPU.</p>
  16. <p>I deployed <a href="https://github.com/piku/piku" rel="external">Piku</a>, set up a CloudFlare tunnel, and then went to town.</p>
  17. <a id="anchor-zero-code-changes-required" class="anchor" href="/space/blog/2022/12/21/0900#zero-code-changes-required" rel="anchor"><h2 id="zero-code-changes-required">Zero Code Changes Required</h2></a><p>In short, what I needed to get <a href="https://jointakahe.org" rel="external">Takahē</a> up and running under <a href="https://github.com/piku/piku" rel="external">Piku</a> was to:</p>
  18. <ol>
  19. <li>Clone the repository.</li>
  20. <li>Create a <code>production</code> remote pointing to <a href="https://github.com/piku/piku" rel="external">Piku</a>.</li>
  21. <li>Edit the supplied <code>ENV</code> and <code>Procfile</code>.</li>
  22. <li>Do a <code>git push production main</code>.</li>
  23. </ol>
  24. <p>It was <em>that simple</em>.</p>
  25. <p>Here’s the configuration I used, annotated. First the <code>ENV</code> file:</p>
  26. <div class="highlight"><pre><span></span><code><span class="c1"># Yes, I went and got it to use SQLite, and it nearly worked 100%</span><span class="w"></span>
  27. <span class="na">TAKAHE_DATABASE_SERVER</span><span class="o">=</span><span class="s">sqlite:////home/piku/takahe.db</span><span class="w"></span>
  28. <span class="c1"># This is what I eventually migrated to (more below)</span><span class="w"></span>
  29. <span class="c1"># TAKAHE_DATABASE_SERVER=postgres://piku:&lt;password&gt;@localhost/takahe</span><span class="w"></span>
  30. <span class="c1"># I actually love Django debugging, and with it on I can see the inner workings</span><span class="w"></span>
  31. <span class="na">TAKAHE_DEBUG</span><span class="o">=</span><span class="s">true</span><span class="w"></span>
  32. <span class="c1"># You know who uses this password, don't you? </span><span class="w"></span>
  33. <span class="na">TAKAHE_SECRET_KEY</span><span class="o">=</span><span class="s">pepsicola</span><span class="w"></span>
  34. <span class="c1"># No, it's not the one I'm actually using.</span><span class="w"></span>
  35. <span class="c1"># Anyway, this next one breaks a little on Piku, so I need to revise parsing for this case.</span><span class="w"></span>
  36. <span class="na">TAKAHE_CSRF_TRUSTED_ORIGINS</span><span class="o">=</span><span class="s">["http://127.0.0.1:8000", "https://127.0.0.1:8000"]</span><span class="w"></span>
  37. <span class="na">TAKAHE_USE_PROXY_HEADERS</span><span class="o">=</span><span class="s">true</span><span class="w"></span>
  38. <span class="na">TAKAHE_EMAIL_SERVER</span><span class="o">=</span><span class="s">console://console</span><span class="w"></span>
  39. <span class="na">TAKAHE_MAIN_DOMAIN</span><span class="o">=</span><span class="s">insightful.systems</span><span class="w"></span>
  40. <span class="na">TAKAHE_ENVIRONMENT</span><span class="o">=</span><span class="s">development</span><span class="w"></span>
  41. <span class="na">TAKAHE_MEDIA_BACKEND</span><span class="o">=</span><span class="s">local://</span><span class="w"></span>
  42. <span class="na">TAKAHE_MEDIA_ROOT</span><span class="o">=</span><span class="s">/home/piku/media</span><span class="w"></span>
  43. <span class="na">TAKAHE_MEDIA_URL</span><span class="o">=</span><span class="s">https://insightful.systems/media/</span><span class="w"></span>
  44. <span class="na">TAKAHE_AUTO_ADMIN_EMAIL</span><span class="o">=</span><span class="s">&lt;my e-mail&gt;</span><span class="w"></span>
  45. <span class="na">SERVER_NAME</span><span class="o">=</span><span class="s">insightful.systems</span><span class="w"></span>
  46. <span class="c1"># This is all Piku config from here on down</span><span class="w"></span>
  47. <span class="c1"># I need IPv6 off for sanity inside Proxmox</span><span class="w"></span>
  48. <span class="na">DISABLE_IPV6</span><span class="o">=</span><span class="s">true</span><span class="w"></span>
  49. <span class="na">LC_ALL</span><span class="o">=</span><span class="s">en_US.UTF-8</span><span class="w"></span>
  50. <span class="na">LANG</span><span class="o">=</span><span class="s">$LC_ALL</span><span class="w"></span>
  51. <span class="c1"># This ensures nginx only accepts requests from CloudFlare, plus a few extra tweaks</span><span class="w"></span>
  52. <span class="na">NGINX_CLOUDFLARE_ACL</span><span class="o">=</span><span class="s">True</span><span class="w"></span>
  53. <span class="na">NGINX_SERVER_NAME</span><span class="o">=</span><span class="s">$SERVER_NAME</span><span class="w"></span>
  54. <span class="c1"># These are caching settings for my dev branch of Piku</span><span class="w"></span>
  55. <span class="na">NGINX_CACHE_SIZE</span><span class="o">=</span><span class="s">2</span><span class="w"></span>
  56. <span class="na">NGINX_CACHE_TIME</span><span class="o">=</span><span class="s">28800</span><span class="w"></span>
  57. <span class="na">NGINX_CACHE_DAYS</span><span class="o">=</span><span class="s">12</span><span class="w"></span>
  58. <span class="c1"># This has nginx cache these prefixes</span><span class="w"></span>
  59. <span class="na">NGINX_CACHE_PREFIXES</span><span class="o">=</span><span class="s">/media,/proxy </span><span class="w"></span>
  60. <span class="c1"># This maps static user media directly to an nginx route</span><span class="w"></span>
  61. <span class="na">NGINX_STATIC_PATHS</span><span class="o">=</span><span class="s">/media:/home/piku/media,/static:static,/robots.txt:static/robots.txt</span><span class="w"></span>
  62. <span class="na">PORT</span><span class="o">=</span><span class="s">8000</span><span class="w"></span>
  63. <span class="c1"># You want to set these, trust me. I should make them defaults in Piku.</span><span class="w"></span>
  64. <span class="na">PYTHONIOENCODING</span><span class="o">=</span><span class="s">UTF_8:replace</span><span class="w"></span>
  65. <span class="na">PYTHONUNBUFFERED</span><span class="o">=</span><span class="s">1</span><span class="w"></span>
  66. <span class="na">TZ</span><span class="o">=</span><span class="s">Europe/Lisbon</span><span class="w"></span>
  67. <span class="c1"># This tells uWSGI to shut down idle HTTP workers</span><span class="w"></span>
  68. <span class="c1"># Saves RAM, but startup from idle is a bit more expensive CPU-wise</span><span class="w"></span>
  69. <span class="na">UWSGI_IDLE</span><span class="o">=</span><span class="s">60</span><span class="w"></span>
  70. <span class="c1"># We need to run at least 2 uWSGI workers for Takahe</span><span class="w"></span>
  71. <span class="na">UWSGI_PROCESSES</span><span class="o">=</span><span class="s">2</span><span class="w"></span>
  72. <span class="c1"># Each worker will have this many threads </span><span class="w"></span>
  73. <span class="c1"># (even though I'm only giving this 2 cores)</span><span class="w"></span>
  74. <span class="c1"># to match the original gunicorn config.</span><span class="w"></span>
  75. <span class="na">UWSGI_THREADS</span><span class="o">=</span><span class="s">4</span><span class="w"></span>
  76. </code></pre></div>
  77. <p>…and only very minor changes to the <code>Procfile</code>:</p>
  78. <div class="highlight"><pre><span></span><code><span class="nl">wsgi</span><span class="p">:</span><span class="w"> </span>takahe.wsgi<span class="err">:</span>application<span class="w"></span>
  79. <span class="nl">worker</span><span class="p">:</span><span class="w"> </span>python<span class="w"> </span>manage.py<span class="w"> </span>runstator<span class="w"></span>
  80. <span class="nl">release</span><span class="p">:</span><span class="w"> </span>python<span class="w"> </span>manage.py<span class="w"> </span>migrate<span class="w"></span>
  81. </code></pre></div>
  82. <p>In essence, I removed <code>gunicorn</code> (which I could use anyway) to let <code>uWSGI</code> handle HTTP requests and scale down to <em>zero</em> (saving RAM). And yes, <a href="https://github.com/piku/piku" rel="external">Piku</a> also supports <code>release</code> activities, thanks to <a href="https://github.com/chr15m" rel="external">Chris McCormick</a>.</p>
  83. <p>And that was it. <em>Zero</em> code changes. None. <em>Nada</em>. And I can use exactly the same setup on <em>any</em> VPS on the planet, thanks to <a href="https://github.com/piku/piku" rel="external">Piku</a>.</p>
  84. <p>After a little faffing about with the media storage settings (which I got wrong the first time around, since <a href="https://jointakahe.org" rel="external">Takahē</a> also uses <code>/static</code> for its own assets), I had a fully working <a href="/space/protocols/activitypub" rel="next">ActivityPub</a> instance, and, well… John Mastodon just happened to sign up:</p>
  85. <figure>
  86. <img src="/media/blog/2022/12/21/0900/4XIkhj5AApzxqSDDOgaKutrv51k=/johnmastodon.jpg" title="John Mastodon, with fail whale by Matthew Inman" rel="hero">
  87. <figcaption>Every ActivityPub server ought to make this their demo account.</figcaption>
  88. </figure>
  89. <a id="anchor-teething-issues" class="anchor" href="/space/blog/2022/12/21/0900#teething-issues" rel="anchor"><h2 id="teething-issues">Teething Issues</h2></a><p><a href="https://jointakahe.org" rel="external">Takahē</a> <em>nearly</em> works with <a href="/space/db/sqlite" rel="next">SQLite</a>, but sadly it relies on <code>JSON_CONTAINS</code>, which is an unsupported feature in <a href="/space/db/sqlite" rel="next">SQLite</a> (but one which <a href="/space/db/postgresql" rel="next">PostgreSQL</a> excels at).</p>
  90. <p>The upshot of this was that the stator <code>worker</code> was very sad and bombed out when trying to handle hashtags–but all critical stuff worked, so there might well be a workaroud.</p>
  91. <p>But I took some time after breakfast to migrate the database, and since my <a href="dev/python/django" rel="nofollow">Django</a> skills are rusty, here are my notes:</p>
  92. <div class="highlight"><pre><span></span><code><span class="c1"># Open a shell to Piku</span>
  93. ssh -t <a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="daaab3b1af9abbb9aeb3acb3aea3aaafb8f4b6bbb4">[email protected]</a> run takahe bash
  94. sudo apt install postgresql
  95. python manage.py dumpdata &gt; /tmp/dump.json
  96. sudo su - postgres
  97. psql
  98. </code></pre></div>
  99. <div class="highlight"><pre><span></span><code><span class="c1">-- Set up the database</span>
  100. <span class="k">create</span><span class="w"> </span><span class="k">user</span><span class="w"> </span><span class="n">piku</span><span class="p">;</span><span class="w"></span>
  101. <span class="k">create</span><span class="w"> </span><span class="k">database</span><span class="w"> </span><span class="n">takahe</span><span class="p">;</span><span class="w"></span>
  102. <span class="k">alter</span><span class="w"> </span><span class="k">role</span><span class="w"> </span><span class="n">piku</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="s1">'&lt;mysecret&gt;'</span><span class="p">;</span><span class="w"></span>
  103. <span class="k">grant</span><span class="w"> </span><span class="k">all</span><span class="w"> </span><span class="k">privileges</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="k">database</span><span class="w"> </span><span class="n">takahe</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">piku</span><span class="p">;</span><span class="w"></span>
  104. <span class="k">alter</span><span class="w"> </span><span class="k">database</span><span class="w"> </span><span class="n">takahe</span><span class="w"> </span><span class="k">owner</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">piku</span><span class="p">;</span><span class="w"></span>
  105. </code></pre></div>
  106. <div class="highlight"><pre><span></span><code><span class="c1"># Reset all the migrations, just in case</span>
  107. find . -path “*/migrations/*.py” -not -name “__init__.py” -delete
  108. find . -path “*/migrations/*.pyc” -delete
  109. <span class="c1"># Reapply them</span>
  110. python manage.py makemigrations
  111. python manage.py migrate
  112. <span class="c1"># Wipe all default entities</span>
  113. python manage.py shell
  114. from django.contrib.contenttypes.models import ContentType
  115. ContentType.objects.all<span class="o">()</span>.delete<span class="o">()</span>
  116. <span class="c1"># Load everything back</span>
  117. python manage.py loaddata /tmp/dump.json
  118. </code></pre></div>
  119. <a id="anchor-performance" class="anchor" href="/space/blog/2022/12/21/0900#performance" rel="anchor"><h2 id="performance">Performance</h2></a><p>Overall, I’m quite impressed with the whole thing. Even with such measly resources and Linux’s tendency to take up RAM with buffers, <a href="https://jointakahe.org" rel="external">Takahē</a> under <a href="https://github.com/piku/piku" rel="external">Piku</a> is taking up around 100MB per active worker (2 web handlers, plus the stator worker), plus less than 50MB for <a href="/space/db/postgresql" rel="next">PostgreSQL</a> and <code>nginx</code> <em>together</em>.</p>
  120. <p>So I’m seeing <em>less than 512MB of RAM</em> in actual use, and a steady &lt;10% CPU load inside the container as the stator keeps picking up inbound updates, handling them (including any outbound requests) and doing all the messy housekeeping associated with <a href="/space/protocols/activitypub" rel="next">ActivityPub</a>:</p>
  121. <figure>
  122. <img src="/media/blog/2022/12/21/0900/-ppthry9RSX9xU_aab-SgYe8sok=/pimox.jpg" title="My Pimox console, showing CPU and RAM" rel="hero">
  123. <figcaption>These are the stats a few hours later in the day, after publishing this post.</figcaption>
  124. </figure>
  125. <p>But here’s the kicker: Since this is being capped inside LXC, that is actually around 5% <em>overall</em> CPU load on the hardware–which should translate to something like 2% of CPU usage on any kind of “real” hardware. </p>
  126. <p>With only one active user for now (but following a few accounts already), this is very, very promising.</p>
  127. <p>I have no real plans to leave <code>mastodon.social</code> for my own domain, but using <a href="https://jointakahe.org" rel="external">Takahē</a> to host a small group of people (or a company) with nothing more than a tiny VPS seems entirely feasible, and is certainly in my future.</p>
  128. <a id="anchor-next-steps" class="anchor" href="/space/blog/2022/12/21/0900#next-steps" rel="anchor"><h2 id="next-steps">Next Steps</h2></a><p>Right now, I’m going to try to contribute by testing various iOS clients (I will be using the <a href="https://jointakahe.org" rel="external">Takahē</a> public test instance as well) and do some minor tweaks to my install, namely:</p>
  129. <ul>
  130. <li>Setting up <code>nginx</code> caching. Cloudflare is already caching one third of the data, but I want to bulk up this setup so that I can eventually move it to Azure, and I’ve been meaning to add that to <a href="https://github.com/piku/piku" rel="external">Piku</a> anyway.</li>
  131. <li>Fine-tuning the stator to see how it scales up or down (I might want to try to scale it down further).</li>
  132. <li>Trying <code>gunicorn</code> to see if it makes any difference in overall RAM and CPU.</li>
  133. <li>Seeing if I can get it to work on Azure Functions (that is sure to be fun, although the current SDK failed to install on my M1 and I haven’t tried since).</li>
  134. <li>Look at how media assets are handled and see if I can add a patch to support Azure Storage via <a href="https://github.com/rcarmo/aioazstorage" rel="external">my own <code>aioazstorage</code> library</a>.</li>
  135. <li>Deploy on my <a href="https://github.com/rcarmo/azure-k3s-cluster" rel="external">k3s cluster</a>, to get a feel for how much it would cost to run on spot instances.</li>
  136. </ul>
  137. <p>There goes my holiday break, I guess…</p>
  138. <h3 id="update-a-few-days-later"><strong>Update:</strong> A Few Days Later</h3>
  139. <p>I’ve since sorted out <code>nginx</code> caching in <a href="https://github.com/piku/piku" rel="external">Piku</a> (and will soon be merging it to <code>main</code>), which makes things significantly snappier. I’ve also filed <a href="https://github.com/jointakahe/takahe/issues/287" rel="external">#287</a> to improve caching via Cloudflare and <a href="https://github.com/jointakahe/takahe/pull/288" rel="external">#288</a> to have <code>nginx</code> immediately cache assets (which works for me, at least).</p>
  140. <p>Before that, I had some fun tuning stator pauses and filed <a href="https://github.com/jointakahe/takahe/issues/232" rel="external">#232</a>, which resulted in a tweak that lowered idle CPU consumption to a pretty amazing 3% in my test instance.</p>
  141. <p>With the caching tweaks, <code>gunicorn</code> doesn’t have any real advantage against <code>uWSGI</code> workers, although I suspect that may be different in higher-load instances.</p>
  142. <p>I’ve also tossed the source tree into an Azure Function and got it to “work”, but not fully. Right now I’m not sure that is worth pursuing given I still need an external database, but I’m really curious to try again in a few months’ time.</p>