index.md 14KB
title: Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)
url: https://ar.al/2022/11/09/is-the-fediverse-about-to-get-fryed-or-why-every-toot-is-also-a-potential-denial-of-service-attack/
hash_url: a6257c9517
Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)
<p>Stephen is a big fish to fry. (I’m here all week.)</p>
Warning: the fediverse is about to get Fryed.
Stephen Fryed, that is.
Following the recent takeover of Twitter by a proto-fascist billionaire man-baby, people have been fleeing to the fediverse. Among them are folks who, on Twitter, at least, had millions of followers like Greta Thunberg and, more recently, Stephen Fry.
“Well, surely that’s a good thing? It’ll get everyone talking about the fediverse, decentralisation, and maybe even that Small Web thing you keep harping on about all the time, Aral, no?”
Well, yes and no… you see, there is such a thing as too much of a good thing. And, on the fediverse today, that appears to be “engagement when you’re popular.” In fact, it could be deadly (to Mastodon instances, that is).
Read on and I’ll try to explain what I mean by using my own account as an example.
How to kill a Mastodon (hint: by being chatty when you’re popular)
Needless to say, I’m not a celebrity.
And yet, on the fediverse, I find myself in a somewhat unique situation where:
-
I have my own personal Mastodon instance, just for me.
-
I’m followed by quite a number of people. Over 22,000, to be exact.
-
I follow a lot of people and I genuinely enjoy having conversations with them. (I believe this is what the cool kids call “engagement”.)
Unfortunately, the combination of these three factors creates a perfect storm which means that now, every time I post something that gets lots of engagement, I essentially end up carrying out a denial of service attack on myself.
Mastodon: denial-of-service as a service?
Yesterday was my birthday.
So, of course, I posted about it on my Mastodon instance.
It got quite a few replies. And, because it’s only polite, I started replying to everyone with thank-you messages.
Oh, no, you poor, naïve man, you. What were you thinking?!…
I’ll let my friend Hugo Gameiro, who runs masto.host and hosts my instance, explain what happened next:
You just get a lot of engagement and that requires a ton of Sidekiq power to process.
For example, let’s look at your birthday post … besides requiring thousands of Sidekiq jobs to spread your post through all their servers (you have 23K followers, let’s assume 3K different servers), as soon as you create the post 3K Sidekiq jobs are created. At your current plan you have 12 Sidekiq threads, so to process 3K jobs it will take a while because it can only deal with 12 at a time.
Then, for each reply you receive to that post, 3K jobs are created, so your followers can see that reply without leaving their server or looking at your profile. Then you reply to the reply you got, another 3K jobs are created and so on.
If you replied to the 100 replies you got on that post in 10 minutes (and assuming my 3K servers math is right). You created 300K jobs in Sidekiq. That’s why you get those queues.
So what does that mean if you’re not into the technical mumbo-jumbo?
It means I was too chatty while being somewhat popular.
What a traffic jam looks like in Mastodon.
So, what’s the solution?
Well, there’s only one thing you can do when you find yourself in such a pickle: scale up your Mastodon instance. The problem with that? It starts getting expensive.
Prior to the latest Twitter migration, I was paying around €280/year (or a little over €20/month) for my Mastodon instance on a custom plan I had with Hugo from the early days. This week, I upped that to a roughly €50/month plan. And that’s still not enough as my birthday post just showed so Hugo, kindly, has suggested he might have to come up with a custom plan for me.
And yet, the problem is not one that will go away. We can only kick the ball down the road, as it were.
(Unless I piss everyone off with this post, that is.)
Thankfully, by running my own instance, the only person I’m burdening with this additional expense is me. But what if I’d been on a public instance run by someone else instead?
Musk you?
If Elon Musk wanted to destroy mastodon.social, the flagship Mastodon instance, all he’d have to do is join it.
Thank goodness Elon isn’t that smart.
I jest, of course… Eugen would likely ban his account the moment he saw it. But it does illustrate a problem: Elon’s easy to ban. Stephen, not so much. He’s a national treasure for goodness’ sake. One does not simply ban Stephen Fry.
And yet Stephen can similarly (yet unwittingly) cause untold expense to the folks running Mastodon instances just by joining one.
The solution, for Stephen at least, is simple: he should run his own personal instance.
(Or get someone else to run it for him, like I do.)
Running his own instance would also give Stephen one additional benefit: he’d automatically get verified.
After all, if you’re talking to, say, @stephen@social.stephenfry.com, you can be sure it’s really him because you know he owns the domain.
Personal instances to the rescue
My speech at the European Parliament on the problem with Big Tech and the different approaches provided by Mastodon, the fediverse, and Small Web.
Wait, I’m confused… didn’t you say that personal instances were part of the problem?
Yes and no: they are and they shouldn’t be.
If ActivityPub (the protocol) and Mastodon (a server that adheres to that protocol) were designed to incentivise decentralisation, having more instances in the network would not be a problem. In fact, it would be the sign of a healthy, decentralised network.
However, ActivityPub and Mastodon are designed the same way Big Tech/Big Web is: to encourage services that host as many “users” as they can.
This design is both complex (which makes it difficult and expensive to self-host) and works beautifully for Big Tech (where things are centralised and scale vertically and where the goal is to get/own/control/exploit as many users as possible).
In Big Tech, the initial cost of obtaining such scale is subsidised by vast amounts of venture capital (rich people investing in exploitative and extractive new businesses – which Silicon Valley calls Startups™ – in an effort to get even richer) and it leads to the amassing of the centres we know today as the Googles, Facebooks, and Twitters of the world.
However, unlike Big Tech, the stated goal of the fediverse is to decentralise things, not centralise them. Yet how likely is it we can achieve the opposite of Big Tech’s goals while adopting its same fundamental design?
When you adopt the design of a thing, you also inherit the success criteria that led to the evolution of that design. If that success criteria does not align with your own goals, you have a problem on your hands.
What I’m trying to say is:
Do not adopt the success criteria of Big Tech lest you should become Big Tech.
Bigger is not better
Today, we equate the size of mastodon.social (the instance run by Eugen) with how successful Mastodon (the software created by Eugen) is. This is very dangerous. The larger mastodon.social gets, the more it will become like Twitter.
I can almost hear you shout, “But Aral, it’s federated! At least there’s no lock-in to mastodon.social!”
This is true.
You know what else is federated? Email.
Have you ever heard of a little old email instance called Gmail? (Or perhaps the term “embrace, extend, extinguish?”)
Do you know what happens to your email if Google says (rightly or wrongly) that you’re spam? No one sees your email.
You know what happens if mastodon.social blocks your instance? Hundreds of thousands of people (soon, millions?) do not get a choice in whether they see your posts or not.
What happens when your instance of one blocks mastodon.social? Nothing, really.
That’s quite a power imbalance.
Decentralisation begins at decentring yourself
Mastodon is a not-for-profit, and I have no reason to believe that Eugen has anything but the best of intentions.
However, decentralisation begins at decentring yourself.
It’s in the interests of the fediverse that mastodon.social sets a good example by limiting its size voluntarily.
In fact, this should be built right into the software. Mastodon instances should be limited from growing beyond a certain size. Instances that are already too large should have ways of encouraging people to migrate to smaller ones.
As a community we should approach large instances as tumours: how do we break them up so they are no longer a threat to the organism?
If you take this approach to its logical conclusion, you will arrive at the concept of the Small Web; a web where we each own and control our own place (or places).
Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download Small Is Beautiful #23 directly, and watch it with your favourite video player.
Small Is Beautiful (Oct, 2022): What is the Small Web and why do we need it?
I’m not saying that the current fediverse protocols and apps can, will, or even necessarily should evolve into the Small Web. In the here and now, the fediverse is an invaluable stopgap that provides a safer haven than the centralised cesspits of Silicon Valley.
How long the stopgap lasts will depend on how successful we are at resisting centralisation. Protocol and server designs that incentivise vertical scale will not necessarily make this easy. However, there are social pressures we can use to counter their effects.
The last thing you want is a handful of mini Zuckerbergs running the fediverse. Or worse, to find yourself having become one of those mini Zuckerbergs.
I love that the fediverse exists. And I have the utmost respect for the gargantuan effort that’s going into it.
And yet, I am also very concerned that the design decisions that have been made incentivise centralisation, not decentralisation. I implore us to acknowledge this, to mitigate the risks as best we can, to strive to learn from our mistakes, and to do even better going forward.
So to the ActivityPub and Mastodon folks, I say:
Consider me your canary in the coal mine…
«Chirp! Chirp! Chirp!»