Simon Willison's Weblog: fediverse

How do I opt into full text search on Mastodon?

2024-06-04T06:14:37+00:00

How do I opt into full text search on Mastodon?

I missed this new Mastodon feature when it was released in 4.2.0 last September: you can now opt-in to a new setting which causes all of your future posts to be marked as allowed to be included in the Elasticsearch index provided by Mastodon instances that enable search.

It only applies to future posts because it works by adding an "indexable" flag to those posts, which can then be obeyed by other Mastodon instances that the post is syndicated to.

You can turn it on for your own account from the /settings/privacy page on your local instance.

The release notes for 4.2.0 also mention new search operators:

from:me, before:2022-11-01, after:2022-11-01, during:2022-11-01, language:fr, has:poll, or in:library (for searching only in posts you have written or interacted with)

Via @robinwhittleton

Tags: search, mastodon, fediverse

Threads has entered the fediverse

2024-03-22T20:15:20+00:00

Threads has entered the fediverse

Threads users with public profiles in certain countries can now turn on a setting which makes their posts available in the fediverse—so users of ActivityPub systems such as Mastodon can follow their accounts to subscribe to their posts.

It’s only a partial integration at the moment: Threads users can’t themselves follow accounts from other providers yet, and their notifications will show them likes but not boosts or replies: “For now, people who want to see replies on their posts on other fediverse servers will have to visit those servers directly.”

Depending on how you count, Mastodon has around 9m user accounts of which 1m are active. Threads claims more than 130m active monthly users. The Threads team are developing these features cautiously which is reassuring to see—a clumsy or thoughtless integration could cause all sorts of damage just from the sheer scale of their service.

Via Gavin Anderegg

Tags: facebook, threads, mastodon, activitypub, fediverse

Phanpy

2024-03-16T01:34:04+00:00

Phanpy

Phanpy is "a minimalistic opinionated Mastodon web client" by Chee Aun.

I think that description undersells it. It's beautifully crafted and designed and has a ton of innovative ideas - they way it displays threads and replies, the "Catch-up" beta feature, it's all a really thoughtful and fresh perspective on how Mastodon can work.

I love that all Mastodon servers (including my own dedicated instance) offer a CORS-enabled JSON API which directly supports building these kinds of alternative clients.

Building a full-featured client like this one is a huge amount of work, but building a much simpler client that just displays the user's incoming timeline could be a pretty great educational project for people who are looking to deepen their front-end development skills.

Tags: javascript, mastodon, fediverse, cors

Where is all of the fediverse?

2024-01-12T18:54:15+00:00

Where is all of the fediverse?

Neat piece of independent research by Ben Cox, who used the /api/v1/instance/peers Mastodon API endpoint to get a list of “peers” (instances his instance knows about), then used their DNS records to figure out which hosting provider they were running on.

Next Ben combined that with active users from the /nodeinfo/2.0 API on each instance to figure out the number of users on each of those major hosting providers.

Cloudflare and Fastly were heavily represented, but it turns out you can unveil the underlying IP for most instances by triggering an HTTP Signature exchange with them and logging the result.

Ben’s conclusion: Hertzner and OVH are responsible for hosting a sizable portion of the fediverse as it exists today.

Via lobste.rs

Tags: dns, hosting, mastodon, fediverse

Quoting Mike McCue

2023-12-18T18:45:49+00:00

Basically, we’re in the process of replacing our whole social back-end with ActivityPub. I think Flipboard is going to be the first mainstream consumer service that existed in a walled garden that switches over to ActivityPub.

— Mike McCue, CEO of Flipboard

Tags: mastodon, activitypub, fediverse

Meta/Threads Interoperating in the Fediverse Data Dialogue Meeting yesterday

2023-12-12T01:05:32+00:00

Meta/Threads Interoperating in the Fediverse Data Dialogue Meeting yesterday

Johannes Ernst reports from a recent meeting hosted by Meta aimed at bringing together staff from Meta’s Threads social media platform with representatives from the Fediverse.

Meta have previously announced an intention for Threads to join the Fediverse. It sounds like they’re being extremely thoughtful about how to go about this.

Two points that stood out for me:

“Rolling out a large node – like Threads will be – in a complex, distributed system that’s as decentralized and heterogeneous as the Fediverse is not something anybody really has done before.”

And:

“When we think of privacy risks when Meta connects to the Fediverse, we usually think of what happens to data that moves from today’s Fediverse into Meta. I didn’t realize the opposite is also quite a challenge (personal data posted to Threads, making its way into the Fediverse) for an organization as heavily monitored by regulators around the world as is Meta.”

Tags: facebook, social-media, mastodon, fediverse, meta

Guppe Groups

2023-01-26T19:45:51+00:00

Guppe Groups

This is a really neat mechanism for helping build topic-oriented communities on Mastodon: follow @any-group-name@a.gup.pe to join (or create) a group, then that account will re-broadcast any messages from people in that group who mention the group in their message.

I found it via the histodons group. I was pondering how something like this might work this recently, so it’s great to see someone has built it already.

Tags: mastodon, fediverse

Jortage Communal Cloud

2023-01-24T23:23:04+00:00

Jortage Communal Cloud

An interesting pattern that’s emerging in the Mastodon / Fediverse community: Jortage is “a communal project providing object storage and hosting”. Each Mastodon server needs to host copies of files—not just for their users, but files that have been imported into the instance because they were posted by other people followed by that instance’s users. Jortage lets multiple instances share the same objects, reducing costs and making things more efficient. I like the idea that multiple projects like this can co-exist, improving the efficiency of the overall network without introducing single centralized services.

Tags: mastodon, fediverse

Mastodon is just blogs

2022-11-08T15:48:17+00:00

And that's great. It's also the return of Google Reader!

Mastodon is really confusing for newcomers. There are memes about it.

If you're an internet user of a certain age, you may find an analogy that's been working for me really useful:

Mastodon is just blogs.

Every Mastodon account is a little blog. Mine is at https://fedi.simonwillison.net/@simon.

You can post text and images to it. You can link to things. It's a blog.

You can also subscribe to other people's blogs - either by "following" them (a subscribe in disguise) or - fun trick - you can add .rss to their page and subscribe in a regular news reader (here's my feed).

A Mastodon server (often called an instance) is just a shared blog host. Kind of like putting your personal blog in a folder on a domain on shared hosting with some of your friends.

Want to go it alone? You can do that: run your own dedicated Mastodon instance on your own domain (or pay someone to do that for you - I'm using masto.host).

Feeling really nerdy? You can build your own instance from scratch, by implementing the ActivityPub specification and a few others, plus matching some Mastodon conventions.

Differences from regular blogs

Mastodon (actually mostly ActivityPub - Mastodon is just the most popular open source implementation) does add some extra features that you won't get with a regular blog:

Follows: you can follow other blogs, and see who you are following and who is following you
Likes: you can like a post - people will see that you liked it
Retweets: these are called "boosts". They duplicate someone's post on your blog too, promoting it to your followers
Replies: you can reply to other people's posts with your own
Privacy levels: you can make a post public, visible only to your followers, or visible only to specific people (effectively a group direct message)

These features are what makes it interesting, and also what makes it significantly more complicated - both to understand and to operate.

Add all of these features to a blog and you get a blog that's lightly disguised as a Twitter account. It's still a blog though!

It doesn't have to be a shared host

This shared hosting aspect is the root of many of the common complaints about Mastodon: "The server admins can read your private messages! They can ban you for no reason! They can delete your account! If they lose interest the entire server could go away one day!"

All of this is true.

This is why I like the shared blog hosting analogy: the same is true there too.

In both cases, the ultimate solution is to host it yourself. Mastodon has more moving pieces than a regular static blog, so this is harder - but it's not impossibly hard.

I'm paying to host my own server for exactly this reason.

It's also a shared feed reader

This is where things get a little bit more complicated.

Do you still miss Google Reader, almost a decade after it was shut down? It's back!

A Mastodon server is a feed reader, shared by everyone who uses that server.

Users on one server can follow users on any other server - and see their posts in their feed in near-enough real time.

This works because each Mastodon server implements a flurry of background activity. My personal server, serving just me, already tells me it has processed 586,934 Sidekiq jobs since I started using it.

Blogs and feed readers work by polling for changes every few hours. ActivityPub is more ambitious: any time you post something, your server actively sends your new post out to every server that your followers are on.

Every time someone followed by you (or any other user on your server) posts, your server receives that post, stores a copy and adds it to your feed.

Servers offer a "federated" timeline. That's effectively a combined feed of all of the public posts from every account on Mastodon that's followed by at least one user on your server.

It's like you're running a little standalone copy of the Google Reader server application and sharing it with a few dozen/hundred/thousand of your friends.

May a thousand servers bloom

If you're reading this with a web engineering background, you may be thinking that this sounds pretty alarming! Half a million Sidekiq jobs to support a single user? Huge amounts of webhooks firing every time someone posts?

Somehow it seems to work. But can it scale?

The key to scaling Mastodon is spreading the cost of all of that background activity across a large number of servers.

And unlike something like Twitter, where you need to host all of those yourself, Mastodon scales by encouraging people to run their own servers.

On November 2nd Mastodon founder Eugen Rochko posted the following:

199,430 is the number of new users across different Mastodon servers since October 27, along with 437 new servers. This bring last day's total to 608,837 active users, which is without precedent the highest it's ever been for Mastodon and the fediverse.

That's 457 new users for each new server.

Any time anyone builds something decentralized like this, the natural pressure is to centralize it again.

In Mastodon's case though, decentralization is key to getting it to scale. And the organization behind mastodon.social, the largest server, is a German non-profit with an incentive to encourage new servers to help spread the load.

Will it break? I don't think so. Regular blogs never had to worry about scaling, because that's like worrying that the internet will run out of space for new content.

Mastodon servers are a lot chattier and expensive to run, but they don't need to talk to everything else on the network - they only have to cover the social graph of the people using them.

It may prove unsustainable to run a single Mastodon server with a million users - but if you split that up into ten servers covering 100,000 users each I feel like it should probably work.

Running on multiple, independently governed servers is also Mastodon's answer to the incredibly hard problem of scaling moderation. There's a lot more to be said about this and I'm not going to try and do it justice here, but I recommend reading this Time interview with Mastodon founder Eugen for a good introduction.

How does this all get paid for?

One of the really refreshing things about Mastodon is the business model. There are no ads. There's no VC investment, burning early money to grow market share for later.

There are just servers, and people paying to run them and volunteering their time to maintain them.

Elon did us all a favour here by setting $8/month as the intended price for Twitter Blue. That's now my benchmark for how much I should be contributing to my Mastodon server. If everyone who can afford to do so does that, I think we'll be OK.

And it's very clear what you're getting for the money. How much each server costs to run can be a matter of public record.

The oldest cliche about online business models is "if you're not paying for the product, you are the product being sold".

Mastodon is our chance to show that we've learned that lesson and we're finally ready to pay up!

Is it actually going to work?

Mastodon has been around for six years now - and the various standards it is built on have been in development I believe since 2008.

A whole generation of early adopters have been kicking the tyres on this thing for years. It is not a new, untested piece of software. A lot of smart people have put a lot of work into this for a long time.

No-one could have predicted that Elon would drive it into hockeystick growth mode in under a week. Despite the fact that it's run by volunteers with no profit motive anywhere to be found, it's holding together impressively well.

My hunch is that this is going to work out just fine.

Don't judge a website by its mobile app

Just like blogs, Mastodon is very much a creature of the Web.

There's an official Mastodon app, and it's decent, but it suffers the classic problem of so many mobile apps in that it doesn't quite keep up with the web version in terms of features.

More importantly, its onboarding process for creating a new account is pretty confusing!

I'm seeing a lot of people get frustrated and write-off Mastodon as completely impenetrable. I have a hunch that many of these are people who's only experience has come from downloading the official app.

So don't judge a federated web ecosystem exclusively by its mobile app! If you begin your initial Mastodon exploration on a regular computer you may find it easier to get started.

Other apps exist - in fact the official app is a relatively recent addition to the scene, just over a year old. I'm personally a fan of Toot! for iOS, which includes some delightful elephant animations.

The expanded analogy

Here's my expanded version of that initial analogy:

Mastodon is just blogs and Google Reader, skinned to look like Twitter.

Tags: blogging, google-reader, twitter, mastodon, activitypub, fediverse, sidekiq

It looks like I'm moving to Mastodon

2022-11-05T05:32:20+00:00

Elon Musk laid off about half of Twitter this morning. There are many terrible stories emerging about how this went down, but one that particularly struck me was that he laid off the entire accessibility team. For me this feels like a microcosm of the whole situation. Twitter's priorities are no longer even remotely aligned with my own.

I've been using Twitter since November 2006 - wow, that's 16 years! I've accumulated 42,804 followers there. It's been really good to me, and I've invested a lot of work generating content there to feed the machine.

I can't see myself putting the same work in to help the world's (current) richest man pay the billion dollar annual interest on the loans he took out to buy the place on a weird narcissistic whim.

So I've started to explore Mastodon - and so far it's exceeding all of my expectations.

My new profile is at https://fedi.simonwillison.net/@simon - you can follow @simon@simonwillison.net in your Mastodon client of choice.

Not ready to sign up for Mastodon? It turns out RSS support is baked in too - you can subscribe to https://fedi.simonwillison.net/@simon.rss in your feed reader (I really like NetNewsWire for macOS and iOS these days).

Why Mastodon?

The lesson I have learned from Twitter is that, even if a service you trust makes it past an IPO and becomes a public company, there's always a risk that it can be bought by someone who very much doesn't share your values.

Mastodon has been designed to avoid this from the start. It operates as a federated network of independent servers, each of which is run by a different person or organization with the ability to set their own rules and standards.

You can also host your own instance on your own domain.

My initial nudge to try this out was from Jacob and Andrew, who figured out how to do exactly that:

The Fediverse, And Custom Domains - Andrew Godwin
Setting up a personal Fediverse ID / Mastodon instance - Jacob Kaplan-Moss

Andrew and Jacob both opted to pay masto.host to run their instance for them. I've decided to do the same. It's on my domain, which means if I ever want to run it myself I can do so without any visible disruption.

I'm paying $9/month. I find it darkly amusing that this is a dollar more than Elon has been planning to charge for users to keep their verified status on Twitter!

If you don't want to use your own domain there are plenty of good free options, though I recommend reading Ash Furrow's post about his shutdown of mastodon.technology to help understand how much of a commitment it is for the admins who run a free instance.

This post by @klillington@mastodon.ie has some good links for getting started understanding the system. I particularly enjoyed Nikodemus’ Guide to Mastodon as it matched most closely the questions I had at first.

Initial impressions

Despite taking the second hardest route to joining Mastodon (the hardest route is spinning up a new server from scratch) it took me just less than an hour to get started. I wrote up a TIL describing what I did - more or less directly following the steps described by Andrew and Jacob.

I signed into my new account and started following people, by pasting in their full Mastodon names (mine is @simon@simonwillison.net). I was initially surprised that this did nothing: your timeline won't be populated until the people you follow have said something.

And then people started to toot, and my timeline slowly kicked into life.

And it was really, really pleasant.

My fear was that everyone on Mastodon would spend all of their time talking about Mastodon - especially given the current news. And sure, there's some of that. (I'm obviously guilty here.)

But there's lots of stuff that isn't that. The 500 character limit gives people a bit more space, and replies work much like they do on Twitter. I followed a bunch of people, replied to a few things, posted some pelican photos and it all worked pretty much exactly as I hoped it would.

It's also attracting very much the kind of people I want to hang out with. Mastodon is, unsurprisingly, entirely populated by nerds. But the variety of nerds is highly pleasing to me.

I've been checking in on the #introduction hashtag and I'm seeing artists, academics, writers, historians. It's not just programmers. The variety of interest areas on Twitter is the thing I'll miss most about it, so seeing that start to become true on Mastodon too is a huge relief.

Considering how complicated a federated network is, the fact that it's this smooth to use is really impressive. It helps that they've had six years to iron out the wrinkles - the network seems to be coping with the massive influx of new users over the past few days really well.

I'm also appreciating how much thought has been put into the design of the system. Quote tweeting isn't supported, for reasons explained by Eugen Rochko in this 2018 post:

Another feature that has been requested almost since the start, and which I keep rejecting is quoting messages. Coming back to my disclaimer, of course it’s impossible to prevent people from sharing screenshots or linking to public resources, but quoting messages is immediately actionable. It makes it a lot easier for people to immediately engage with the quoted content… and it usually doesn’t lead to anything good. When people use quotes to reply to other people, conversations become performative power plays. “Heed, my followers, how I dunk on this fool!” When you use the reply function, your message is broadcast only to people who happen to follow you both. It means one person’s follower count doesn’t play a massive role in the conversation. A quote, on the other hand, very often invites the followers to join in on the conversation, and whoever has got more of them ends up having the upper hand and massively stressing out the other person.

Mastodon so far feels much more chilled out than Twitter. I get the impression this is by design. When there's no profit motive to "maximize engagement" you can design features to optimize for a different set of goals.

And there's an API

Unsurprisingly, Mastodon has a powerful API. It's necessary for the system itself to work - those toots aren't going to federate themselves!

Poking around with it is really fun.

First, a friendly note. @pamela@bsd.network wrote the following:

Hacky folks, please resist finding ways to scrape the fediverse, build archives, automate tools and connect to people via bot without their consent.

[...]

Whatever your thing is, make it 100% opt-in. Make it appropriate for a significantly more at-risk user than you are. Make sure it forgets things, purges info about servers it can't contact, can't operate in any sort of logged-in mode where consent is an issue.

We will straight up help advertise your cool thing if it respects users properly and takes the time to consider the safety and preferences of every person involved. There are a lot of fun, thoughtfully-designed toys! And there are a lot of people really tired of having to come and tell you off when you wanted to help, honestly. Help yourself and ask around before you flip on your cool new thing, let folks point out what you're missing.

(Read the whole thing, it's great.)

So far I've done a couple of things.

I built a Git scraper to track the list of peer instances that various servers have picked up. This feels like a reasonable piece of public information to track, and it's a fun way to get a feel for how the network is growing.

I also figured out how to Export a Mastodon timeline to SQLite using the timelines API and my paginate-json and sqlite-utils CLI tools, so I could explore it in Datasette.

Running my own instance means I have no ethical qualms at all about hammering away at my own API endpoint as fast as I like!

I like to follow a lot of different people, and I don't like to feel committed to reading everything that crosses my timeline - so I expect that the feature I'll miss most from Twitter will be the algorithmic timeline! This is very much not in the spirit of Mastodon, which is firmly committed to a reverse chronological sort order.

But with access to the raw data I can start experimenting with alternative timeline solutions myself.

I'm somewhat intrigued by the idea of iterating on my own algorithmic timeline, to try and keep the variety of content high while hopefully ensuring I'm most likely to catch the highlights (whatever that means.)

Past experience building recommendation systems has taught me that one of the smartest seeming things you can do is pick the top 100 most interesting looking things based on very loose criteria and then apply random.shuffle() to produce a final feed!

I have a hunch that this is going to be a lot of fun.

Tags: accessibility, twitter, mastodon, fediverse