- cross-posted to:
- dataisbeautiful@lemmy.ml
- datahoarder@lemmy.ml
- cross-posted to:
- dataisbeautiful@lemmy.ml
- datahoarder@lemmy.ml
I was recently inspired by this post made by @mookulator@mander.xyz to get the subscription statistics for a list of Lemmy instances.
I did this by scraping the data from the Communities tab from every Lemmy instance listed on the awesome-lemmy-instances from GitHub. So all of this data is available publicly.
I separated it as follows:
Local instance: The instance that the data is being scraped from
Community: The name of the community
Community instance: The instance that the community is hosted on
Local Subscription count: The subscription count of that community coming from the local instance
If the Local instance is equal to the Community instance the Subscription count is actually the total number of users subscribed to that community across the Threadiverse.
Since I was web scraping these websites the data is a bit rough because I had to convert stuff like 42K into 42000 so it isn’t going to be 100% accurate.
Also, this scrape doesn’t include instances that weren’t on the list when I pulled the CSV and alternatives to Lemmy like Kbin or Mastodon users subscribing to Lemmy communities.
This was gathered over the course of a day starting from 12:00 PM EST to about 7:00 PM EST today.
The data could be better if I used the API or added the information from lemmyverse.net on the total subscriber counts but I spent a lot of time on this as is and don’t know how to use the API.
I hope someone uses this to make a data visualization of subscription patterns for Lemmy because I would really like to see that.
P.S. On the post that inspired this post, there was some discussion about whether Lemmy users would like this to be done. So if you guys don’t like it I will delete the data.
Your approach may have ended up being for the best. lemmyverse.net can’t index lemmy.world (broken DB => broken API => “well, we didn’t want to be part of the fediverse anyway”), and if you’d tried to use their API yourself, it might have totalled your project the same way it did for lemmyverse.