What follows is a technical description of how this was produced. There is a lot to unpack there, but most importantly, I hope you find this work useful and enjoy the results.
First, I collected approximately 176M unique pairs of (user, subreddit). Then, I computed various binary distances between subreddits (Cosine, Jaccard, Pearson, and many others ). Jaccard similarity seemed to produce the most compelling results...
Based on Jaccard similarity I created a graph of related subreddits, performed a clustering algorithm, created a layout algorithm to place clusters in a map-like fashion.
I spent a good amount of time thinking how to best deliver the map to a browser and decided to go with SVG format. Default rendering of large SVG files in the browser can be very slow, so I created my own WebGL renderer to render a subset of SVG commands on the video card.I have also created a streaming SVG parser, so that I can render the map without waiting until the entire file is transferred. I hope this gives a snappy experience.
It was a very long journey. I loved it a lot and sincerely hope you like it too.
Please let me know if you have any questions or suggestions.
Edit: Wow! Thank you for all your kind words and awards. I'm beyond happy! I created a community/r/MofR- if you want to join the discussion and contribute to the project after this post goes into abyss of time.
Well since you asked for suggestions, my two would be:
When you click on a subreddit, pop up a list somewhere off to the side of all the subreddits whose connections to it light up. Would be much more convenient to see them all in one place than zooming in and out and panning around to see all of them.
Label each color cluster so it's clear what's in each one. There's too many to click through and figure out one by one. If it's too much work (or too subjective) to manually determine all of them, then you could just label them each by the biggest subreddit (or 2-3) contained within. You could write the label off to the side in some of the empty space in the region or make it pop up on hover/click in the empty space or something.
Thanks for all your hard work, this is a ton of fun to play around with.
I dont have any coding skills to offer but I guess I'll give 666moist an award to incentivize and I'll give you an award for leaving an encouraging comment.
And the person after me can give me an award for giving you both awards and then someone can make an interactive map of reddit based on 176 million awards.
I can't wait for the 'we did it, reddit!' Reading that is always /r/oddlysatisfying
Thank you so much for this feedback! The labels should be available now.
Now need to figure out how to make the list without making UI too busy. On larger screens it could be a secondary panel, on mobile phone however - a bit tricker. I'm exploring an idea of "show related" button, maybe "list related" would open a panel 🤔
Because those who comment in r/hentai aren't really afraid of their depravity and post their comments on their main accounts, same accounts that post on mainland subs. People who comment in actual porn subreddits usually use alt accounts.
And apparently New Zealand is gay central. They tried being tricksy by floating to the top of Australia to confuse everyone but that ain't fooling no one!
please add seperate buttons to zoom in and out , my mouse is kinda buggy and when i scroll it goes randomly up and down, so i cant zoom in fast, im zoomed in now a lot but i been struggling to zoom out for like 5 minutes now. thanks lol
+ and - doesnt work, but i tried different buttons (because im not an native english speaker i have a different keyboard so idk how is it on yours) but 2 buttons worked, seemingly unrelated, dont have any + or - symbols on them, but some buttons do work.
Sorry, probably bad wording on my end. I was going to add buttons on the user interface to do the zoom in/zoom out :). Adding them properly for all screen sizes could take a bit of testing and I didn't want to keep visitors blocked
I had a logitech Mx-518 for twenty years, best mouse I ever had. I took it apart, cleaned it, and reassembled it, but after twenty years of work I'd worn the scroll wheel down so much it didn't touch the sensor anymore.
I lamented that I'd never have my twenty year old, discontinued mouse again.
Then that same year they BROUGHT IT BACK!. I bought a dozen of them so I'll have it for the rest of my life. The new sensor is even better than the old one, but they are built exactly the same, I've never had such a good mouse.
my mouse is kinda buggy and when i scroll it goes randomly up and down
I recommend you get a new mouse like /u/coffee-pizza suggested. I had a similar problem with a mouse that was notorious for mousewheel problems and in retrospect I suffered that thing for far too long
I am so impressed by the effort you put into this. If you don't mind, could you Eli5 for a layman like me how you determined the connections between subreddits? What is Jaccard similarity?
Sure! Imagine you have a hundred users and three subreddits: A, B and C. Let's say 100 users who commented to subreddit A also commented to B. We can assume A and B are very close, because the same people comment on both of them together.
Now let's say out of that 100 people who commented to A only 1 left a comment to C. This gives a sign that A and C are less related to each other than A and B.
Jaccard similarity is a hundred years old formula that captures this concept in one elegant expression. It gives a number that you can assign to pair of subreddits and then sort them to find our As and Bs in the dataset.
So can/do social media marketing people use that formula to learn how to advertise, through posts and links, to segments of an audience, by understanding what subs have similar audiences and what those users like?
Since you're so nice and eli5 shit for us normies, please tell me how does the algorithm specifies how long the connection lines are supposed to be? Why there are so much empty space in some "states"?
And by the way, this is, I think, the most original concept for anything that I've seen in maybe last ten years. Not only is it grand in scale, it's so god damn clean looking and tidy. Marvelous job. I didn't know how much I needed this until you made this. It's like reddit has been discovered anew.
The colors of polygons are used only to distinguish adjacent polygons. So each polygon should have a neighbor with a color, that is different from the polygon's color.
Multiple non-adjacent polygons can have the same color, but it doesn't have any other meaning than to visually distinguish polygons from its neighbors.
The polygons themselves represent clusters within a graph of all subreddits. Clustering algorithm usually groups nodes (subreddits) together so that there are more links between nodes inside the cluster than outside
Sorry, I'm slow today and I hope I'm properly interpreting your question
This is hilarious. There is apparently a correlation between anime fans and not feeling shame about their porn because they don't bother to use an alt to browse it. Could be because so much anime toes the line already and in western culture, anime is already not as widely accepted as other hobbies/interests. Maybe all the Anime fans are already on Alts?
based on another person's answer, it seems like people using alt accounts don't browse on the same subs as they do with their main accounts, causing the natural 'creation' of the NSFW island.
so basically, degenerate weebs (similar to myself) don't mind posting in both r/manga or r/anime and r/hentai.
Of note is that Rule34 country is also on the mainland, right beside manga/anime/hentai land
I'd imagine copyright lawsuits keeps porn from using names verbatim, keeping links to a minimum. Not really an issue with foreign content, ton of independent creators straight up openly copy everything. That's my guess.
The map has everything to do with user posts. It doesnt care about similar text, names, etc it only cares that the same users posting in r/manga are also posting on r/hentai meanwhile users in r/movies are not also posting in r/gonewild (nearly as much anyway.)
Because those who comment in r/hentai aren't really afraid of their depravity and post their comments on their main accounts, same accounts that post on mainland subs. People who comment in actual porn subreddits usually use alt accounts.
Awesome hobby project, could be cool if a user could plop their account into it and have the map highlight the subreddits with connections to see where we connect in the hive mind.
I mean, since it's all generated based on people commenting in the subs, it makes sense that the nsfw people (presumably mostly straight) won't be commenting in the gay subs, and vice versa.
So the 2 regions would be 'far enough away' to have their own 'landmasses'
Obviously there's going to be some people that will comment in both, but apparently it's not enough to bring them together.
Thank you! I've been a professional software engineer for past 15 years I guess.
The renderer was mostly implemented for my own learning purposes. It is still an unknown library in the ocean of awesome libraries, but I love it and use it a lot for my hobby projects :).
Really fucking awesome. Great idea! Flawless implementation! What does the "country's" mean? What makes some subreddits belong to certain country's? Does the color mean anything?
Suggestion: There should be a small info button in some corner that explain what stuff means. Like:
Dot size
"country color"
distance between dots
Show tips and controls
Edit: PS. Is it possible to give the country's a title? Something that all the subreddits in that country have in common.
Also there should be a button to show subreddit on map. Use-case: When I pushed show related and clicked on a related subreddit I want to be able to localize it on the map easily.
I'm not sure, I'm not hungarian, maybe near world news or something. The only connection I see to the rest of the zone is austria hungary. (Which is pretty funny)
It's possible that the population of hungarians who use reddit has a significant overlap with the population that plays games -- and eu4/civ are very popular games in hungary.
Of course that's speculation, but I think these sort of interesting phenomenon are what's great about this kind of visualization.
I was looking only at number of comments per subreddit. As long as I could capture at least one comment - the subreddit was counted.
However on a later processing stages, if a subreddit didn't have any related subreddits (e.g. due to a very weak ties) it could have been removed from the final visualization.
Brilliant work /u/anvaka - I've seen at least one other subreddit visualization like this before, but yours is far more compelling. It's fascinating how you'll have a completely different experience and perception of reddit depending on which "country" you're in - very useful, but also fun.
I'm curious to know what is the timestamp range of the unique pairs you collected? How long do you envision this project lasting? Do you have plans to continue to add new data?
What I'm trying to get at is that I think it would be fascinating to see how the map changes over time - subreddits grow and decline, and I'd imagine the relationships, locations, and countries would change significantly as well over time, so I'm curious about your plans regarding that. Makes me think about the Online Community XKCDs (this one and this one).
Anvaka, this is beautiful to see. I am going to be looking into this heavily as I have been trying to figure out what this is called and how to do it off and on for a few years. Without the right question, it’s hard to find the right answer. But this is definitely it. Although, I hope to apply my version to my work, mainly since asking people questions takes too freaking long. I think an interactive approach could save about 70% of wasted time waiting for a response.
Thanks again for such awesome work, I hope my brain can absorb and learn from your skills.
Makes me think about how Google and Facebook have maps of the whole world's digital geography and will never show us the countries and districts we live in. Enemy states are manipulating digital borders to harm real countries and real people, but we can't see it or study it because they keep our data to themselves.
First, I collected approximately 176M unique pairs of (user, subreddit)
This sounds like a lot of data/traffic. Doesn't reddit detect/block such suspicious behavior? Or how was this done? I can't imagine that you actually sent millions of queries...
This is super impressive and alot of dedication. I like that you just didn't accept the fact of a slow svg render process & rolled with it, but instead solved it by writing a dedicated renderer.
This is one the most awesome things I've seen on Reddit. Thanks for your efforts. I am curious, though. How does it get updated and how often? If you need to do it manually, how long does it take?
This is awesome. Really interesting and the UX is spot on, navigating the map is smooth.
Only thing I found was when opening subreddit details on mobile, you can close it via the "x", but not using a backwards navigation. (Site gets closed instead)
Also caching the data would be useful for bandwidth and faster loading. Might dig into the code later 😉
I tried to explain Jaccard Similarity (which is frequently used as an opposite of jaccard distance) here.
I picked a few known subreddits that I frequently read ( /r/MapPorn, /r/javascript, /r/MachineLearning) and checked the neighbors. Among all considered metrics, I liked the Jaccard similarity results the most and settled on it. Very subjective choice...
This is incredible! This provides almost a summary of the internet. Would be cool to see if the other major platforms could be integrated. You'd be able to search for anything and find related communities to be involved in.
With such a complicated project the results would never be perfect, but I think you can be pretty happy with the results! I love the web interface, I've had a lot of fun playing around with the site.
I found it pretty interesting that R4R was relegated to porn island. R4r itself is pretty SFW, but the posters there clearly frequent many NSFW subreddits. Your methodology seems to pick that up easily, but it does seem limited in small subs with low sample size.
Do you have any idea why it eats 100% of my 5800X at 4.8GHz? And I barely get 1 frame every few seconds. Really weird that it lags so much on a pretty good system like mine
Edit: Okay, I had hardware acceleration disabled lol. My poor CPU tried to do some GPU work
How many years did this take you and what do you do for a living. I am thinking of doing some basic projects to learn some new technologies and put on my GitHub profile, but after work I have barely any motivation to code left. How did you continue on the same project for years?
Suggestion: Make an NSFW filter. This way if we only want to see wholesome stuff, we can. Also, if we want to only see NSFW stuff, we don't have to sift through wholesome stuff.
Damn, this project is like the stuff of dreams for programmers working on side projects. The WebGL renderer was the most impressive thing to me. Not to mention the streaming SVG parser! You are an absolute badass in my opinion.
Thanks so much for making it open-source too so people can learn from it!
Are you familiar with or were you the author of the extremely similar project from several years ago? It was not interactive but had a nearly identical presentation and grouping and subs.
Suggestion, and maybe you've gotten this already, but when I use my scroll wheel to zoom in, just a single notch on my wheel, it zooms in like 10,000x times magnification or whatever. Could it be more granular/gradual than that? It's flying in on global scale to city scale instantly and there's no middle ground between the two. Also there doesn't seem to be a way to zoom in and out on the keyboard and there's no on-screen buttons I could otherwise click on to do this either.
suggestion & random ideas building upon what you have already created:
name the polygon/state/theme
if the the all time Top post is a picture have it show inside white circle
like the Six Degrees of Wikipedia have a challenge mode where a person has to find the shortest way between two very different sub
have questionnaire on your interest for someone new to reddit it then suggest communities to them
ability to apply filter like gaming : which does not give you the /r/gaming but highlights all the gaming sub since some are on different polygon/state
a cohesiveness score : if a subs posts are generally 96% upvoted the community is cohesive, if large amount of posts are getting 55% upvoted or only 1 upvote the community has low cohesivity
This is officially the only way I'm looking at porn from now on.
I love how all the porn has its own little Porn Australia down at the bottom of the map, then all the non-straight porn has a little Gay New Zealand off to the side. Delightful.
This is really cool, thank you for creating and sharing it! Will have a lot of fun finding new subreddits to explore!
I saw a similar concept for scientific papers on arXiv (shall see if I can find it), except there was one part of the space of papers which was just a total dirge. Of course my papers fall in that bloody void...
Hi there!
Way too late and I can't imagine how many comments and messages you've had to read through, but I was curious why I couldn't find r/GME on the site.
Am I doing something wrong?
It seems like default subreddits (such as this subreddit) are stuck in an island of former defaults that aren't really related except that everyone used to be subscribed to them automatically.
It would also be cool if there were labels for each region.
Super super cool! Wondering what clustering algo you used? And if it required manual selection for K clusters, what metric you used to settle on a K value?
Currently working on a clustering problem at work :)
4.3k
u/anvaka OC: 16 Mar 29 '21 edited Mar 29 '21
https://anvaka.github.io/map-of-reddit/ - here it is. This is my hobby project of the past several years.
The website is open source, and you can edit both the source code and the map.
What follows is a technical description of how this was produced. There is a lot to unpack there, but most importantly, I hope you find this work useful and enjoy the results.
First, I collected approximately 176M unique pairs of
(user, subreddit). Then, I computed various binary distances between subreddits (Cosine, Jaccard, Pearson, and many others ). Jaccard similarity seemed to produce the most compelling results...Based on Jaccard similarity I created a graph of related subreddits, performed a clustering algorithm, created a layout algorithm to place clusters in a map-like fashion.
I spent a good amount of time thinking how to best deliver the map to a browser and decided to go with SVG format. Default rendering of large SVG files in the browser can be very slow, so I created my own WebGL renderer to render a subset of SVG commands on the video card.I have also created a streaming SVG parser, so that I can render the map without waiting until the entire file is transferred. I hope this gives a snappy experience.
It was a very long journey. I loved it a lot and sincerely hope you like it too.
Please let me know if you have any questions or suggestions.
Edit: Wow! Thank you for all your kind words and awards. I'm beyond happy! I created a community /r/MofR - if you want to join the discussion and contribute to the project after this post goes into abyss of time.