Pretendo

May Progress Update

Published by
Jon Jon
on June 2, 2025

Hey all, Jon here! We know it's been quite a while since our last blog post. We planned to get these out more often, but alas more important issues and updates came first. We apologize for the lack of blog posts, and promise to get them out as often as we can. With that, halfway into the year seems like as good a time as any to summarize what we've been up to!

(Note: This blog post will not cover all changes made since the last post, as there are far too many to list. We will be highlighting the more relevant/larger ones here)

Technical Updates

Many of you voted to have more technical updates in blog posts from now on, and today marks the first time we do!

Splatoon Database Performance

Around May 26th, 2025 we noticed random instability in Splatoon. The game would work for a while, allowing players to create new, and join existing, matches, before seemingly randomly locking up. This locking up would cause subsequent requests to never get a response from the server, resulting in failed connections. Developers shutterbug and quarky quickly began investigating. We were unsure where the issue was at, but we had a pretty good idea of where to look. The locks could be happening in 2 places:

  1. The application layer. A few examples being:
    1. A locked mutex which never releases. Our servers are written in Go, and make use of goroutines for concurrency. Primitive Go types such as slices and maps can only be accessed by one goroutine at a time. To ensure this, a mutex is used to lock access when a goroutine needs the data, and is released when it's done with it. If a goroutine were to lock up and never release the mutex, then future goroutines will never be able to access it, and also be locked up waiting for the mutex to release.
    2. A poorly structured SQL query could lock up the database connection at the application layer, causing future requests to no longer be able to access the database.
  2. The database layer. A few examples being:
    1. Database corruption could lead to queries failing or getting stuck, locking up the connection.
    2. Poor indexing causing full table scans on massive tables, locking up the connection.
    3. Bad queries causing the table to infinitely loop and lookup in itself, locking up the connection.

Shutter began by checking the code for locked mutexes and the server logs for any abnormalities. Our game server libraries are split into 3 separate modules, one of which we colloquially refer to as "common". This module holds all the "common" default implementations of features used by many games. Quarky began by investigating the database itself. Together, they were able to narrow down the problem areas to 2 functions. Lockups would happen in the MatchmakeExtension::AutoMatchmake_Postpone and MatchMaking::UnregisterGathering functions. These functions are responsible for creating/joining multiplayer sessions, and for deleting them when the game is over.

There did not appear to be any areas where a mutex would not release, so we switched our attention back to the database itself. We had noticed some wonky behavior from the database earlier as well, later fixed by Will, and I had a hunch that this is where the issue would lie as well. Upon investigation, we found several instances of poor database performance, such as a lack of indexes on some "hot spots" resulting in slower queries. Most notably, we found 2 queries that had become locked, running over 7 and 9 hours respectively. These queries are what was causing the database to lock up, and requests to not get responses. But we still didn't quite know WHY they were locking. It was not initially clear if the issue was due to a looping query, poor indexes, or database corruption. The database in question only held ephemeral data about temporary matches, however, so we opted to just rebuild the database and add in the missing indexes at the same time. Doing so would cover both possibilities of database corruption and missing queries causing the locks, and so far things have once again become stable.

During the investigation I also noticed a lack of a proper timeout mechanism in the Go functions that made the database queries. Regardless of the reason for why the queries locked up, having a timeout would prevent things from looping forever.

This investigation has highlighted some key areas of improvement in regards to our database performance, and we have already begun investigating using even better indexes (currently being tested in Minecraft) and will begin to implement proper query timeouts. So while it is unfortunate that Splatoon was down, in the end this will result in even better performance moving forward.

DataStore Protocol Rework

As mentioned above, many games share a "common" implementation of certain features. This allows us to define this functionality once and simply tell a server to use it. In the past, our philosophy was to be as unopinionated as possible in this regard, only implementing basic boilerplate for the functions and providing developers a series of configurations and hooks to use. The actual logic of the functions was still up to the developers to set. This was done so that developers outside of our team could more easily integrate our code into their systems, by just hooking up whatever stack they already use.

We have recognized the limitations of this approach however, and have decided to become more opinionated. This first took place in the matchmaking code, made by Dani, making it so that all the logic for matchmaking now lives in our "common" module. This now allows our code to be more "plug-and-play", simply telling servers to use matchmaking and letting our logic do the rest.

Another protocol which used this unopinionated philosophy was our DataStore implementation. DataStore is a protocol developed by Nintendo allowing for "object" (file) storage in games, essentially acting as a frontend for S3. This protocol is what powers games like Super Mario Maker (courses are "objects"), Animal Crossing: New Leaf (dreams are "objects"), Mario vs. Donkey Kong: Tipping Stars (levels are "objects"), and more.

We have now begun the process of moving our DataStore implementation into a more opinionated implementation, allowing for the same sort of "plug-and-play" nature as seen in the matchmaking changes. Progress can be seen here. Once finished, this will be our first fully complete protocol implementation, allowing for MANY games to get their "object" features working without any extra work. This is also why Mario vs. Donkey Kong: Tipping Stars is still offline, it is so old that it needs to be rewritten from the ground up and is mostly just DataStore, making it the perfect candidate to test these new changes against.

Eventually we hope to do this with all protocols in the future.

New Games/Updates

Several new games have joined us since our last blog post! Both existing games coming out of beta, and entirely new games!

Also a big thank you to everyone outside of our team as well for helping debug and suggest changes!

Splatoon

As a belated celebration for Splatoon's 10th anniversary (we deeply apologize that the stability issues mentioned above prevented us from doing anything on time), for the next round of rotations (starting June 7th) our rotations will feature previously banned stages! Allowing banned stages has been a highly requested feature for a long time, so we felt this was a good time to try it out!

Animal Crossing: New Leaf

Thanks to the work by shoginyan and shutterbug, Animal Crossing: New Leaf has officially left beta testing and is available to everyone! Note that Dream Suite is not yet implemented (see DataStore Protocol Rework), but multiplayer is supported! We've also identified an issue regarding best-friend messages, of which a fix is being actively tested.

Mario Kart 7 Communities

Mario Kart 7 now has partial support for communities! Leaderboards are not currently implemented, as Mario Kart 7 uses a version of the Ranking that is very different to the one used by modern games.

Yo-kai Watch 2 and Yo-kai Watch Blasters

shoginyan has also kick-started Yo-kai Watch 2 and Yo-kai Watch Blasters both into development! Yo-kai Watch 2 has left beta and is available to everyone, while Yo-kai Watch Blasters (and variants) is currently in beta testing.

Monster Hunter 4 Ultimate

Monster Hunter 4 Ultimate has officially left beta testing! We have also removed the region lock, allowing for experimental crossplay between MH4U and MH4G players! You may encounter technical difficulties when playing across regions due to this experimentation, so please report any issues you encounter to the game's issue tracker.

Swapdoodle

Swapdoodle was one of the first titles Dani wanted to work on, prior to him joining our core dev team. After joining, focus was shifted elsewhere and work stopped on Swapdoodle fairly early. However thanks to the work by outside contributors Silver-Volt4 and CenTdemeern1, work on Swapdoodle has once again picked up! They took it upon themselves to implement the server entirely, with promising results! For more information, see:

Dr. Luigi and Dr. Mario: Miracle Cure

Both Dr. Luigi and Dr. Mario: Miracle Cure have officially left beta testing! Both titles are now available to the public!

The Legend of Zelda: Tri Force Heroes

After several months of downtime, The Legend of Zelda: Tri Force Heroes is officially back online! This was one of our legacy games requiring upgrades to our latest libraries, and has finally had its issues ironed out. Due to some recent changes to our general matchmaking code, the games multiplayer region lock was unintentionally returned. This is being looked into and should be fixed in due time.