We do an expanding paintbucket pattern that expands in 3 dimensions to whatever has been built there. Those are certainly bases, large Minecraft builds. But, which locations should we use up our clicks per second budget on? It focused on areas that are being changed rapidly, and areas that are different from the base vanilla terrain. That’s well over a billion blocks that we could click on, and learn which Minecraft block they are. And there are about 250 players online on 2b2t at any given moment in time. The area that is loaded in by a player is 144 by 144 by 256. We are downloading bases one click at a time, and bases are huge. This means it needs to get back up to speed with their entire base, as fast as possible, in the seconds before they move away to other chunks, or log off the server. Sometimes someone would log in to 2b2t at their base, after having not played for weeks at a time (so the data is in Postgres, not the map in RAM), we wanted to be able to pick up right where we left off on downloading what they had built. The reason why this needed to be so fast, is that the use case is downloading bases. It ended up at around (roughly) fifty million entries, served hundreds of thousands of requests for block data per second, and requested historical data (from the Postgres) at about thirty thousands SELECTs per second (from a table with over ten billion rows). High frequency trading software was perfect for this use case, because it greatly reduced GC pressure on the JVM, since the data was stored off-heap. We needed a mapping from block coordinate to a bit of data about that location, such as the last timestamp we checked it, what Minecraft block was there last, whether the block is interesting, uninteresting, different from seed, etc. In terms of what it took to make this happen, the stars of the show were probably the main Postgres, which was hand-tuned on top of a hand-tuned ZFS, on top of a NVME drive that was IOMMU passthrough'd to this VM for maximum performance. We used this a lot to grab what members were present at a given item stash or base, to make sure we weren't stepping on anyone's toes. As well as clicking any cluster and seeing the usernames associated with it. You could click any player and see their travel history, the route they had taken to get to that location. There was also a web UI that showed all active tracks. Then it groups by the root, which in practice means grouping chunks together by which ones semantically have been determined to be part of the same base, then makes a report by quantity per base. Here's that query Roughly, it grabs chunks with chests in that timeframe, then uses a recursive Postgres CTE to traverse the that we used to implement DBSCAN, from the leaf to the root. We'd then travel to these locations to borrow items. One of the most common queries for actual usage though, was "what bases in the last 24 hours (or whatever) have the most chests currently". For example, the headliner "blocks per second" was well over three thousand a few weeks before then. Since the ratelimit was heavily in place by then, the numbers are pretty low. Here's what that looked like near the very end. For monitoring on a meta-level, we used Prometheus and Grafana.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |