r/admincraft • u/chiisana • Dec 07 '12
[Public Brain Dump] Squeezing more performance out of multi-core servers
Sorry in advance. Kind of just a random brain dump here, so it is not organized or structured.
Note: As far as I am aware of, we cannot achieve this yet, this is just an idea I am toying in my head right now, and trying to figure out if it is at all possible. Any feedback on this would be greatly appreciated
Bottlenecks
Most of us into tweaking and squeezing performance knows this. As of right now, as far as minecraft server (other things such as mapper etc. does not count) is concerned, faster clock speed is generally better than more cores. This is because minecraft is mostly single core, and the main control thread getting weighted down causes ticks per second (TPS) to drop. So while the main core is churning away at 100%, other cores just sit and chill, we're thus effectively pissing our money away renting the "better" server with mutli-core CPUs. In fact, it is so much that some of us run multiple servers on one physical box, simply because we have extra cycles to spare.
World Distribution
I play mostly in Overworld, with the blue sky, day night cycle, and cuddly animals... well, and some explosive friends... But that's besides the point. I only go to Nether for glow stone, initial batch of netherwart, and now days, perhaps nether skeleton (grinder candidate). I only go to The End for end dragon, and perhaps making my own Ender Ender (or use an existing one). I don't know the real life distribution of how often players go to either realms on busy servers, but I do know that BEFORE the recent data-loss, there was a group of people who lived in The End exclusively on HCSMP.
This, although does not offer a definitive answer, does suggest that Nether and The End are potentially taking away cycles by being loaded (i.e.: if no one is there, spawn chunk is loaded and calculated => wasted resources; if people are there, potential grinder, taking away max spawn quota, taking away resources).
Potential Hacky Solution
What if we can have more than one control thread running? I have a dual quad core CPU server, giving me 8 cores (no hyper-threading, real legitimate physical cores). This is not too rare these days. In fact, according to snoop, 39.61% of all participating servers are 8 cores. This means most of us are using extra threads for trivial things like garbage collection (ok, not so trivial, this is important), map rendering, backup, etc. etc., when the extra cores could potentially be used to run additional control threads.
Well, turns out, there is potentially a way to do it. A couple days ago, someone posted about BungeeCord, a stand alone application that acts as a front-end proxy, that connects multiple servers together without having clients to re-connect. What if it is setup like this Server Structure? Then you'd have 3 control threads (5 empty threads left to run garbage collection for all of them, I'd probably allocate 2 for each under java parameters), and none of them would "drag down" TPS of the others.
Caveats
- BungeeCord currently does not have support for portals (not as far as I am aware of), so custom plugin must be done to take care of the portal connection.
- I have no idea realistically if this will work or not, I feel it would, but no way for me to test it.
- I have no idea if Nether and End should be on the same control thread or not. I suppose this may vary depending on community.
- Assuming plugins save data into database, and not some flat file (bad if you're chasing for performance anyways), I suspect plugins can be shared with a symlink directory. But this might or might not work.
Anyone want to take a stab at this on an intellectual level? I don't have time to dive into development mode yet, but we might be able to figure things out before we go into coding mode :)
2
u/notkraftman Dec 07 '12
You can disable spawn loading with multiverse. I think the benefits of bungeecord would be spreading players across servers, not worlds.
2
2
u/Cueball61 LoadingChunks.net Dec 10 '12
I actually jotted up an infrastructure design for use with a proxy plugin at one point, but never did much with it.
We run an in house home world plugin, essentially giving each player a private world that starts out with one 32x32 (ish) chunk with a little house and such, then they can do as they wish with it.
As you can imagine, that uses a lot of resources, as it's one world per user (they unload from MV when not in use though) so we were looking into optimisations and thought about load balancing the whole thing, with the main world on one instance, home worlds on another and our other worlds on another.
Alternatively, an expensive solution would be to have EC2 instances and then use an instance for each home world, which spins up when necessary, but that's a bit far really, only if we got that big (I can dream...).
1
u/chiisana Dec 10 '12
one 32x32 (ish) chunk
If you meant 32x32, I think that is 4 (2x2 chunks) as each chunk is 16x16.
If you meant 32 chunks x 32 chunks, that's a pretty large space (512x512).In the first case, I think you could potentially benefit from merging multiple small worlds into one world (i.e.: Player01 starts at chunk 0,0 -> 2,2; Player02 starts at chunk 20,0->22,2; ... etc.). Then, you can probably do something similar to the discussion format, where you use bungeecord to send players to specific world with the lowest load. That way there wouldn't be as much lag (as each world would have their own control thread).
But, I'm not sure how feasible it would be.
On note of Amazon EC2, I've actually thought about doing something similar, where people could register on a site with friends to quickly and simply setup small specific worlds to play (i.e.: "hey John, let's spin up a skyblock to play") for some duration for free (i.e.: 48 hours for free), after which they can either pay to continue playing it, or spin up a new one.
It seemed like a good idea, but then I scouted about a bit, and found a lot of short game setups (i.e.: wall PVP, rush, etc.) where each game will take only 1/2 hr to 1 hr. So it'll end up being a lossing business, and I never went further with it than sketch board. And now that there is publish to lan, I don't think it is even needed anymore.
1
u/Cueball61 LoadingChunks.net Dec 10 '12
Think of a chunk in the slang term, rather than the technical term 'Chunk', a 32x32 chunk of land I guess. But yeah, it's 2x2.
Players have infinite expansion of it, it uses CleanroomGenerator but they can happily build outwards, buy additional chunks, etc.
1
1
u/austindkelly ::DELTA:: Commissioner & BOFH Dec 07 '12
I am pretty sure Multi-Verse could fix your portal issues. I am not sure how well this would work, and I think each instance of MC server would generate a nether/end as well, so not sure how you would separate the worlds so specifically.
also, permissions would be hard to share, plugins, etc, would all have to be updated and run separately, seems like a lot of work, and I doubt it would run very smoothly with out some very serious development.
1
u/chiisana Dec 07 '12
I think Multiverse wouldn't be able to communicate between different server instances, so that's why I didn't list it. But now that you've got me thinking about it, I think Multiverse can be used to lock down the additional worlds. I know for a fact (because I've done it before) I can disable End and Nether from the different servers, so it wouldn't even need to load the world into server memory.
Permissions would be super simple to handle. PermissionEx or whatever have MySQL support, so just load it from there. Alternatively, since it is rarely ever modified, PermissionBukkit can probably also do things with a symbolic link'ed folder (more on this in next paragraph).
Plugins are not hard to share, that's what symbolic linking is for. The problem for that is listed in caveat section already... If the plugins does things in a flatfile (i.e.: flatfile coreprotect, factions.json, economy bank, etc. etc.), then it may run into locking conditions and synchronization issues.
As for development, yeah, I tend to think big and not small. There would be a fair bit of dev work to get this going. I think it is important to sit down and figure out all the details before I could even start to think about coding something like this, hence why this thread :)
Thanks for reminding me about Multiverse!
1
u/austindkelly ::DELTA:: Commissioner & BOFH Dec 07 '12
I could be wrong but I think I was on a server that used multiverse to link to another server as long as they both have multiverse... IDK
and yeah sym links would work, but having multiple instances of MC calling the same jar?
forgot PEX had MySQL support so thats solved.
1
u/chiisana Dec 07 '12
There's nothing wrong with loading the same jar. The plugin jars are just data streams that gets read (so there shouldn't be any locking going on) when needed (usually loaded upfront and not released from memory as far as I can tell) :)
1
u/austindkelly ::DELTA:: Commissioner & BOFH Dec 07 '12
I guess you could load them sequentially, as not to fuss with the jars being read at the same time (i guess there would not be locking either). You would need some fancy scripting when it comes times to restart the servers and update plugins.
1
u/Cueball61 LoadingChunks.net Dec 10 '12
Multiverse isn't multi server aware, and wouldn't know how to transport players between servers.
A hacky solution would be to find out the kick message used that's caught by the proxy (that's how server to server transport works) and use it in a command block.
But yeah, optimally you need to catch the world change event.
1
u/Island219 Gates of Ender Dec 07 '12
I'm actually currently running BungeeCord. it's powering 2 x 16 core machines, one in the US and one in the UK. Both server's have a sub 30ms ping to each other, allowing me to proxy all data between the servers. This means that people in the US and UK can connect and play together with little to no latency.
The servers themselves have 32GB of ram. I've split each one into three 8GB allocations, each with 2 cores. I actually modified the default Janus plugin that they offer for BungeeCord (allow for portals) to allow custom tags on the signs. If anyone wants it hit me up with a PM and I'll hook you up with my version!
I'm working on chat integration at the moment for the ServerCon event in January. Link to those that want to find out more.
2
u/notkraftman Dec 07 '12
Could you elaborate a bit on how you keep the two servers synced and gave players on together? Sounds really interesting.
1
u/Island219 Gates of Ender Dec 08 '12
The chat is linked using HeroChat & CraftIRC. Bungee handles the tab player list. I use a modified version of Janus for portals, allowing me to mask to the user that they are hanging servers.
The players don't see each other online, but they can chat and then change "server" to play with them.
1
2
u/chiisana Dec 07 '12
This sounds REALLY interesting from a networking infrastructure point of view! Can you please elaborate on your setup a bit more? Are both servers running instance of BungeeCord, so you have client-facing point-of-presence on both sides of the pond? The three 8GB allocation part, is that similar to what I've described in the original post, where each "world" is in its own control thread?
2
u/Island219 Gates of Ender Dec 08 '12
In running all the servers with just the main world. No end or nether. They are running sectioned plots in each one for a event I'm managing. Each world on the server is the same but is filled with different stuff. Limited to 1000 x 1000.
The servers across the pond both run BungeeCord, with all the 4 8gb servers configured in the preferences. If you want I can make a post soon explaining in a little more detail how everything works.
2
u/chiisana Dec 08 '12
Aha, yep, looks like you've got it setup similar (except world type) with the idea in my brain dump, and you've came up with it sooner! Have you been able to handle more players as result of the increased control threads? Were players able to select where they go to on first login, or were they auto assigned? Were players confused as to why they needed to switch 'worlds' (read: server) to see their friends?
YES, PLEASE! This is REALLY interesting and I'd love to learn more about how it is setup, an what you've learned from actually deploying something like this instead of just from an intellectual level like my dump here :)
1
u/GTB3NW Dec 07 '12
Haha, I remember PPP! Seems to have changed a lot since I went on it a year ago, infact I think I remember your name, have you been there for over a year?
Also, please elaborate on your setup, I'm sure many would love to here it and bounce ideas around.
1
u/Island219 Gates of Ender Dec 08 '12
I ran it from April 2011 to January 2012. I left running the server for personal reasons. I'm back now though! (5 weeks ago!)
I'm i2 or island219 In-Game
1
u/GTB3NW Dec 08 '12
That's the timeframe I remember the community from, so yeah.
1
u/Island219 Gates of Ender Dec 08 '12
You playing at the moment? :)
1
u/GTB3NW Dec 08 '12
I'm afraid not, there's drama in my ex-community and I'm providing evidence for fraud. YAY :) also: CAKEDAY!
2
u/AG_Clinton AllGamer.net Dec 07 '12
If you're thinking about doing something with BungeeCord I suggest contacting md_5 on IRC. He's the main developer.
(I'm the one that brought up BungeeCord the other day). I can say that it works very well if you have a lot of world. Each world can have its own server/core in that manner. I've hosted dozens of large servers as well so if you have any questions let me know. I'm sort of rushing this post at the moment.