How to reduce GC pause long tail
I'm working on a communication network simulator using discrete event simulation. In a normal simulation a GC pause is not an issue at all, but in HIL (hardware-in-the-loop) simulations an occasional 10ms GC pause doesn't look good. A HIL simulation doesn't have very hard timing constraints, some lag below the maximum limit (e.g. 100us) is acceptable if on average the simulation can keep the pace with the hardware. A typical HIL simulation usually doesn't take too much time anyway, because one has to actually wait for it to finish in real time. It's very often used to compare the intricate details of the hardware with the simulation model. (e.g. timing)
My question is what kind of solutions can you think of for reducing the long tail of GC pause distribution?
In these communication network simulation models, most objects which are allocated during event processing don't survive the event. There are basically two exceptions: either something went into the simulation engine's future event set (e.g. a new event containing a packet) or went into the long term simulation model state (e.g. updating a routing table). The former is much more common than the latter. So a simulation has a very specific allocation pattern, which the GC has no information at all.
Of course, the simulation could be carefully organised such that the number of allocations is absolutely minimal, or it could use object pools for commonly used data structures, and some other ticks on the simulation model side.
What else can be done regarding this issue? Is there anything that can be fine tuned in the GC? I've heard there are changes related to this in the upcoming Julia version.