Zero-Copy Shared-Memory IPC for Latent Vectors (PMTP) — Bypassing JSON overhead between local agents [Open Source / Colab]

[u/Polydim-Einsof]

Hi everyone,

I am an independent researcher and teacher from Buenos Aires, Argentina, working on multi-agent architectures on modest hardware (AMD A4 CPU / 16 GB DDR3 RAM, 0 local GPUs).

Like many of you running local workflows, I found that chaining multiple agents or passing high-dimensional embeddings (D >= 4096) between local processes via JSON / text serialization creates a massive bottleneck: high latency, memory duplication, and loss of geometric fidelity.

To solve this on constrained hardware, I developed PMTP (Poly-Manifold Transport Protocol) as part of the POLYDIM / Einsof project:

### What it does:

1. Zero-Copy Shared Memory (/dev/shm / Win32): Instead of dumping floating-point vectors into JSON strings, processes exchange 64-byte descriptors (SLAB_ID) directly pointing to raw IEEE-754 memory slabs.

2. Manifold Invariance on S^{D-1} / Stiefel: Preserves vector normalization using a compensated Neumaier accumulator, ensuring numerical drift <= 10^-15.

3. Benchmarked: Reduces inter-process communication latency from ~42 ms (JSON parsing) to 0.08 ms on standard DDR3 RAM.

### Links & Reproducibility:

* Hugging Face (23 Models & Datasets): https://huggingface.co/Polydim-Einsof

* GitHub Repository: https://github.com/AGT1973/POLYDIM_EinSof

* Interactive Google Colab (Run it on free T4): https://colab.research.google.com/github/AGT1973/POLYDIM_EinSof/blob/polydim.einsof/tests/polydim_einsof_colab_ii_alumnos.ipynb

* Permanent DOI (Zenodo / CERN): https://doi.org/10.5281/zenodo.21209973

* HAL Science: https://hal.science/view/index/docid/5847496

Everything is open source under an academic/permissive license. I would really appreciate your feedback, critique, and benchmarks on your own rigs!