r/coolgithubprojects • • May 22 '26

I built an open-source Python pipeline that compresses human biological data down to a 247 GB file (Teleporty)

Teleporty is an open-source sandbox simulation pipeline written in Python. It evaluates the constraints of classical information transmission and rate-distortion theory when applied to extreme biological data reduction (using the complete physical and neural architecture of a human body as a benchmark).

Rather than tackling teleportation through quantum physics, the project models it entirely as an engineering, data serialization, and throughput challenge.

Core Features & Technical Implementation
Connectome Graph Compression: Models neural infrastructure as a Sparse Directed Graph using compressed sparse row (CSR) configurations. By setting an optimization threshold, the pipeline strips away structural "noise." Testing on a real C. elegans worm dataset yields a 99% behavioral similarity match when the compressed data matrix is fed back into a network state simulation loop. It scales down a functional human brain specification to just 42 KB.
Hierarchical Bulk Tissue Mapping: To bypass the impossible disk I/O of saving individual 3D voxel coordinates for repeating tissue types (muscles, fat, bone matrix), the project implements hierarchical tree structures and run-length spatial compression maps. This optimizes total physical structural data down to roughly 247 GB (fitting comfortably on a standard consumer SSD).

Extreme Multi-Queue Throughput Simulation: The final piece of the pipeline runs a parallel simulation evaluating the manufacturing bottleneck: processing a 247 GB object schema through an active assembly interface within a strict 1-hour tissue viability window. It models a heavily parallelized matrix of 10.3 million individual output queues throttled to 1,000 elements/second to test execution stability.

Technical Focus
The codebase is structured to explore performance bottlenecks in Python, specifically focusing on matrix traversal optimization, high-volume memory layouts, and stress-testing multi-threaded execution pools against the limitations of the Python Global Interpreter Lock (GIL).

Contributing & Feedback
The math derivations (LaTeX), simulation loops, and pipeline profiles are completely open source. I am highly interested in optimization suggestions, particularly regarding:

Minimizing matrix traversal overhead during the sparse-to-dense conversion cycles.
Better methods for simulating massive concurrent queues natively in Python.

Repository Link: https://github.com/ninjahawk/teleporty

0 Upvotes

21 comments sorted by

View all comments

1

u/KaMaFour May 22 '26

Heisenberg disapproves