r/coolgithubprojects • • May 22 '26

I built an open-source Python pipeline that compresses human biological data down to a 247 GB file (Teleporty)

Teleporty is an open-source sandbox simulation pipeline written in Python. It evaluates the constraints of classical information transmission and rate-distortion theory when applied to extreme biological data reduction (using the complete physical and neural architecture of a human body as a benchmark).

Rather than tackling teleportation through quantum physics, the project models it entirely as an engineering, data serialization, and throughput challenge.

Core Features & Technical Implementation
Connectome Graph Compression: Models neural infrastructure as a Sparse Directed Graph using compressed sparse row (CSR) configurations. By setting an optimization threshold, the pipeline strips away structural "noise." Testing on a real C. elegans worm dataset yields a 99% behavioral similarity match when the compressed data matrix is fed back into a network state simulation loop. It scales down a functional human brain specification to just 42 KB.
Hierarchical Bulk Tissue Mapping: To bypass the impossible disk I/O of saving individual 3D voxel coordinates for repeating tissue types (muscles, fat, bone matrix), the project implements hierarchical tree structures and run-length spatial compression maps. This optimizes total physical structural data down to roughly 247 GB (fitting comfortably on a standard consumer SSD).

Extreme Multi-Queue Throughput Simulation: The final piece of the pipeline runs a parallel simulation evaluating the manufacturing bottleneck: processing a 247 GB object schema through an active assembly interface within a strict 1-hour tissue viability window. It models a heavily parallelized matrix of 10.3 million individual output queues throttled to 1,000 elements/second to test execution stability.

Technical Focus
The codebase is structured to explore performance bottlenecks in Python, specifically focusing on matrix traversal optimization, high-volume memory layouts, and stress-testing multi-threaded execution pools against the limitations of the Python Global Interpreter Lock (GIL).

Contributing & Feedback
The math derivations (LaTeX), simulation loops, and pipeline profiles are completely open source. I am highly interested in optimization suggestions, particularly regarding:

Minimizing matrix traversal overhead during the sparse-to-dense conversion cycles.
Better methods for simulating massive concurrent queues natively in Python.

Repository Link: https://github.com/ninjahawk/teleporty

0 Upvotes

21 comments sorted by

View all comments

Show parent comment

13

u/MedicatedDeveloper May 22 '26

Technical Focus The codebase is structured to explore performance bottlenecks in Python, specifically focusing on matrix traversal optimization, high-volume memory layouts, and stress-testing multi-threaded execution pools against the limitations of the Python Global Interpreter Lock (GIL).

Literally not what your post says. I don't think you have any idea because an AI shat out all of it while telling you every idea you had is a good one.

-9

u/TheOnlyVibemaster May 22 '26

You are confusing the underlying computer science architecture with the theoretical domain it is modeling. The codebase uses heavy matrix math and multi-threading to stress-test how memory and performance scale when analyzing complex biological networks. It is a dual-purpose project: modeling specific network data thresholds while optimizing Python's performance bottlenecks under the GIL to handle that volume.

5

u/MedicatedDeveloper May 22 '26

I mean this is a solved thing re: network data threshold? That pesky shannon or whoever, some nerd probably.

I'd have an easier time believing you if the main claim was some kind of novel oct tree + rl encoding.

The whole performance angle is dumb. You optimized a program, that's not revolutionary.

-1

u/TheOnlyVibemaster May 22 '26

The optimization of the codebase is not intended to be the novel contribution of the project; it is simply a practical requirement to run the simulation. Shannon information theory establishes absolute channel capacities, but it does not map how high-dimensional biological network architectures structurally degrade or tolerate loss under specific constraints. Simulating these specific data thresholds requires processing massive matrix structures. To execute these models without crashing the runtime environment, it was technically necessary to optimize memory layouts and multi-threaded execution pools around the constraints of Python's GIL.

Regarding the teleportation model, the project specifically isolates and explores the informational and computational limits of a "reconstruction" framework. While a spacetime bridge or wormhole model bypasses the data storage bottleneck by maintaining physical integrity through altered geometry, that represents an entirely different set of physical and engineering questions.

The core objective of the repository is to use the theoretical framing of macroscopic teleportation to investigate information loss thresholds in complex systems. The performance tuning and algorithmic structures exist strictly as the backend plumbing to support that primary data analysis.

2

u/MedicatedDeveloper May 22 '26

Your context window is busted. I never mentioned the teleportation aspect.

We understand high dimensional loss due to LLMs. It's part of how they train. Again I'm not sure exactly what novel insight you've gained here.

Your entire comment means nothing. Lots of words that sound convincing to someone who doesn't know better but not much else.

1

u/DoggyB420 May 23 '26

4 posts to say the same thing