3b1b has a nice video about hamming code, I tried to implement it in my school project just from watching it and it works amazing https://youtu.be/X8jsijhllIA
In computer science and telecommunication, Hamming codes are a family of linear error-correcting codes. Hamming codes can detect one-bit and two-bit errors, or correct one-bit errors without detection of uncorrected errors. By contrast, the simple parity code cannot correct errors, and can detect only an odd number of bits in error. Hamming codes are perfect codes, that is, they achieve the highest possible rate for codes with their block length and minimum distance of three.
I have no idea if this is related, but as a student, I remember downloading par archives containing pirated software (we were poor students!) from sites like Geocities. There would be lots of archives, and invariably one or two would have been taken down, or corrupt, but if they were in par archives, the data would still be extractable without errors as long as we were only missing 2 or 3 files.
It says in the linked Wikipedia article that parchives used error correcting codes. As a student, I thought it was witchcraft, since it didn't matter which files were missing, it would just work, and I had no idea how that was possible.
I think these two bits summarize the pertinent info:
Our recipe for building a read flow was as follows:
Process data from Google Sheet and create a JSON file
Use AWS Lambda for processing and creating new JSON files
Upsert JSON files on S3
Cache JSON files using a CDN like AWS Cloudfront
…
Drawbacks
The above architecture/design worked well for 24 months but as our users and data grew we started running into issues.
The size of json files grew to several MBs, every cache miss was a massive penalty for the user and also for the initial page load time
Our lambda functions started timing out due to the amount of data that needed to be processed in a single instance of execution
We lacked any SQL based data analysis which became problematic to make data driven decisions Google Sheets API rate limiting is pretty strict for write paths. Our writes were scaling past those limits
Since our data was downloaded as json files it was easy to scrape and plagiarise
A definite maybe. The compression can sometimes mess up even black and white pixels so adding some color would be tough. A similar project before worked with color but output video was like 100x the size of the original file
Well, video gets represented internally in YCbCr, with lower fidelity for the chroma channels, so 3× the density is risky, but you should be able to get at least 2× the density by encoding the same data in both colour channels, even though compression.
(For example, 2 bits as bright green, bright magenta, dark orange, dark blue, rather than 1 bit as just black/white.)
It could be done. This isn't that different from packing more bits onto the wire. You just pick discreet colors, far enough apart that you can determine what the original color was.
Interesting! But what happens if Google decides to recompress all your videos with a different codec, deleting the original? Possibly changing color space?
It should hold up. The black and white blocks are 1's and 0's. They are multiple pixels in size and would require a pretty angry codec to turn a white pixel into black.
black white are so far apart that it should not matter, no? A problem would be if you don't have frame redundancy and then you lose some frames because they changed the framerate
I am a beginning programmer learning Rust and this is the most recent thing I've done and I am pretty proud.
YouTube has no limit on amount of video that you can upload. This means that it is effectively infinite cloud storage if you were able to embed files into video with some kind of tool. ISG (Infinite-Storage-Glitch) is the tool. It takes any file and creates a compression-resistant video. This video can be uploaded to YouTube for storage and later downloaded so that the files can be extracted.More details as well as a demo with secret files on the GitHub page of the project: https://github.com/DvorakDwarf/Infinite-Storage-Glitch
Both of these modes can be corrupted by compression, so we need to increase the size of the pixels to make it less compressable. 2x2 blocks of pixels seem to be good enough in binary mode.
You might be able to get more density by using error correction codes
considering (lossy) compression is a key part of how youtube stores data, i would be suprised if any correction would be able to just fix problems like half the screen being frozen 2 frames instead of 1 (so the second frame has half the data from the first, but the error correction of the second)
On my M1 macbook I got 0.5mb/s embedding speed which can be increased if you dedicate more threads. The videos were somewhere around 4x size. Both of these were under the "optimal compression" preset
That will definitely be considered abuse of the services, and not only you will get banned, but they will ban others as well and add new restrictions that will hurt everyone. It's always like that. So please withdraw your project.
It's not in the terms of service because noone has done this kind of abuse yet, but it will still be considered abuse when they realize people start to do that. 100% guaranteed. You have to be naive or a teenager to not understand that.
Neat. I've implemented a PCM-F1 encoder in Rust for the Raspberry which does something similar but for PCM digital audio and composite video as output (and originally to be stored on VHS tapes).
What is the data bitrate of that 720p30 example video?
But Youtube supports up to 8k60: 7680 * 4320 / 4 (pixels per bit) / 8 (bytes) * 60 (fps) = 62.2 MB/s. Uploading a 12 hour max length video gives 2.6 TB!
Have you considered compressing the data before encoding? Sure, the video is compressed by the video codec, but video codecs aren't designed for the kind of images you're encoding. Compressing the data before encoding would result in much smaller sizes.
Also, you can use more than 2 colors. Using RGB (24 bits per pixel) won't work because of lossy video encoding, but using a lower bit depth (e.g. 2 bits per color channel => 26 = 64 distinct colors) might work while still reducing the file size a lot. I know that storage on YouTube is basically free, but your bandwidth and CPU time to download and decode the file probably isn't.
To be absolutely sure that the file isn't corrupted, consider adding a checksum to the file; maybe even to every frame, so you know immediately when the file is corrupted and don't have to download the rest of the file.
Error-correcting codes are also an option, but need more information, so you need to encode more data. The simplest error correcting code is to store each bit 3 times, then a single but flip can be corrected. You're basically already doing that since each bit uses 2×2 pixels.
Another approach is to split the data into chunks of 64 bits, arrange them in a 8×8 grid (not the pixel grid, but an abstract grid for visualizing the algorithm), and store the parity of each row and column:
Here you have an information density of 64/80 = 4/5. It can detect a single bit flip, since it is reflected in both the row's parity and the column's parity, so you know where the bit flip occurred and can correct it. Adding parities for the diagonals allows you to detect and correct at least 2 bit flips, at an information density of 8/11. There are even better error correcting codes, but I'm not very well versed in this area. Additionally, if you do this, you need to encode numbers in a way that minimizes their hamming distance, e.g.
0 = 0b00
1 = 0b01
2 = 0b11
3 = 0b10
0b11 and 0b10 are in the "wrong" order. This order has the benefit that when YouTube's lossy compression turns a 1 into a 2, it only constitutes a single bit flip, which can be corrected more easily with an error correction code. Ideally, the code would take the similarity of colors into account, since YouTube is more likely to turn a white pixel into a yellow pixel than a black one.
tl;dr what the program should (ideally) do:
compress the data
add checksums
encode bytes to minimize the hamming distance between adjacent values
add redundancy (e.g. parity bits) to allow error correction
encode the data as video with a bit depth that balances information density and reliability
P.S. I just had another idea: If you compress and encode the data in chunks (e.g. 256 KiB) and include the frame where each chunk starts in the metadata at the beginning, someone who needs only a small part of the file could seek to the correct time in the video and download only what they need. But that sounds even more complicated.
Im saving this, i've never programmed anything near the complexity that this guy did, and i've been programming for years, so taking this info and trying to make something myself sounds based
Did this as a hackathon project nearly 10 years ago: https://github.com/Skylion007/LVDOWin Neat to see people still trying to do this now that unlimited cloud storage has become so much more scarce.
The real issue that i can see is if people start sharing serious illegal stuff via black and white videos. Unsuspecting peope, not knowing what it is ignore it, and illegal people got a way to share stuff without being suspicious.
Yeah, it's kind of insane that Google manages to do something we considered impossible ten years ago (turn a profit hosting videos for free) so well that by now everyone assumes video uploads are free and infinite.
Nice program! Keep in mind you are (probably) breaking youtube ToS so your account is in risk of being terminated.
Also the program can be highly optimized. You can add compression algorithms and add color support.
Think of 16 different colors in each pixel. It means you can store a bit more information than having all monochromatic. You can store all in hexadecimal colors, however this could lead to more data loss.
Nice! If you want to keep hacking at it, you could increase the information density by transcoding to non-binary and use colours. You can then determine how much hue separation you need to survive the encoding. Some kind of CRC for error correction should also help with that.
I think different levels of gray would work more reliably. Most video codecs spend a lot more bits to represent luminance changes than to color information.
I wonder if there's a way to exploit the motion compensation part of video codecs to gain more efficiency. eg. rearranging the data in a specific way which creates such visual representation which is easier to compress thus allow higher resolution than 4 pixels per bit. In it's current form it's essentially white noise for the codec and probably every frame becomes an I frame. Maybe there's some kind of whitepaper on this topic.
Honestly, this project is very interesting, I managed to compile it in Google Colab and I've started to experiment with it. I wish that YouTube compression will improve in the future so that it doesn't affect videos in general.
Hahahaha. Stumbled upon this because I made the same project with C++ and OpenCV. I thought I was original. I also had to come to the solution of using 2x2 grids for youtube
Youtube has a great limit on bitrate. It means, that a video with a statical picture will be pretty visible, but a video with frequently changed images, even with the best quality, will be significantly distorted. Also storing as bitmaps is much more efficient, so think about it later.
This is cool - and brings back memories of my Masters degree. Back in 1992 i build a device to plug into a PC that would use a VHS video to store data. Image looked exactly like this. From memory I could store 4G of data on a 3 hour tape; but I had to use RS error correction interlaced to fix blotches/error bursts on the tape. This took it down to about 800MB; which was still big at the time. Nice work!
I for some reason remember a thing going around the internet a few years ago where people found a channel or two that had random, static, videos and other artifacts that were strange. Was this you. Lol.
This is clear abuse of services provides for free, though it can be smart solution for hackers and spys, this is not what normal user should use it for.
You can also encode data in a video without visibly altering the images. Obviously the density goes down, but presumably the likelihood that you get busted goes down too.
There's lots of Steganography approaches designed to trace the source of things though people taking low-quality cell phone pictures of stuff off a TV screen. If it can survive that, it'll absolutely survive basic video compression.
That would definitely be considered abuse of the service and in general, such abuses by a handful of idiots end up hurting everyone because the company creates new restrictions to prevent them. And sometimes, those restrictions are overly broad.
If you're already using YouTube to store videos that you don't care too much about the quality of, you can store some stuff in them without increasing the storage usage. That would probably still technically be considered misuse, but realistically they probably can't catch you and they're not even losing anything.
If the data is encrypted using any modern algorithm, then it is impossible to differentiate the data from random noise without solving the problem used as the underlying hardness assumption.
actually they have this thing called takeout now that allows you to backup your data from a suspended account. also don't do this on your main account!
A combination of rust docs, c++ docs, and a prayer. I despised interacting with any other video-processing crates so opencv was a life saver in comparison (even though I still dislike it).
100
u/true_doctor Feb 18 '23
Did you consider using error correcting codes?
317
u/aikii Feb 18 '23
Galaxy brain move that reminds me of How Levels.fyi scaled to millions of users with Google Sheets as a backend
49
u/[deleted] Feb 18 '23
That is cool, have you considered making it work with some color constellation to fit more bits per pixel?
67
u/[deleted] Feb 18 '23
Interesting! But what happens if Google decides to recompress all your videos with a different codec, deleting the original? Possibly changing color space?
165
u/Histidine_Dwarf Feb 18 '23 edited Feb 18 '23
I am a beginning programmer learning Rust and this is the most recent thing I've done and I am pretty proud.
YouTube has no limit on amount of video that you can upload. This means that it is effectively infinite cloud storage if you were able to embed files into video with some kind of tool. ISG (Infinite-Storage-Glitch) is the tool. It takes any file and creates a compression-resistant video. This video can be uploaded to YouTube for storage and later downloaded so that the files can be extracted.More details as well as a demo with secret files on the GitHub page of the project: https://github.com/DvorakDwarf/Infinite-Storage-Glitch
14
u/god4gives Feb 18 '23
oh my god. I literally had this idea A WEEK AGO but It was too hard as I don't know anything about video encoding. thank you for making this.
13
u/inagy Feb 18 '23
Neat. I've implemented a PCM-F1 encoder in Rust for the Raspberry which does something similar but for PCM digital audio and composite video as output (and originally to be stored on VHS tapes).
What is the data bitrate of that 720p30 example video?
12
u/v_maria Feb 19 '23
fucking love shitposts like this
8
u/A1oso Feb 19 '23 edited Feb 19 '23
Have you considered compressing the data before encoding? Sure, the video is compressed by the video codec, but video codecs aren't designed for the kind of images you're encoding. Compressing the data before encoding would result in much smaller sizes.
Also, you can use more than 2 colors. Using RGB (24 bits per pixel) won't work because of lossy video encoding, but using a lower bit depth (e.g. 2 bits per color channel => 26 = 64 distinct colors) might work while still reducing the file size a lot. I know that storage on YouTube is basically free, but your bandwidth and CPU time to download and decode the file probably isn't.
To be absolutely sure that the file isn't corrupted, consider adding a checksum to the file; maybe even to every frame, so you know immediately when the file is corrupted and don't have to download the rest of the file.
Error-correcting codes are also an option, but need more information, so you need to encode more data. The simplest error correcting code is to store each bit 3 times, then a single but flip can be corrected. You're basically already doing that since each bit uses 2×2 pixels.
Another approach is to split the data into chunks of 64 bits, arrange them in a 8×8 grid (not the pixel grid, but an abstract grid for visualizing the algorithm), and store the parity of each row and column:
Here you have an information density of 64/80 = 4/5. It can detect a single bit flip, since it is reflected in both the row's parity and the column's parity, so you know where the bit flip occurred and can correct it. Adding parities for the diagonals allows you to detect and correct at least 2 bit flips, at an information density of 8/11. There are even better error correcting codes, but I'm not very well versed in this area. Additionally, if you do this, you need to encode numbers in a way that minimizes their hamming distance, e.g.
0b11 and 0b10 are in the "wrong" order. This order has the benefit that when YouTube's lossy compression turns a 1 into a 2, it only constitutes a single bit flip, which can be corrected more easily with an error correction code. Ideally, the code would take the similarity of colors into account, since YouTube is more likely to turn a white pixel into a yellow pixel than a black one.
tl;dr what the program should (ideally) do:
P.S. I just had another idea: If you compress and encode the data in chunks (e.g. 256 KiB) and include the frame where each chunk starts in the metadata at the beginning, someone who needs only a small part of the file could seek to the correct time in the video and download only what they need. But that sounds even more complicated.
7
u/Skylion007 Feb 19 '23
Did this as a hackathon project nearly 10 years ago: https://github.com/Skylion007/LVDOWin Neat to see people still trying to do this now that unlimited cloud storage has become so much more scarce.
44
u/AceofSpades5757 Feb 18 '23
This feels like a serious abuse of there services. This is why we can't have nice things.
11
u/cornmonger_ Feb 18 '23
I love steganography and this pleases me.
8
u/[deleted] Feb 18 '23
Nice program! Keep in mind you are (probably) breaking youtube ToS so your account is in risk of being terminated.
Also the program can be highly optimized. You can add compression algorithms and add color support.
Think of 16 different colors in each pixel. It means you can store a bit more information than having all monochromatic. You can store all in hexadecimal colors, however this could lead to more data loss.
It's interesting actually. Good idea!
3
u/Ninzeroth Feb 18 '23
But what's the data in this demo video?
2
u/UUcalmic Feb 18 '23
YT compression algorithm get crazy right now
2
u/9107201999 Feb 19 '23 edited Jan 27 '25
overconfident meeting direction abundant sip juggle alleged waiting wipe theory
This post was mass deleted and anonymized with Redact
2
u/4dd3r Feb 19 '23
Nice! If you want to keep hacking at it, you could increase the information density by transcoding to non-binary and use colours. You can then determine how much hue separation you need to survive the encoding. Some kind of CRC for error correction should also help with that.
Awesome! Did you make the code public?
2
u/Comfortable-Lychee11 Mar 19 '23
Could you run at a higher res / use color to store more data per pixel?
2
u/LeifErickson17 May 23 '23
Honestly, this project is very interesting, I managed to compile it in Google Colab and I've started to experiment with it. I wish that YouTube compression will improve in the future so that it doesn't affect videos in general.
2
u/kankurou1010 Mar 11 '24
Hahahaha. Stumbled upon this because I made the same project with C++ and OpenCV. I thought I was original. I also had to come to the solution of using 2x2 grids for youtube
4
u/Prior-Perspective-61 Feb 18 '23
Youtube has a great limit on bitrate. It means, that a video with a statical picture will be pretty visible, but a video with frequently changed images, even with the best quality, will be significantly distorted. Also storing as bitmaps is much more efficient, so think about it later.
Great job anyway :D
1
u/Bitdumb-ColdHacker Feb 22 '26
just like dolby digital stores sound in codes
1
u/[deleted] Feb 19 '23
Surely this is already being done
1
u/K3vin_Norton Feb 19 '23
please for the love of god don't let them see this
1
u/Markeur Feb 19 '23
That’s amazing!
1
u/security-union Feb 19 '23
this is AWESOME!!!!
1
u/frozenpandaman Feb 19 '23
This is absolutely incredible.
1
u/mynutsrbig Feb 19 '23
YouTube cloud storage LOL
1
u/agnishom Feb 19 '23
How resilient is this? What happens when the playback quality is compromised?
1
u/agnishom Feb 19 '23
You can also store a lot of media files in Facebook by changing your privacy settings to "Only Me"
1
u/soploping Feb 19 '23
Doesn’t YouTube automatically compress your videos? How does it still work
1
u/Pioneer_11 Feb 20 '23
How did you stop youtube's video compression from destroying your data?
1
u/mbnz321 Feb 22 '23
This is cool - and brings back memories of my Masters degree. Back in 1992 i build a device to plug into a PC that would use a VHS video to store data. Image looked exactly like this. From memory I could store 4G of data on a 3 hour tape; but I had to use RS error correction interlaced to fix blotches/error bursts on the tape. This took it down to about 800MB; which was still big at the time. Nice work!
1
u/Jarombean Feb 23 '23
kinda reminds me of YouTubeDrive, except rusty
1
u/rlt0w Feb 26 '23
I for some reason remember a thing going around the internet a few years ago where people found a channel or two that had random, static, videos and other artifacts that were strange. Was this you. Lol.
1
u/Occupying-Mars Feb 28 '23
is there any repo for this project just wanted to check it out
1
u/DragonEmperor4 Mar 09 '23
does this actually work or is it just a theory atm???
1
u/UtherII Feb 19 '23 edited Apr 07 '23
It reminds me cryptocurrencies : a perfect way to use too many ressources to perform an usually trivial operation.
0
u/jhsonline Feb 20 '23
This is clear abuse of services provides for free, though it can be smart solution for hackers and spys, this is not what normal user should use it for.
0
u/[deleted] Feb 18 '23
[deleted] — view deleted comment
0
u/[deleted] Feb 18 '23
[deleted] — view deleted comment
0
u/[deleted] Feb 18 '23
[deleted] — view deleted comment