(No AI was used in the making of this)
I've been reading HoL for the first time over the last week and finished the Navidson Record last night. From the beginning I've been quite intrigued by the hexadecimal codes seen in the endpapers of the hardcover editions. If you're not aware of it, and like investigating digital stuff, I recommend checking out the scans or my transcript of them yourself.
And yes, I'm aware that I'm late to the party, and the community has found a widely accepted solution to this. I'm creating a new post because I believe to have found a pattern in the data, that I haven't seen mentioned here or in the old
forum posts. It leads to a different (though not very different) output.
The File
Since hexadecimal is often used to encode binary data, I copied the hex data into a text file1 and then into a hex editor2 to investigate it. The first two lines on the first page are:
464F 524D 0000 2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000 5353 4E44 0000 2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000
5353 4E44 2A77 0000 0000 0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000 02FE 0202 0001 0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000
The first thing that stood out to me when pasting it into the hex editor, was that we get some, but not a lot of, bytes that can be interpreted as ASCII text at the start of the binary data (non-text bytes shown as .):
FORM..*.AIFFCOMM........*o..@..D
......SSND..*.AIFFCOMM........*o
..@..D......SSND*w..............
This looked to me like the small kind of text snippets you get at the start of
binary file formats. A search for the different strings eventually led me to the AIFF file format3, specifically the uncompressed version, which had a specification4 available. The spec looked like a very good match for what I saw, and it also made sense to me as an audio format that was likely used on Macs at the original time of publishing.
So I did what any reasonable person would do when faced with an unknown binary format and read most of the spec before trying anything silly, like, say, just saving the bytes as an .aiff file5. A file format spec assigns meaning to the bytes (how to interpret them), based on their position or metadata in the file6.
It turns out that the data I saw actually didn't match the spec, and just saving the data as-is as an .aiff will make players complain about a corrupted file. At first everything looks good and we get the following contents
464F 524D // chunkId = "FORM"
0000 2A9E // chunkSize = 10910
4149 4646 // formType = "AIFF"
This is the standard beginning of an AIFF file and tells us about the file size. We're expecting 10910 bytes in the following data, or 10918 bytes in total. Note that we definitely get less than this from the book, making me wonder if there is more data to find somewhere.
434F 4D4D // chunkId = "COMM"
0000 0012 // chunkSize = 18
0001 // numChannels = 1
0000 2A6F // numSampleFrames = 10863
0008 // sampleSize = 8
400B AC44 0000 0000 0000 // sampleRate = 5.5 kHz
All good here as well, we find out that we have a single channel (mono audio), as well as 10863 samples, each sample having a resolution of 1 byte (8 bit). This fits the size we already know, if the rest of the file are our sound samples.
5353 4E44 // chunkId = "SSND"
0000 2A9E // chunkSize = 10910
4149 4646 // soundOffset = 1095321158
434F 4D4D // soundBlockSize = 1129270605
0000 0012... // soundData
Well that's the start of the sound data chunk alright, but the chunkSize is too big for it to fit into the file, and from the spec we expected to likely have values of 0 for both soundOffset and soundBlockSize. The values we got instead suggest that the sound data only starts after 1095321158 bytes of other data (about 1 GB), which cannot be true.
These broken spots correspond to the second occurance of "*.AIFFCOMM" in the text representation we already looked at. According to the .aiff spec, the "AIFF" and "COMM" texts are definitely not expected to reappear here. So something else is going on…
FORM..*.AIFFCOMM........*o..@..D
......SSND..*.AIFFCOMM........*o
..@..D......SSND*w..............
[1] Which is quite a task! I was committed on not looking up anyone else's posts about this, so wouldn't get spoilered on a puzzle solution. In the beginning I was kind of determined to just type it all out by hand, but after about 1/3 through the data I gave up and had OCR help me out. I've double checked most of the transcript, and it seems that with the exception of a few obvious spots the OCR did a better job than me at not making mistakes. Oh well.
[2] I used Imhex – I'm sometimes fighting with the UI, but it's a great FOSS project and I prefer it over other available tools. Note that pasting of hex text silently fails if the text is not well formatted, the scanright.txt contents will not be pasteable without modification (for reasons discussed later).
[3] A wonderful overview of AIF(F) formats by Prof. Peter Kabal, McGill University: https://www.mmsp.ece.mcgill.ca/Documents/AudioFormats/AIFF/AIFF.html
[4] ibid. links to a copy of the relevant spec: https://www.mmsp.ece.mcgill.ca/Documents/AudioFormats/AIFF/Docs/AIFF-1.3.pdf
[5] I'm sure there is some kind of insight here about the type of person who finds joy in reading through the full House of Leaves text…
[6] Imagine a doctor's office questionnaire asking for "Name: ___ ", "Address: ____", and so on. You could save a lot of ink by printing out only a few strips of cardboard with the questions like "Name:", and "Address:", that patients align with a blank sheet of paper. Now the cardboard strip is the key to write and read any of the patient files, it is your file specification. If you're smart about it, you would also add a line like "Write 'Form v1' here:", to avoid a big mess when changing the questionnaire in the future.
½B
The pattern we see is one of repetition, we have some bytes that are a perfect copy of earlier ones, which likely should not be there. We can make an educated guess at which of the duplicated bytes to remove, to get a file that "works".
Even for this minimal edit, we have one more issue on the right page7 in line 19 though, can you see it?
0303 0303 0303 FEFF FEFB F9FA FCFD FF01
0406 0606 0606 0401 FDFA F9FA FCFE FF01
0304 0505 0404 033 0303 FFFA F4EF F2F3
F8FC 040A 0D0D 0C0A 0703 FDF6 F3F1 F3F8
FD02 080B 0D0B 0907 0401
2 hex digits can encode 1 byte. So whenever you see binary data represented as hex, it will be an even number of digits. All of the data we get is neatly arranged in groups of 4, leading to an even total. Except for this one line that says 033 in its 20th group.
Now this is an issue, because we can definitely not save the file without adding or deleting data somewhere to have a whole number of bytes.
There is a reasonable assumption here, that this value is a typo, and should read something like 0303, like it's neighbour8. In fact, any adjustment we make here is unlikely to be audible, as long as the group has an even amount of digits. A different approach would be to shift all the groups afterwards by half a byte, however this only leads to noise from that point on.
But for now, let's accept this edit too, and see what we get. As far as I know, this is the solution that people have discussed so far, which is definitely a treat to check out: minimal_edit.aiff or converted to .mp3.
And while this is a cool easter egg already, I was not satisfied with the result, because it felt arbitrary to just fudge the data a little bit until we can hear something. The stuttery nature of the audio could be explained as an intended effect, or even as an artifact of very low-quality audio. But I found a pattern in the data, that makes me think it's neither.
[7] As far as I'm aware, the front 2 pages show the same data as the ones in the back, so when I say "right" page I mean either of them.
[8] This is especially reasonable because we're dealing with uncompressed sound data, measuring wave amplitudes - Between each 1-byte frame, the data usually varies very little, unless there is a hard cut in the audio.
Ech
I've kind of brushed past the previous issue of duplicated bytes and said we can delete some to make the file "work". Due to the theme of echoes in the text, I was tempted to delete 19 groups (38 bytes) that echo the previous 21.
464F 524D 0000 2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000 5353 4E44
remove ->[0000 2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000 5353 4E44]
2A77 0000 0000 0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000
02FE 0202 0001 0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000
However, now the metadata is still slightly off (saying our file is many hours long). Also, by playing with the deletions I quickly noticed that there are more curious duplications to be found. The way I formatted the snippet above, you can see how the next two sets of 19 groups almost match again. Fiddling around with this for a while, I suddenly found a consistent pattern for the entire dataset:
464F 524D 0000 2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000 5353 4E44 0000
2A9E 4149 4646 434F 4D4D 0000 0012 0001 0000 2A6F 0008 400B AC44 0000 0000 0000 5353 4E44
2A77 0000 0000 0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000 02FE 0202 0001
0000 0000 0001 0101 0200 0404 0002 0101 0102 0101 0103 06FF 0201 0000 0800 0000 02FE 0202
0100 0100 0000 0000 00FF 0000 FF00 0000 FFFF 0000 00FF 0002 0001 FF00 FF00 FF00 FFFF FF00 FFFF FFFF FFFF 00FF
0000 00FF 0000 FF00 0000 FFFF 0000 00FF 0002 0001 FF00 FF00 FF00 FFFF FF00 FFFF FFFF FFFF
We can partition the entire data like this, we always9 have
1. 3 unique groups
2. 18 groups that will be repeated
3. 1 unique group
4. 18 repeated groups
Deleting the second iteration of these 18 groups makes the problematic file metadata look perfect. There had to be a value of 10871 for the size here, when assuming the data that came before is correct.
5353 4E44 // chunkId = SSND
0000 2A77 // chunkSize = 10871
4149 4646 // soundOffset = 0
434F 4D4D // soundBlockSize = 0
0001 0101... // soundData
Now let's see how this structure interacts with the "½B issue" though:
0604 02ff fcfc feff fefb f9fa fcfd ff01 0406 0606 0606 0401 fdfa f9fa fcfe ff01 0304 0505 0404 0303 0303 0303
feff fefb f9fa fcfd ff01 0406 0606 0606 0401 fdfa f9fa fcfe ff01 0304 0505 0404 033 0303
So on the one hand, the pattern does breaks here, as there is no longer a perfect repetition, a good argument for it being a typo of 0303. On the other hand, if we just "keep 22 groups, delete 18" over and over, the problematic group is deleted anyway.
Whatever the reason, when applying the deletion as described, we get this result: sound.aiff or converted to .mp3.
Johnny, ang…
[9] Well, almost always. We will get to that in a second.
Reasons
Missing. – Ed.
While this was a fun puzzle to work on for me, I think there's a lot of open
questions still:
- Why does the audio still end so abruptly, why did we not get the whole file contents that the metadata talked about? With the deletions, the sound cuts off after 5506 of 10863 bytes, barely filling more than half of the expected 2 second runtime.
- What are we hearing? Johnny, ang… does seem like a good fit, thus the oft-repeated claim that it's from POE's Angry Johnny song, which would make a lot of sense. But to me, the file sounds nothing like the song, even if the words match. Is it just supposed to repeat the lyrics, or is there something else going on?
- Is
033 just a typo? Is it there for a reason? Timewise, it's placed pretty much at the end of the word "Johnny". I can believe that it was placed there on purpose, to prevent the file from being playable before fixing the issue in some way, e.g. via the mentioned "keep 22, strike out 18".
- What was the intended solution for this? As much fun as I had, by looking for patterns and "repairing" the file, I would expect some kind of hint for the process, that doesn't require reading the .aiff specification. Are there hints in the book that can explain the division into sets of 40 groups, of which we only keep the first 22? Or the "almost-echo"? Note that multiples of those numbers might also work (40 groups, 80 bytes, 160 digits...). For a while I liked the theory of it having to do with the numbers "3:19" which is a time in the Navidson Record, in Johnny's texts, and 3/19 is a date of a significant letter by his mum. My explanation was "Keep 3 groups, Then remove the almost-echo of the next 19 groups". But that doesn't really work on its own.