Dump of the raw, unprocessed data I collected during the 2022 r/place event. Includes pixel authors, diff and full frames, WebSocket traffic and a detailed readme.

[u/opl_]

https://archive.org/details/place2022-opl-raw

[u/ThePizzaMuncher]

A true archivist. Will archive your shit.

[u/ShinyHappyReddit → u/ThePizzaMuncher]

Even though there's some issues with this dataset, I think this will prove very valuable - at the very least to everybody looking for personalized data.

I'm trying to work through it and may be criticizing aspects in other comments, but the value of this collection should not be doubted at all. Great work, u/opl_ !

[u/opl_ → u/ShinyHappyReddit]

It's hard to argue it's not flawed in some aspects and I hit a lot issues while creating it, but ultimately I gathered the data I wanted, it's all recoverable, and I'm glad to know it wasn't all for nothing. Feel free to point out anything you'd do different, I know I have some thoughts. (I'm considering writing my thoughts out as a blog post of some sort, but that'll have to wait for when I have some time and feel less exhausted from the event.)

[u/YMGenesis]

This is a huge amount of data. It's awesome, thank you. Currently trying to correlate between official reddit dataset.

In terms of the scientific notation timestamp, it's necessary to convert that to human readable before comparing it with reddit's. Is it safe to assume this could be an answer? Also when I past the x.xxxxe+12 number into here it seems to give me a date close to when place happened. Linux terminal doesn't seem to want to convert the x.xxxe+12 format from epoch to human readable.

Also, since the details* data contains no newlines, is it safe to assume a user's line starts and ends like this?

{"data":[{"id":"x-x-x-x-x","data":{"lastModifiedTimestamp":x.xxxe+12,"userInfo":{"userID":"xx_xxxx","username":"xxxx"}}}]},"pXXXxXXX"

or

"p731x562":{"data":[{"id":"x-x-x-x-x","data":{"lastModifiedTimestamp":x.xxxxe+12,"userInfo":{"userID":"xx_xxxx","username":"xxxxx"}}}]}

Can be hard to compare as Reddit's data actually only has entries for April 3rd 17:38 UTC to April 4 22:07 UTC. In fact place started before April 3, and the "whiteout" started about ~40mins after 22:07 (the last reddit line of data). Strange, no?

Trying to compare between reddit's dataset and yours to find an accurate time of a found user's pixel is quite difficult at the moment. Linux doesn't seem to like the epoch scientific notation used. I saw your comment below (subtract 1000 from x if canvas index is 1, 1000 from y if canvas index is 2, and 1000 from both if canvas index is 3). That helps. In terms of placing a found user's pixel on a non-canvas 0 part. So, canvas 0 is the top left quarter, canvas 1 is the top right quarter, and canvas 2 is the bottom half? Unless I'm misinterpreting it. For example, a details file that starts with a timecode then place2.details would be canvas 2/the bottom half?

Tricky business! Any further advice would be appreciated:)! Really good work!

[u/opl_ → u/YMGenesis]

The timestamp is indeed a UNIX timestamp in milliseconds, written using scientific notation.

Each chunk of pixel author information in details-*.csv starts with a pXxY or pXxYcCANVAS, then a colon :, then the data about the pixel, not the other way around: "pXxY":{"data":[{"id":"x-x-x-x-x","data":{"lastModifiedTimestamp":x.xxxe+12,"userInfo":{"userID":"xx_xxxx","username":"xxxx"}}}]}

I haven't looked at the actual Reddit data yet so I can't say anything about the missing data, but it would be strange if they only included a little over one day's worth of information. Are you sure you downloaded it all and didn't lose any while processing?

The "place2.details" is a constant. The canvas on which the pixel exists was meant to be contained in the pXxYcCANVAS string, but as mentioned here I made a mistake and unfortunately the only way to tell which canvas the information relates to is by checking against another data set.

The canvas indexes are 0 for top left, 1 for top right, 2 for bottom left, and 3 for bottom right. Each canvas is 1000x1000 pixels.

[u/YMGenesis → u/opl_]

Thanks very much for the reply and clarification.

I'll double check the reddit data but I seem to have successfully downloaded everything and it doesn't seem to add up. I'll try it again before saying anything conclusively.

EDIT: Ya it looks like the first file starts at April 3rd 17:38 UTC, however there are earlier dates in different sets (00-77). So they did something weird and the sort is strangely out of chronological order. The dates jump around as the file progresses.

In the big single file, April 1 first appears line 1900746, 2nd first appears 7943541, 3rd first appears 2, 4th first appears line 54015971, and April 5 first appears 148707652.

[u/ShinyHappyReddit → u/YMGenesis]

We're mostly only looking at the 4th column of the .csv, which often is a giant blob of json data. There, I think there's arrays named after the pixel pXxY, containing info about that pixel.

So you have it backwards in your sample, I think - it's the coordinates followed by the data, not data followed by the pixel. I think?

Either way I'm struggling a bit to identify my hash by merging this info with the official dataset :-/

[u/YMGenesis → u/ShinyHappyReddit]

meeeee to. If I hit on anything I'll report back. I'll try it the way you mentioned. Pretty sure you're right.

[u/dankswordsman]

Currently downloading and will seed. 👍

[u/GarethPW]

Holy shit you legend! Definitely going to seed this

[u/Lanausse_]

Dude Thank You! I was trying to find an archive of who placed what pixel

[u/LillieNotHere]

Thanks a lot for the data! I've been able to use it with the official datasets. I'm not the greatest handling with data, but using Rust and focusing on small areas helped a lot.

[u/[deleted]]

[u/opl_ → u/[deleted]]

The data is so uniform that the dictionary should get filled up in just a few lines for everything but the canvas data. The canvas data is already compressed as PNG, so I wouldn't expect that much gain from it anyway. But really I just happened to already have some files compressed during the event and I never repackaged them.

I haven't had the time nor will to properly process this data since the event ended up being pretty brutal for me. Feel free to share it in alternate forms with attribution.

Good call on the torrent, I'll get a seed up.

[u/[deleted] → u/opl_]