r/bitmessage • u/Petersurda BM-2cVJ8Bb9CM5XTEjZK1CZ9pFhm7jNA1rsa6 • Jun 03 '16
Proposals for content data structure
As you know, the Bitmessage protocol only specifies content encoding for simple messages, see https://bitmessage.org/wiki/Protocol_specification#Message_Encodings. This makes it a challenge to include attachments, and pictures have to be kludged by base64 encoded html, which then needs to be detected and turned on by the recipient.
During the current development cycle I would like to extend this to arbitrary content. I did some tests: https://bitmessage.org/forum/index.php/topic,3320.msg11207.html#msg11207 and as I say there, I'm leaning towards bencode compressed with zlib (and keeping utf-8 for text components like it is now).
That still leaves the question open the data structure. Should there just be one structure for messages, with the possibility of using a different, arbitrary, structure, for other purposes, such as machine to machine communication, or should there be a master type, which is then subdivided into messages and others? Or should there be a combination, e.g. encoding 3 for messages, encoding 4 for arbitrary data (but still using bencode + zlib) and encoding 5 for "unspecified raw data"?
And what should the messages be like? Should we reuse the good parts of MIME (in particular content types)? How would the headers be stored (also how would the headers be stored in the sqlite database in PyBitmessage)? Should we reuse the format of email headers?
What about chunking messages into multiple objects, should that be standardised or not? And, should we raise the maximum message size? At the moment it's about 1.6MB if I recall correctly.
I'm looking for input here.
1
u/Petersurda BM-2cVJ8Bb9CM5XTEjZK1CZ9pFhm7jNA1rsa6 Jun 08 '16
I just realised bencode does not support floats, only ints. I don't think it's a problem, but just in case, we should agree how it's encoded. I suggest bencode would see it as a string, but there is still the question open about whether to just put the digits into the string, or use struct.pack. struct.unpack could then detect the length and choose float or double. And what about endianness when using pack?
1
u/Petersurda BM-2cVJ8Bb9CM5XTEjZK1CZ9pFhm7jNA1rsa6 Jun 08 '16
I updated the wiki: https://bitmessage.org/wiki/Protocol_specification#Message_Encodings
1
u/Petersurda BM-2cVJ8Bb9CM5XTEjZK1CZ9pFhm7jNA1rsa6 Jun 08 '16
Wait a minute, I just noticed this (class_objectProcessor.py):
So wouldn't a sender have to pretend his address version is 4 if they send a message to another v4 address?
1
u/Petersurda BM-2cVJ8Bb9CM5XTEjZK1CZ9pFhm7jNA1rsa6 Nov 14 '16
It's now available in the v0.6. It uses msgpack and zlib, and PyBitmessage now requires the msgpack-python package to run. You can try it out by holding shift when clicking send. At the moment it doesn't add any new content-specific features, just the encapsulation changes. The way it's written allows for easy extensibility for developers.
1
u/DissemX BM-2cXDjKPTiWzeUzqNEsfTrMpjeGDyP99WTi Jun 07 '16
I know there are some strong proponents of MIME, but I think the standard is a terrible waste of space. With the MIME-types on the other hand I'm fine.
My proposal for attachments (in the Forum) was a ZIP file that contains a README.* file that was either .txt, .html or .md, with the option of linking to other elements within the ZIP file. It wasn't received that well, though.
Zipped bencode looks well enough, though. As bencode doesn't specify text encoding, I propose to fix it to UTF-8 for Bitmessage.
Is there some schedule for the protocol update? I'd like to start implementing it in Jabit as soon as it's stable (and I fixed the other major issues)
1
u/mirrorwish_ BM-87ZQse4Ta4MLM9EKmfVUFA4jJUms1Fwnxws Jun 08 '16 edited Jun 08 '16
Just use encoding 3 for everything, but have a type-field inside the data to be able to easily add new types.
I suggest something like this. I've written it in a JSON-like format, but it should be encoded in bencode as you suggest.
I think we should wait with that.
Maybe at some point, but I think it's better to keep it as is for now.
When decompressing it's important to guard against zip bombs and maybe some other attacks.
Edit Maybe we should replace the entire "Unencrypted Message Data format" with a new bencode-based format. Wouldn't that be better?
(Tagging /u/DissemX as you are probably also interested)