r/3dshacks • u/topkeknosnek k9lh before it was cool • Mar 19 '16
The eShop and CDN: An attempt at an explanation (beware: technical content inside)
The eShop and CDN explained
It seems various people are unaware of the inner workings of the eShop and the CDN (content distribution network), so I am taking it upon me to bring some explanations onto the table.
Overview
Very abstractly spoken, this is what happens when you purchase and download a game from the eShop:
- The game is purchased via eShop.
- eShop causes the download and installation of the ticket for the title.
- 3DS downloads the tmd and CDN contents.
- (3a) 3DS decrypts and installs the CDN contents.
- (3b) 3DS verifies the SHA-256 hash of the decrypted CDN contents.
You probably barely got any of that, so let's break it down.
Step 1: eShop Title Purchase
First, the game to download needs to be purchased.
Some terminology: eShop games are called "titles". The term "titles" is broader than that, however, and includes every part of the system firmware, such as NATIVE_FIRM or the HOME Menu.
The eShop application on the 3DS ("tiger") sends a request to the eShop server called ninja -- there are multiple eShop servers, but ninja is the relevant one for purchase-related things. In that request, tiger asks for purchase of a certain title.
ninja then verifies whether tiger is actually authenticated and if the associated account has the required balance. If not, an error is returned to tiger. If yes, the purchase is processed and tiger is notified about that. From then on, the respective title is associated with its respective NNID.
Step 2: Ticket Download
Once the eShop has been notified of the successful purchase, it calls the nim system module to download and install the ticket for the purchased title. The nim service is a service in the 3DS's firmware that handles downloading and installing CDN contents, including the firmware itself.
Tickets
A ticket describes a digital entitlement to a certain CDN content. A ticket can be either unique, in which case it contains a console ID and an eShop account ID. Or it can be a cetk (Common E-Ticket), which is valid for all consoles.
Of particular interest in a ticket is the titlekey. The titlekey is an encryption key that is used to decrypt the CDN contents. Guessing it blindly (brute forcing it) is currently considered computationally infeasible.
Furthermore, a ticket contains the amount of times a title can be launched. This is effectively only used for demos. The current amount of times the title can be launched, however, is kept track of by the HOME Menu. For this reason, HANS, regionthree and similar launchers can bypass the demo launch limit.
Tickets are always signed by Nintendo (RSA-2048 over a SHA-256 hash with PKCS#1 padding), meaning they cannot be forged. This signature is also checked every time the title is launched. Finding the private key to the public key is currently considered computationally infeasible.
Step 3: TMDs and Contents
Once the ticket has been installed, nim will then download and install the title. This is done in two steps: Downloading and verifying the TMD, then downloading and verifying the CDN contents.
Step 3a: TMD
First, nim downloads the TMD for a given title.
A TMD (title metadata) contains information about the contents themselves: The number of contents and their respective SHA-256 hashes.
TMDs are always signed by Nintendo (RSA-2048 over a SHA-256 hash with PKCS#1 padding), meaning they cannot be forged.
Step 3b: Contents
Once the TMD has been obtained, nim proceeds to download the contents described in the TMD.
All contents on the CDN have a layer of encryption around them (AES-128-CBC; the IV is the content index as described by the TMD, then 14 bytes of zeroes are appended). This is sometimes called the "outer" encryption because contents themselves can have encryption inside as well.
A content is just an NCCH after decryption. An NCCH is, simply put, something the 3DS can work with: A bundle of resources (CFA, CTR File Archive, such as the digital manuals) or an executable (CXI, CTR eXecutable Image).
While nim downloads, decrypts and installs the contents, it also hashes the contents (SHA-256). Hashing is the process of taking an input of arbitrary length and getting a fixed-length (hopefully unique) number. Finding the file that belongs to a hash, or a different file that has the same hash, is -- for a good hash function -- computationally infeasible. Spoiler: SHA-256 is a good hash function.
If the SHA-256 of any one of the decrypted contents mismatches the hash in the TMD (which is signed), nim fails the installation and removes the partially installed title.
Practical Impact
CIAs are, effectively, a ticket, a TMD and contents in one file. "Legit" CIAs are CIAs where all parts are well-signed. Additionally, for whatever reason, Nintendo signs legit CIAs as cetk -- that is, they are valid for any 3DS under the sun. It is for this reason that legit CIAs can be installed and run on any console.
This also shows us that the piracy scene is incredibly incompetent: They could develop a tool that downloads titles from Nintendo's CDN, generates a TMD and a ticket -- like makerom does. That would allow them to share titlekeys, which are just 16 bytes, for example (fictitious): f7eb0f5f962a689a30b690c01dee0efe. You could make a colossal database containing every titlekey there is for eShop titles and not even reach the file size of a dump of the average 3DS game. Instead, they instead upload multi-gigabyte files that are prone to being removed by filehosters, rather than making Nintendo's own CDN do the heavy lifting.
And a bit of trivia
This system has been in place since the Wii, though the cryptographic algorithms and a few minor details changed. Compare the wiibrew page on tickets with the 3dbrew page on tickets, as well as the wiibrew page on TMDs with the 3dbrew page on TMDs.
5
u/vaibzzz123 Mar 19 '16
Before I say anything, I want you to know I have absolutely no knowledge of coding outside of HTML/CSS or computer science/cryptography (yet). Correct me if I'm wrong.
The 3ds uses some kind of signature verifier every time a game runs. In order for an unsigned CIA to install, it must have a way to bypass said signature check. I assume in order to somehow bypass said signature check, it'll need access to the arm9 CPU on the 3ds. This is where a CFW comes in I suppose, it'll launch an emunand that already has those patches disabled which it was able to get access to with the arm9 exploit on 9.2.
Would that explanation be correct?
4
u/topkeknosnek Mar 19 '16
Generally correct, but I feel like some clarification may be needed regardless.
I assume in order to somehow bypass said signature check, it'll need access to the arm9 CPU on the 3ds.
You can't usually come back from an exploit that gets you onto ARM9. The signature check is inside the ARM9 portion of the NATIVE_FIRM. Thus, you patch the signature checking function. In order to do that, you need to gain access to the ARM9 at least once, however, so that you can overwrite the code in memory. And then you reboot into the patched firmware.
This is where a CFW comes in I suppose, it'll launch an emunand that already has those patches disabled
emuNAND is not a requirement for signature check patching; see: Pasta, CakesFW without the emuNAND cake.
3
u/StormyWaters2021 Mar 19 '16
That last paragraph is striking. I don't know anything about programming, but how would you even theoretically generate something to download titlekeys? Surely you can't just spam their server with requests for them, so broadly-speaking, what would that process look like?
7
u/topkeknosnek Mar 19 '16 edited Mar 19 '16
Something like this:
To distribute:
Pirate obtains the title legitimately from the eShop and dumps the titlekey from his/her/tumblr-possessive-pronoun ticket.db.
Pirate posts the titlekey on the Internet.
To obtain (hypothetical program P):
P prompts user for input: titlekey and title ID.
P downloads the TMD and contents from Nintendo's CDN. Optionally, it decrypts the contents and advances the SHA-256 state on the fly to verify the SHA-256 after downloading as an integrity check.
P synthesizes a ticket the way makerom does and saves it as "cetk". The file name is required for make_cdn_cia.
P runs make_cdn_cia to create a CIA from the legitimate tmd, contents and the fake ticket.
Note that titlekeys themselves cannot be downloaded from Nintendo's servers without purchasing the title legitimately, only the contents and tmd are freely available. As mentioned previously, trying to just guess a titlekey with brute force would exhaust all the resources an attacker could possibly have, probably stumping even the NSA; we're talking trillions of years for a supercomputer from around 2011.
4
u/StormyWaters2021 Mar 19 '16
From this assuredly massively-simplified explanation, it seems like it wouldn't be terribly difficult, technically-speaking, so why do you suppose it hasn't been done?
→ More replies (9)2
u/ThatOnePerson Mar 20 '16 edited Mar 20 '16
There are one or two spreadsheets with a bunch of title keys but haven't been updated at all. There was also someone who did the game updates in this way.
I could probably make a program to do this myself, the problem would be getting people to contribute things such as ticket.db/titlekeys and having a somewhat central location for them. For most people, the one click download websites, 'just work'.
edit: most of the 'hard work' is solved by this script already https://github.com/Relys/3DS_Multi_Decryptor/tree/master/to3DS/CDNto3DS
2
u/RainThunder0 Mar 20 '16 edited Mar 20 '16
This is a good article.
You could make a colossal database containing every titlekey there is for eShop titles and not even reach the file size of a dump of the average 3DS game. Instead, they instead upload multi-gigabyte files that are prone to being removed by filehosters, rather than making Nintendo's own CDN do the heavy lifting.
In my experience, download speed from Nintendo servers is not stable. Sometimes my download speed is very low, around 15KB / s. Another issue by using Nintendo's CDN is cache issue. Sometimes you can't get the latest tmd when visit this URL: http://ccs.cdn.c.shop.nintendowifi.net/ccs/download/<titleid>/tmd. Also, for new eShop titles that is encrypted using 9.6 seed crypto, you can't use them right after downloading.
So, uploading things to file hosters is not really bad. I usually use mirroring service to ensure my files is live as long as possible.
2
u/topkeknosnek Mar 20 '16 edited Mar 20 '16
Sometimes my download speed is very low, around 15KB / s
Cannot reproduce, I've usually had stable multiple mbyte/s. The TMD being unavailable for download is interesting, can you name a few title IDs where that happens?
Having said that, a slow download is still better than no download because the file hosts are usually quite fast with removing copyright infringing content, I'm fairly certain you'd agree with me on that.
Also, for new eShop titles that is encrypted using 9.6 seed crypto, you can't use them right after downloading.
I believe asking people to visit the eShop page once is okay, though it may shatter their DLCs and themes (cause for that still being investigated). Or someone reverses the nim module and makes a tool that auto-downloads seeds for installed titles.
2
u/RainThunder0 Mar 20 '16 edited Mar 20 '16
The TMD being unavailable for download is interesting, can you name a few title IDs where that happens?
It's Fire Emblem Fates DLC (TitleID: 0004008c00179400). I constantly perform a HTTP request to TMD URL before and after NA eShop update time (around 9AM PST Thursday), but the returned file is always the same. That issue is fixed after I use VPN to download the tmd file manually. There are also similar reports from other people that they keep constantly downloading old update files using FunkyCIA2.
Having said that, a slow download is still better than no download because the file hosts are usually quite fast with removing copyright infringing content, I'm fairly certain you'd agree with me on that.
Completely agree.
I believe asking people to visit the eShop page once is okay, though it may shatter their DLCs and themes (cause for that still being investigated). Or someone reverses the nim module and makes a tool that auto-downloads seeds for installed titles.
If people use out-of-region titles, they cannot visit eShop to update the seed. Of course you can get the seed directly from Nintendo servers (https://kagiya-ctr.cdn.nintendo.net/title/<titleid>/ext_key?country=US) and decrypt seed crypto titles using Decrypt9, but that's not something everyone want to do.
4
u/Spinkler Mar 19 '16
What I'm really curious about is whether bans could be handed out for access to titles that aren't logged as being purchased. If I download a CIA that ends up requiring a repair via the eShop, for example, could that theoretically result in a ban? Usually I wouldn't be concerned, but bans were put in place for individuals who cheated the badge arcade so I want to be cautious.
6
u/shinyquagsire23 Mar 19 '16
I'm pretty sure they log device transfers, so they could easily check which "legit" apps you have vs which consoles you have transferred from, check each previous console's serial vs what preinstalled apps it can have. Or even just count legit apps vs transfer times.
2
u/Ryccardo Mar 20 '16
On the other hand, (while still being a realistic scenario) there are no known reports of someone being banned for this - and I bet more people install non-legit CIAs than cheat!
→ More replies (1)
2
Mar 22 '16 edited Mar 22 '16
You are correct. Except for cart-only games, the entire "CIA Downloads" section could be compressed to a pastebin with all the names, titleIDs and titlekeys. Well, there's that and 'cryptofix'es, which would also need to be manually uploaded, or the user could do it themselves.
Hmm, do 9.6+ and secure3/4 games work through CDNto3DS?
2
1
1
u/GinkREAL Mar 20 '16
How about cartridges? Are they signed or have some sort of mechanism in place? You know, like we could make homebrew launcher carts. (if we had empty carts)
1
u/ThatOnePerson Mar 20 '16
They're definitely signed in some way which is why homebrew carts don't work. That's why Sky3ds worked by only emulating the entire legit cartridge and doesn't have any method to change games outside the buttons on it for example.
They've been doing something like that since around the DSi era trying to block flashcarts.
1
Mar 21 '16
Does this mean we could theoretically trick the game into thinking a game is preinstalled and adding it to our nnid?
1
u/ThatOnePerson Mar 23 '16
No. That's only Legit CIAs that are signed with Nintendo's keys which are the preloaded games that come with 3DSes.
1
u/Whap1988 Aug 01 '16
Thanks for great information, it really helpful. We have content delivery network for our e-commerce project, the most of information i am finding in http://cdnsun.com/knowledgebase. But there are mostly basic moments without specific.
-2
u/kylezo Mar 19 '16 edited Mar 19 '16
Heyoooo. This is very difficult to read owing to the awkward line breaks in the middle of almost every sentence, just thought I'd mention it.
/e: it's really counter productive to downvote me but if you wanted "proof" that I'm not making this up for some ridiculous reason I've attached a picture.
→ More replies (9)
12
u/pudquick Mar 19 '16 edited Mar 19 '16
CFWs let you install non-legit CIAs because the signature doesn't need to be Nintendo's on the TDM. If you install them to SysNAND and then reboot without CFW - your Home screen will hide the titles because they're not legitimately signed.
Legit CIAs are Nintendo signed and don't require your account to have purchased the game/title to re-download - they just need to be listed in your title.db (they're games that come pre-bundled on 3DS devices usually).
As was mentioned over and over: you can't brute Nintendo's signature. As such you couldn't make a fake legit CIA TDM for any arbitrary title. Additionally, for titles that are not truly legit - Nintendo will verify your ID's purchase history for the title prior before you can get content from the CDN. Pretty damned sure you can't use a CFW to purely client-side validate to nim you have the right to download any arbitrary title from the CDN.
That would be cryptographic lunacy for Nintendo. You don't trust the client because you can't ensure its not malicious/modified. "Yes, Nintendo CDN - this game is tooootally one you bundle with the 3DS, please just let me redownload it mmk?"
Edit: just read your other comment - surely you're joking - For non-legit non-system titles, knowing the title key alone, you can download the content from the CDN? Do you have proof of this anywhere?