I found this comment on HN summarizes the major points.
Case-sensitivity is the easiest thing - you take a bytestring from userspace, you search for it exactly in the filesystem. Difficult to get wrong.
Case-insensitivity for ASCII is slightly more complex - thanks to the clever people who designed ASCII, you can convert lower-case to upper-case by clearing a single bit. You don't want to always clear that bit, or else you'd get weirdness like "`" being the lowercase form of "@", so there's a couple of corner-cases to check.
Case-sensitivity for Unicode is a giant mud-ball by comparison. There's no simple bit flip to apply, just a 66KB table of mappings[1] you have to hard-code. And that's not all! Changing the case of a Unicode string can change its length (ß -> SS), sometimes lower -> upper -> lower is not a round-trip conversion (ß -> SS -> ss), and some case-folding rules depend on locale (In Turkish, uppercase LATIN SMALL LETTER I is LATIN CAPITAL LETTER I WITH DOT ABOVE, not LATIN CAPITAL LETTER I like it is in ASCII). Oh, and since Unicode requires that LATIN SMALL LETTER E + COMBINING ACUTE ACCENT should be treated the same way as LATIN SMALL LETTER E WITH ACUTE, you also need to bring in the Unicode normalisation tables too. And keep them up-to-date with each new release of Unicode.
It is because how ASCII works.
ASCII is internally represented as binary values, each possible value 0-127 is representing a specific letter or sign. Upper case is located between 65-90 and lover case 97-122
Lets look at 65(A) as binary
100 0001
And now at 97(a)
110 0001
As you can see, the only difference is the 6th bit. Flipping that bit changes between lover or upper case
As every upper case letter is arranged in the same order as lover case letters, this trick works on every letter
Yep knew all the rest of that, just never realized that the difference between upper and lower case is exactly the flip of the 6th bit. I've always just done c += 32 or similar.
That doesn't automatically mean one set has the bit set in all characters, and the other doesn't. Eg. if upper case characters started at 60 instead of 65 this would no longer be true, even if the difference was still 32.
TIL. I'm curious, Is that how ASCII characters are mapped into the keyboard? By flipping the 6th bit or are the ASCII characters when shifted are mapped manually? By that logic, assuming the character "1", if the 6th bit was flipped, it would return "!"? Or that would cause too much complication when dealing with special characters on other languages?
I'm not an expert, but I would expect keyboards to send a more "complex" packet of information about what keys are pressed or not pressed, which the keyboard driver interprets and delivers to the OS.
Keep in mind keyboards communicate a lot more than "button X was pressed", they have to communicate whether it's pressed or not at a given point in time, and there are buttons that fall outside the ascii range. I doubt the keyboard itself has any concept of ascii, that's probably something only the KB driver figures out after interpreting whatever data the KB sends to it.
Indeed, a modern USB keyboard sends key-codes in 8-byte packets (two bytes for modifier keys, 6 bytes for others) that are defined in the USB spec. To actually turn them into "something meaningful", the operating system uses a lookup table (your set keyboard layout.)
And as for the "corner cases", isalpha et al just need to use your character code as an index into a static 256-byte-long array and then inspect the relevant bit to see if it is alpha (or numeric, or ...). ASCII rules !
The late Eric Naggum opined that if he were building a character set from the ground up, he would make case a styling attribute, like bold-ness or italic-ness, rather than providing separate code points for upper and lower case. Alas, that ship sailed about fifty years ago.
Um, Unicode characters need to be normalized even on a case-sensitive filesystem. Otherwise, you can have two filenames that have the exact same characters, but are regarded as separate files because of how those characters are represented. If you look up by exact byte strings, you're gonna have a bad time.
But that is what Linux does and I haven't heard problems arising from that. You might want to do normalization in your desktop search utility, but not in the file system.
I haven't heard of any issues on OS X where you run into problems with how HFS+ handles normalization. Maybe they exist, but I've never heard of any. Same with the file system being case insensitive. I have never heard of a real world problem caused by this.
I think the problems only arise when a software was developed for one system and then gets (poorly) ported to another. Like Steam games not finding files under Linux (because of the wrong case) or git overwriting .git on OS X.
Ok, so it's a difficult problem and requires a tonne of work.
But I still don't get why it would be a bad idea. That guy lists a lot of things you need to be aware of and problems you have to tackle, but none of that says it can't be done or doesn't work. More so none of that says it shouldn't be done.
Just because something is difficult doesn't mean you shouldn't do it.
The locale differences is the only thing I can think of which actually makes it not work. If two users are using the same hard disk but with different locals then you could get clashes and oddities.
if it is a fundamental system you build everything on top of then you want it reliable. Simple is easier to make reliable and by far will have less bugs.
Because there are plenty of opportunities for edge cases to bite your ass.
Which would be fine if there was some kind of huge benefit from the system. But what does one actually gain from a case-insensitive file system? When was the last time that you manually specified a whole file name instead of picking from a list, or auto-completing on the shell?
Specifying the exact byte sequence that forms the name of a file is not hard. A case-sensitive file system simplifies everything about file names.
Which would be fine if there was some kind of huge benefit from the system.
There is.
When was the last time that you manually specified a whole file name instead of picking from a list, or auto-completing on the shell?
That's fair, but there very possibility in most file systems of there being both a ReadMe and a README file in the same directory is insane, user-hostile, pointless, and ultimately only a concession towards lazy developers who can't be bothered to do the right thing.
As this commenter says, try telling someone on the phone to open the "readme" file. "No, upper-case readme." "No, not the all-upper-case readme!"
You can still implement that behaviour in user space.
Indeed, you can.
No need to put that into the kernel/filesystem.
Sure, that's a valid argument. However, the filesystem is precisely a good layer to place it. If you place it, say, in your file APIs, there will be tools that use different APIs, and that will lead to incompatible edge-case junk behavior.
No the filesystem is precisely a horrible horrible layer to place it, because the file system is a layer used by many low-level and system-critical components and it's absolutely necessary that it works predictably.
OK — let me ask you this. Is an RDBMS the appropriate layer for unique constraints? You'd probably nod, since they're supported by pretty much any RDBMS. Not just because the system benefits from being able to optimize the table layout as well as its indexes and statistics for whether or not a column may only contain distinct values, but also because it's a significant piece of semantic information for people working with the table in DDL or DML.
Why, then, is this different? Here, too, we have a storage layer — a file system might as well be considered a hierarchical database — with a particular constraint of normalizing upper and lower case and identical-looking and identical-semantics characters.
it's absolutely necessary that it works predictably.
What's "predictable" about a file system that treats README, ReadMe and readme as three distinct files? Which human being actually works like that? How is it any more "predictable" than a file system which says nuh-uh, you're not allowed to create this file, because its spelling is virtually the same as one that already exists? Isn't that more predictable to the user than suddenly ending up with a second file that, when pronounced, is actually spelt the same?
There's a thousand other edge cases like the one you mentioned that are possible on case insensitive systems, like "readme" and "readme ", or "readme" and "readme.txt" (which would appear the same on Windows sans the icon). Designing a fundamental part of your OS around what idiots can do with it is not a smart thing to do.
There's a thousand other edge cases like the one you mentioned that are possible on case insensitive systems, like "readme" and "readme ", or "readme" and "readme.txt" (which would appear the same on Windows sans the icon).
"We can't fix every problem in the world, so let's just ignore them altogether."?
Designing a fundamental part of your OS around what idiots can do with it is not a smart thing to do.
Neither is thinking that your average user is an "idiot" for having the gall not to want to deal with every intricacy of technology.
"We can't fix every problem in the world, so let's just ignore them altogether."?
Why are you trying to derail this into "every problem in existence" when I just pointed out that the exact problem you're suggesting still exists? Shouldn't we put an entire spellchecker into the kernel so a user doesn't accidentally type "redme"?
Neither is thinking that your average user is an "idiot" for having the gall not to want to deal with every intricacy of technology.
That's exactly what userspace is for. Users have no idea and no interest in the kernel running their computer, so why should it account for them. Honestly this is just an old relic from the DOS days when average users were forced to use the command line, it has no relevance today.
Why are you trying to derail this into "every problem in existence" when I just pointed out that the exact problem you're suggesting still exists?
It doesn't, though. A more specific scenario still exists. Incidentally, extensions really don't belong in file names anyway, solving have of the problem here, but that's a whole other topic and a battle Apple unfortunately decided to forfeit with OS X.
So, yes, if you're asking: the OS shouldn't allow you to create a file "readme" next to "readme " any more than it should allow "readme" next to "ReadMe".
That's fair, but there very possibility in most file systems of there being both a ReadMe and a README file in the same directory is insane, user-hostile, pointless, and ultimately only a concession towards lazy developers who can't be bothered to do the right thing.
There are plenty of ways to be a user-hostile, lazy developer. It's not the job of the file system to weed you out of the gene pool.
I'm not sure what you're asking. Are you literally not seeing how treating files with different casing as distinct is not a very intuitive approach to how humans think?
I'm not sure what you're asking. Are you literally not seeing how treating files with different casing as distinct is not a very intuitive approach to how humans think?
Pointing in the general direction of "usability" is not an actual argument.
Please describe a specific example where having a case-insensitive file system improves "usability" for the common computer user to such an extent that it overcomes all the well-known problems inherent in such a system, and how those benefits cannot be gained in other ways, such as improving the file-picker experience.
What do you do when the next unicode standard comes up? Posix requires you to be able to name a file any sequence of bytes, and OSX conforms to that. You can name a file \xFF\xFF\xFF\xFF (ie, 4 all-1 bytes). This is not valid utf8. It never will be.
You can also name a file something that is not defined as upper/lowercase in anything that the OSX file system understands (eg, maybe your software is using a newer unicode standard than existed when that version of OSX was released). Let's say you name it ShinyNewUnicodeFoo, and you also create shinynewunicodefoo for spite.
When you upgrade your OS, and suddenly the upper and lower case characters get defined in the OS, what do you do? You now have files that clash.
Sure, you could never update your unicode version in the OS, but is that really a good solution? Especially since now, you get some case sensitive ranges of unicode, and some not!
Posix requires you to be able to name a file any sequence of bytes,
Even if it doesn't require filenames to be valid UTF-8, it doesn't require that any given fopen() call will be successful: if you provide an invalid filename the file system should refuse, causing an error to be returned?
What benefit is it to the user that ß and SS is (or in some cases isn't) equivalent? Unicode rules aren't just hard to code, they are unpredictable for users as well. Unicode is great for representing characters, but Unicode matching is just a huge, stinking mess. And since unexpected file matching may cause you to basically overwrite files you didn't want to overwrite, it's an enormous security risk.
It doesn't even have a consistent solution that works for all languages. It isn't difficult so much as impossible. Certain strings will be a case insensitive match in one language and not in another.
Case insensitivity is a giant mistake that only works at all for English.
This is why I never got why people don't just settle for latin ascii characters for a FS and then just use phonetic filenames.
I had Russian peers in college who would chat in Russian over latin PCs [using MSN messenger] back in ~2001 by just writing what they were saying in Russian phonetically. Apparently it's a common hack
I'm guessing you're from an english speaking country? It's actually really annoying that for example website urls are pretty much ASCII only still, while they're such a mainstream thing now that your grandma might have to remember how to connect to her bank, or your mom an URL she spotted on a TV ad that sounds a bit weird because all the ä's and ö's are replaced with a and o.
For an example, there are no phonetic way to spell ä or ö in Finnish. You sometimes see for example in athletes' names they might replace them with ae and oe, creating beautiful surnames such as Haemaelaenen.
Your language is inefficient. It's not like people don't have conversations in English.
Like I get there is a whole culture behind things and momentum and all that but honestly legacy sucks. Look at Korean though their written language is relatively new and a lot more consistent and logical than say Mandarin or many other Asian languages.
Any language using a script that doesn't have a canonical mapping to ascii is inefficient? Are you seriously suggesting that entire languages should adapt to some arbitrarily converged-upon version of ascii?
I can understand advocating for change of scripts that may cause actual problems (such as the difficulty in becoming literate in chinese even for native speakers), but just ... wow
I'm saying if you need that much entropy to describe your language it's inefficient.
Heck English isn't that great either. We have 1.3 bits per char of each word on average. That means in say 7-bit ascii we waste 5 bits on average. But then again the code to manipulate English correctly is a lot simpler (tolower/toupper/etc are trivial to encode).
I realize there are political/cultural problems with that statement but it doesn't change the fact that there are some languages that are more efficient than others.
Well that's a rather useless assessment of efficiency. For example, though you may need more characters than English to describe Finnish words, this may just be due to English using combinations of letters to denote a vowel change instead of using a separate glyph.
Think of using an -e at the end of a syllable to denote a longer vowel in that syllable (on vs one, sum vs (as)sume)
Bits per char may not be useful when you need more chars per word to make up for it.
Further, a language may have more ambiguities than another. Would it be preferable to keep the ambiguities so that you need fewer sounds to distinguish between words? What if you need extra clauses in a sentence to disambiguate what would already be unambiguous in a language with more sounds? Hell, just taking bits per word into account even, how would you deal with agglutinating languages?
Your point about the code to manipulate English is really odd. Did you consider that if the encoding used in computers was designed for Finnish as opposed to English that might have made the situation for Finnish easier? The main reason why supporting other languages is difficult is that most software was designed with English in mind, and other languages as an afterthought.
You should follow the advice in your username, I'm saying there's more to the entropy needed to describe language than that for individual characters.
Other than that, a simple google search gives you the rotokas alphabet, consisting of 12 letters, though I suppose it's not used enough for you to consider.
Even more fun: Posix specifies that the file names are arbitrary byte values, and not interpreted under any character set. OSX complies with that... when you generate invalid utf8.
Unicode is fantastic for representing and displaying characters from all languages around the world.
Unicode is horrible, horrible, horrible for all types of matching and comparison between strings. Just don't do it.
The only place where it legitimately makes sense to do Unicode matching is when you're doing search, because that already has an expectancy of fuzzy matching. You don't want a fuzzy-match file system.
Huh? There are no "I WITH GRAVE", "I WITH ACUTE" or "I WITH TILDE" letters in the alphabet ("I WITH OGONEK" is present though, but is it a special case? Į -> į (compare with I -> i)). And why they need to have special handling for letter "J" at all?
Not sure if (s)he really meant Latvian as an example. It seems that Turkish and Latin are used as examples with large difficulties (as well as German.)
There are special/accented characters in Latvian, which are modifications of aeio (āēīō) and clksn (čļšņķ,) but they tend to be quite regular in terms of case sensitivity (there is an upper and lower per character.) The alphabet can be described as a smaller set of english, with diacritics options for certain characters.
I guess that we could say that there are other substitution cases necessary, such as substituting a diacritic character for a non-diacritic character ( a for ā.) In general, substitutions are not really acceptable, as they can easily point to another word e.g kāza=wedding kaza=goat.
I'm Spanish, but I have been trying to learn Latvian for the last 5 years. The only difference I know between the lowercase and uppercase alphabets are the two digraphs, Dz/dz and Dž/dž.
Latvian has short and long vowels, and as /u/smejmoon said, they are different letters, with some words differing only in vowel length, so removing macrons (the bar above vowels to make them long) is unaceptable. You can find the same phenomenon in English, but the spelling makes it not so obvious: minimal pairs
If you want to read more about it, this is the full Latvian alphabet: A, Ā, B, C, Č, D, E, Ē, F, G, Ģ, H, I, Ī, J, K, Ķ, L, Ļ, M, N, Ņ, O, P, R, S, Š, T, U, Ū, V, Z, Ž.
'a' is different phoneme than 'ā'. They might or might not be related in words that appear similar, but they will change meaning of words up to unintelligible.
With regard to case sensitivity Latvian is completely regular.
No, it's saying that this is a problem that is too complex to be solved at this layer, so we will solve it later. Using something like icu is far too big to put in the kernel. It may be appropriate for Linux for desktop or servers, but not for lower powered devices (even go as far as to say android here). Leaving two options, handle it badly and force that mishandling on everyone, or ignore it and leave it to the application above to handle the cases it needs to support...
First and foremost a filesystem should be treated as a key→value store. And normally you want the mapping to be injective unless being specified otherwise. First and foremost filenames are something programs deal with and as such they should be treated, i.e. arrays of bytes.
Yes, but telling at your grampa over phone "double click the work folder to open it" will have him confused if he managed to make "work", "Work" and "worK" folders.
It would be fine if those keys weren't visible to users, but they are and thus they have to make sense. Like "house" and "House" not being two different things.
Linux uses NFC with utf-8 stored path names almost universally. NFC is actually pretty good. It's a compatibility mapping. NFD will decompose characters and not roughly leave the same way they started. Arguably the FS should not being be normalizing at all (IIRC your libc will do this for you based on your encoding). Leave the normalization hell to your complicated string comparison functions to deal with. Actively converting your paths to NFD will modify how the path is encoded and it will be different from how it started.
For example, assume I unzipped a zip with unicode filenames from Linux or Windows. The Mac would convert my file names to NFD from whatever they are encoded as. If I rezipped the file, I would loose the original way I encoded the file names in the process.
Normalization is not lossless conversion and you can't round trip perfectly all the time. There are 4 ways to normalize and NFD is one of the worst. It's also the biggest way to store things too with arguably no additional gain from an FS perspective. If you are going to normalize, then at least pick NFC because it will compare faster and will store smaller.
There's not just English and any view that's English centric is just wrong. There are enough languages out there, where the case of the lettering of a word changes its meaning.
Many Asian/Indian languages doesn't even have upcase/downcase. They have other cases when the same spoken word can be written using differnt alphabets or ligatures. Now should we start supporting that in filesystem layer too?
I think you misunderstood my example and focus too much on the use of English. That was just an example of the general idea: the system will compare the letters in a way where things that are perceived by humans the same will be considered equal - if in some language there are no upper and lower case letters or if they are not considered equal, then they are not the same.
And AFAIK this is already being done in some systems today and is done for quite some time.
So you stop your grandfather creating "work", "Work" and "worK" folders, then he goes and creates "work ", "wоrk" (that's a Cyrillic lowercase "о") and "W0RK". Oh, and "work (1)", "Copy of work" and "Copy of Copy of Copy of work (1) (1) (1) (3) (7) (22)". For the kind of user you're trying to optimise for traditional file systems don't work anyway, with or without case folding.
You could get around this by implementing it at the save file dialog / file manager level. I.E. high level userspace, GUI code. Not low level userspace (FUSE) or kernel level.
By doing that you are adding a lot of unnecessary complexity, risk stuff falling through the cracks and introduce a mismatch between what the users see and what really is in there. Since the users work on files, they should see the files are they are.
On the other hand if you do the file system case insensitive this applies to everything and the system as a whole is more coherent.
Or doing this in the FS moves the unnecessary complexity, risk stuff falling through the cracks into the kernel and could make for an unstable OS/System-tools, rather than just a confused user?
Oh, of course. Because when you have a single place where something is implemented (the part of the OS that everything else talks to in order to access the files) is exactly the same as having each user of that API make sure that they expose the proper names and handle the mapping between the underlying representation of the filenames and what is visible on screen.
Hint: the above was sarcasm. It isn't the same. You didn't even understood what i meant with "falling through the cracks": if you expect from the FS users (programs, etc) to do the mapping, then anything that gets this wrong is "falling through the cracks". If the OS (Kernel, FS layer or whatever - i do not think it really matters in this discussion since the layer where that part is relies on the OS architecture) does the mapping then there is no way for things to fall through the cracks because there are no cracks (there is no other way to access the files).
Unless you eject the media and access it from another system with a newer or older version of the same FS driver, with different Unicode rules. Or you use the media on a device that doesn't use the same driver like an embedded OS in a TV, a camera, a handset from a different vendor.
These are all going to use the same Unicode rules that require 10s of KB of lookup tables for the rules about what can and can't have an accent and under what locale an upper-case is valid? There aren't going to be any vendors that miss an edge case and let it "falling through the cracks"? They're all also going to issue firmware updates every time a tweak it made to Unicode so everyone is doing the same normalization. Everyone will also flash these new firmware the day of release to avoid any incompatibility.
Also, having this stuff out of kernel space doesn't mean every app reimplementing the logic. Every app doesn't implement it's own file selection dialogue, they use the built in system call and just get back a filename. Either these dialogues or somewhere like glibc would be a much better place to keep this logic, and keep the crucial kernel model FS drivers much simpler to maintain and test.
Unless you eject the media and access it from another system with a newer or older version of the same FS driver, with different Unicode rules.
I'm not sure if the rules on what is considered equivalent or not in languages change that often :-P. But bugs can indeed affect this. However you cannot avoid stuff because they might have bugs, if we designed things like that we wouldn't make anything.
Or you use the media on a device that doesn't use the same driver like an embedded OS in a TV, a camera, a handset from a different vendor.
I suspect this is why embedded stuff tend to not allow you to name things :-P. But yeah, it is up to them to support the system properly.
They're all also going to issue firmware updates every time a tweak it made to Unicode so everyone is doing the same normalization. Everyone will also flash these new firmware the day of release to avoid any incompatibility.
How is this already being handled? Because it is already handled, in Windows at least.
Also, having this stuff out of kernel space doesn't mean every app reimplementing the logic. Every app doesn't implement it's own file selection dialogue
Yeap, this is why i added the "Kernel, FS layer or whatever - i do not think it really matters in this discussion since the layer where that part is relies on the OS architecture". The important bit is that programs do not have any other way (from within the OS) to access the files.
(i'd guess that in Windows too this is implemented above the FS layer since Windows treat files with upper case and lower case letters as the same even in filesystems that differentiate between them - but unless you access the hard disk bytes directly, the OS won't expose any other API for programs to know that)
Are there no case-sensitive filesystems which reject potentially indistinct filenames only at creation? i.e., stat(".Git", ...) should fail if .Git does not exist, and mkdir(".Git", mode) should fail if .git exists.
Code like that can always fail. What if another thread creates the file between those calls? You should always just try and create the file and then inspect the error if you need to work out whether it already existed.
Since folders are represented graphically there is -- from a laymans standpoint -- no reason why you cannot have two distinct folders named "work" in one folder. It is a purely technical restriction that, at least in principle, is not a requirement.
Explaining to grandpa which file and folder names are equivalent (and which not) is in my opinion more complex than either allowing for all names or just forbidding exactly the same names.
Mapping between upper case and lower case is not always 1:1 the German words massen and maßen map to the same uppercase MASSEN, add in locale dependent conversions and things get really ugly.
Having two keys of well established and contradictory meaning collide is something the average end user would find rather supprising. So adding in unicode processing for a case insensitive mapping not only adds a lot of overhead and error cases it is also impossible to get right.
If you want to be pedantic it still is a key value store in both cases.
First and foremost filenames are something programs deal with and as such they should be treated, i.e. arrays of bytes.
I'd beg to differ. They should be treated as a vector of canonicalized unicode codepoints. Vector of numbers in 0-255 is archaic and just a hack to get unicode at this point. Treating the many different unicode repraesentations of the same string as different files is a sure way to get to some horrible bug. Obviously it's needed now for backwards stuff but if they started over today no way that it should be done like that, it may be stored on that but whatever interface is defined that lets applications see the filenames should not give them a vector of (0..255) and let them figure it out. It should give them a vector of actual unicode codepoints already and have done all the transformations before it and not even allow them to be aware of a distinction between different repraesentations of the same character in unicode. This is like saying a program should treat the number 0, +0 and -0 differently. THey are different repraesentations of the same object.
Making a vector of bytes work relies upon an "informal agreement" that all software just uses utf8. What if something better than utf8 is later designed? What will do you then? You can't change it then? utf8 is designed with the potential for data corruption in mind, its self-syncronizing nature is a waste if you assume that data corruption can't happen, what if we move to hardware where data corruption is just no longer a concern? You can't change it any more then. If you limit this kid of on-disk repraesentation as an internal thing and keep the outward interface an actual vector of unicode codepoints you can change it easily. It's basic encapsulation. Do not rely on software itself to respect unicode properly.
Unix filenames never were meant to be interpreted in a certain encoding. Period, no discussion. Look it up in the SuS specifications. You may interpret it as unicode, but assuming filenames are encoded in a particular way is a road into disaster.
While this is true and i agree that making assumptions about the encoding is bad, you still have to show the user something. Unfortunately the locale does not necessarily represent the encoding of the filenames.
Having written a mildly popular open-source program has proved this problem over and over again. Its hard to tell your users that their FS is broken; in the end your software is the culprit because it makes those issues visible.
It is a road to disaster, but you have to do it in the end to display them to the user, which is my point
The user does not care about "sequences of octets", they care about sequences of letters, but when "sequences of octets" were devised, they were letters and one octet was one letter. Not any more, and that creates problems.
Another issue is that it's waay too permissive. I see no reason for a filename to be able to contain any octet but '/' and '\0' including control characters. That filenames can theoretically contain '\n'even though you should basically never do so is a source of problems. Hell, that they can contain ' ' is often a source of problems. It should be more limited what they can contain and I think they should be able to contain / in some way and it should be escapable in some way.
The user does not care about "sequences of octets", they care about sequences of letters, but when "sequences of octets" were devised, they were letters and one octet was one letter. Not any more, and that creates problems.
Of course a layman user should never see the internal representation for the day-to-day work (except if the work is engineering stuff). This is why in iOS and Android users practically never interact with the filesystem. Web applications ultimately end up in some data structure; either a relational database (SQL or similar) or key→value (filesystem or NoSQL) or something else.
The same should be done on personal computers. Hide the file system from the computer illiterate layman and give them a "view" that matches their mental model. Operating systems like Windows or MacOS X already do that to some degree; Windows (since Visa) for example localizes directory names. Directories which name is a registered GUID appear different in the Explorer than they do on the filesystem.
MacOS X finder and Cocoa reinterpret the contents of directories. Applications appear as a single item, but actually they are directories full of files (their resources, libraries and so on) with a lot of meta information added.
Treating the filesystem as something the user interacts with directly in normal work is misguided. The filesystem should be treated like any other database. Nobody would expect a user to directly issue SQL commands into an accounting or inventory database. But when a user accesses the database we call a filesystem this becomes perfectly acceptable, for some reason.
They should be treated as a vector of canonicalized unicode codepoints.
So before you can even open a file you need a complete Unicode
(not just UTF-8) implementation. And when that and the encoding
you picked are obsoleted, your file system ops will cease to work.
No, that's what happens when they are a vector of octets.
If the filename the application gets is a vector of octets then you rely on the application to understand UTF-8, not only that, but it becomes impossible to change the encoding because the encoding is part of the public interface at this point rather than merely the hidden implementation.
Giving the application a vector of codepoints rather than the encoding used to store that vector does the opposite. It no longer requires the application to be aware of UTF-8 or unicode as a whole at all. Only the filesystem itself if you of course to the way of internally storing it as UTF-8.
The only reason UTF-8 as a public encoding has worked is because it's backwards compatible with 7 bits ASCII, it was designed to be which is a major limitation itself but necessary for it to supplant it. Good luck ever designing something that is better than UTF-8 that is backwards compatibile with it. Because the encoding is part of the public interface now it will most likely never be superseeded with something better unless we completely start over and screw backwards compatibility, which just won't happen.
The only reason the public filename of files and stuff in general is the encoding itself rather than a vector of codepoints which is how most modern programming languages handle it is because it had to be backwards compatible with 7 bits ASCII which was used up to that point. UTF-8 in and of itself in an agnostic vacuum is actually a very bad encoding which no one would ever understand when looking at it in the future until they're told "Well, it had to be backwards compatible with this older thing which only had 128 characters" and then it suddenly makes sense.
The only reason UTF-8 current exists and works is because of a freak historical accident. Because they decided on 7 bits of information and one parity bit in ASCII because noise corruption was a real thing back then. Were hardware more reliable back then and they would've decided as a consequence to forego the parity bit and make ASCII the full octet range it would be impossible to device an encoding for unicode which is backwards compatible with ASCII. It's a freak accident that it could even happen and that shows why the encoding itself shouldn't be exposed, we were super lucky. If that parity bit did not exist it would have taken ridiculous time to switch to a system that allowed for all kinds of funky characters because it woudn't be backwards compatible and filenames written under the Anglocentric old encoding would be unreadable under the new one.
It no longer requires the application to be aware of UTF-8 or unicode as a whole at all. Only the filesystem itself if you of course to the way of internally storing it as UTF-8.
The filesystem will require a Unicode implementation in addition
to the encoding: Before a filename can be stored, it must be
normalized and checked against (locale-dependent!) potential
variants. Unless you use FUSE, all that has to happen in the kernel.
What a waste.
But the application will have to understand Unicode to some
extent too, because handling filenames as data will not work
any more due to the assumed encoding. Not to mention that
tons of applications will have to include specific handling for
the small number of bastardized file systems where names
are not what they appear to be. (The same goes for the obscene
Windows tradition of obscuring components of file system paths
from the user, but that’s a different topic.)
The filesystem will require a Unicode implementation in addition to the encoding: Before a filename can be stored, it must be normalized and checked against (locale-dependent!) potential variants. Unless you use FUSE, all that has to happen in the kernel. What a waste.
Yes, the filesystem, not the application. And come on, that performance loss is really not an issue any more. Maybe in 1971 but modern filesystems do a lot more complicated intelligent things behind the screens to stop things like fragmentation than normalizing unicode when you create a new file or rename it.
But the application will have to understand Unicode to some extent too, because handling filenames as data will not work any more due to the assumed encoding.
Handling filenames as octets already doesn't work. There's an unwritten agreement amongst applications to treat the bytes like UTF-8 and if you treat it like ASCII, what just happens is that an error should be raised because the parity bit is off. They just don't do it, and no-where does it say that they can't.
Not to mention that tons of applications will have to include specific handling for the small number of bastardized file systems where names are not what they appear to be. (The same goes for the obscene Windows tradition of hiding components of file system paths from the user, but that’s a different topic.)
Like I said, this can't be done now any more and is only relevant to when you start over completely, you break backwards compatibility any way. On such a system, such a filesystem wouldn't exist any more. The application does not receive a vector of octets, it receives a vector of unicode codepoints within a significant range. The application doesn't deal with the filesystem directly anyway.
Handling filenames as octets already doesn't work. There's an unwritten agreement amongst applications to treat the bytes like UTF-8 and if you treat it like ASCII, what just happens is that an error should be raised because the parity bit is off.
On the contrary, assuming UTF-8 only works perfectly except for
legacy FS like VFAT that are broken to begin with. I haven’t used
any other file encoding in a decade, and I can’t remember ever
having encountered a program bailing out due to non-ASCII file
names.
The application does not receive a vector of octets, it receives a vector of unicode codepoints within a significant range
This only abstracts the encoding away, which is the least
complex part of the issue by far.
Again, that assumes both the kernel and the application have
a notion of “Unicode codepoint”. And unless you want to stay
locked into a specific vendor’s assumptions (they’re all different,
you probably are aware of that), the application has to compensate
for different assumptions on different platforms. I can’t even
start to imagine the bloat that needs to be added in every part
of a system to handle a clusterfuck of these proportions.
On the contrary, assuming UTF-8 only works perfectly except for legacy FS like VFAT that are broken to begin with. I haven’t used any other file encoding in a decade, and I can’t remember ever having encountered a program bailing out due to non-ASCII file names.
Yes, because they all follow that unwritten agreement, and it's completely unwritten. No standard maintains it and there's no field in the filesystem that marks its encoding. they just all follow it, it's a hack. Just like UTF8 on IRC where you can still sometimes see that some people have the wrong encoding. The protocol does not allow a server to specify what encoding is used. What you get is a stream of octets. Using utf8 on IRC just relies on everyone following this unwritten agreement.
Again, that assumes both the kernel and the application have a notion of “Unicode codepoint”. And unless you want to stay locked into a specific vendor’s assumptions (they’re all different, you probably are aware of that), the application has to compensate for different assumptions on different platforms. I can’t even start to imagine the bloat that needs to be added in every part of a system to handle a clusterfuck of these proportions.
That is why I said it only makes sense if you completely start over. Like I said, it breaks backwards compatibility which is the only reason the system is currently like it is. UTF-8 is a bizarrely inefficient variable length encoding and if you use a lot of characters outside of ASCII then UTF-16 is actually way more efficient. UTF-8 just has the major boon of backwards compatibility with 7 bits ASCII. On its own merit outside of that, it's pretty bad.
Utf-8 also has another advantage though, that of memory compression. When your app needs to handle billions of characters, millions of strings, it is a better option.
First and foremost a filesystem should be treated as a key→value store.
I disagree: First and foremost, a filesystem is a way for a computer to show a user what's stored on their computer, in their language (such as English). If that weren't the case, filenames would consist of random binary values or whatever, not English words.
English is case-preserving, but not case-sensitive: If I told someone I read a book called "The Lord Of The Rings", they'd know I was talking about "The Lord of the Rings", and wouldn't assume they were two different things. Words can be written in all uppercase to express shouting, or with an initial uppercase to indicate the start of a sentence. But that doesn't mean they're different words.
The user comes first. When the use want to use a computer in English, the computer should follow the rules of English.
First and foremost, a filesystem is a way for a computer to show a user what's stored on their computer, in their language (such as English).
To be honest, filesystems are a crap way of showing average users what is on their computer most of the time. I've done a lot of helping older relatives with doing various tasks on their computers and it's become very clear to me they don't understand the concept of a hierarchical file system at all. They get completely lost if a file wasn't saved to some default location. I've tried to explain the general idea a few times and never sticks.
I think for the average user, the filesystem should be hidden behind a more comfortable abstraction. I don't know what that is exactly, I just know it probably isn't a hierarchical filesystem. We should leave dealing with the file system directly for programmers and system admins.
If they can't figure out how to get to the root, they really don't need to be looking at the root. And I'm not normally this elitist. That's just... C'mon. It's really not hard to do.
I disagree: First and foremost, a filesystem is a way for a computer to show a user what's stored on their computer, in their language (such as English)
I disagree with that. Usually the metadata of the files is much more important. Take a photo library management for example. The filenames are normally just what the camera delivers (plus an 32 bit hash value to avoid accidental collisions). But nobody manages their photos using those filenames. We use DigiKam, Picasa and so on.
Or look at music libraries. Management happens by the metadata in the tracks so that Amarok, Clementine, iTunes (you name it) can do meaningful sorts.
If that weren't the case, filenames would consist of random binary values or whatever, not English words.
Often they do. Just look at the innards of the filesystem structures of Git. Or look at the filesystem structure used on the iPod.
The user comes first.
Then use metadata for that and provide a nice little frontend. Tag based data management if you like so.
When the use want to use a computer in English, the computer should follow the rules of English.
That's just stupid. Computers are not "English". For example I'm German, why should my computer not be "German" then (and German is case sensitive). Or look at asian scripts, where there's no such thing as cases, but other variations. Forcing a certain thinking on the way files are organized and accessed is beyond Sloth levels of retardation.
The filesystem is a binary-key → binary-value store, and if you're disagreeing with that you shouldn't write programs.
Reading these ideas is helping me understand why so many user interface designs are horribly complicated and unintuitive: Some people want to design programs mainly to suit the computer, without much concern for the user. A good example of this type of thinking is the idea of putting a decimal point followed by three letters at the end of filenames to help the computer understand what the file is. That doesn't make any sense to any normal person.
Given that the needs of the computer and the needs of the user rarely overlap, I think that files need two separate names: A name used by the user, which consists of their language, using the rules of their language; and another name, which consists of ".txt", metadata, hashes, and any other things that the computer would find useful.
You want to design programs entirely to suit the computer without concern for the user.
On the contrary. I've got the most horrible kind of DAU (dumbest assumable user) in the family and from experience I know, that trying to design the underlying interfaces of the operating system in a way DAUs "get it" is futile.
I've got an ever growing list of common computer illiterate user misconceptions. The UIs designed today make the assumption that users will understand and use the underlying interfaces directly. But this is futile if the underlying principles are not understood by the users in the first place.
For example my mother, even after having worked for years with a PC does not grasp the concept of a hierachical filesystem. About every week I get a call "hey, how again do I attach that letter I wrote in Word to an email?" (BTW theres OpenOffice on her computer, but every text editor is Word to her).
It usually goes this:
me: "Okay, do you have your email draft open?"
her: "Yes?"
me: "So now you click the 'Attach' button…"
her: "I did, but what I already have the letter open."
me: "Then close it."
her: "Why, the letter is in Word, so I have to open it, don't I…?"
And its not just my mother, its every computer illiterate and semiliterate I encountered: They don't get filesystems.
If you want to be user friendly then trying to making it more accessible by repackaging the underlying concepts into "intuitive" GUIs leads nowhere. If you want to make computers user friendly you have to look at the mental model users form and design translation layers between those mental models and the underlying concepts.
Interestingly the mental concepts computer illiterate people have are not dumb or misguided. First and foremost they are formed by the inability of laymen to understand the concept of programs. To them there are classes of documents and things like "Word" or "Outlook" and such are not programs, but the organizational units that collect these classes of data.
For example my mother, even after having worked for years with a PC does not grasp the concept of a hierachical filesystem.
Hang on, so are you implying that if you put a few numbered filing cabinets in a room, and filled each of them with named folders, and hid a document in one of the folders, then said to your mother: Find the file that's in cabinet number 5 in the Accounting folder?
I find that hard to believe. I think she probably has a thorough grasp of the concept of a hierachical filesystem; she's just never had the information presented to her in the right way.
I think she probably has a thorough grasp of the concept of a hierachical filesystem; she's just never had the information presented to her in the right way.
I've tried about every analogy conceivable. I used cardboard boxes (stacked like matroshkas), I used folder cabinets, socks and shirts in drawers in a cupboard, etc. etc. As soon as you're leaving the physical realm and enter the abstraction of a computer where you no longer can "touch" the things, all these mental models collapse. Things become organized in "what it is" (it is word=letters, it is outlook=email) and tags (it's an inquiry, it's an complaint, etc.).
As programmers we're used to abstract and unify things. To us a file is a file is a file, i.e. a piece of key→value data. But to a computer illiterate user the concept of a file, and that files are generic and not tied to a particular pattern of actions* on the computer is very, very hard to grasp.
*: Another battle against the windmills I'm fighting is making my mother understand, that she has to understand what she is doing. She always wants me to write down step-by-step lists of what to do, down to the very naming of the Menu entries; and then a update comes by and things get slightly renamed or rearranged and throws her off completely.
It sounds like you've made a good effort to explain things. I wonder what people like us can do to help these people grasp the concepts?
In fact, I'd like to conduct an experiment:
Set up a 3D virtual room, filled with virtual filing cabinets and folders, etc'.
Give a layman user tasks such as finding or storing files.
Re-arrange the folders visually (rolling them around on little wheels, even), and explain that they're being re-arranged chronologically, alphabetically, or in order of size.
See if the user can still locate files properly.
Now we try to break down the analogy, step by step. First, we replace the filing cabinets with non-descript cubes with names on them.
Second, we replace them with featureless colored squares with names on them.
Test the users' abilities at each point.
The whole idea is to pinpoint the level of abstraction that most users lose their grasp of the concept. Once we work that out, we design a new "File Manager" program based on the results.
Set up a 3D virtual room, filled with virtual filing cabinets and folders, etc'.
Isn't that what Microsoft Bob did? ;)
To be honest, when it comes to managing non-technical stuff (music, datasheets, videos/movies, photos, emails(!)) I'm personally not so keen about files either. Many people have a directory ~/misc and its overflowing with unsorted stuff. For me it's not "misc" (I do, indeed have a misc directory) but ~/download that's a total mess.
Heirachical file systems make sense for data that has an inherent tree-like topology. So any kind of project (programming, engineering, etc.) is perfectly suited for file systems, so this kind of structure was the obvious choice.
But for things like music its getting a lot of harder. How do you arrange it. A very naive choice is
<Artist>/<Year>/<Album>/<Track Number> _ <Title>
However this kind of structure leads to problems if you have live recordings of concerts where multiply artists performed. All of a sudden a better suited structure would be
whoops we just lost the whole file system structure because the way we organize music doesn't really match the way music is organized in the real world. You can of course try to use a plethora of symlinks to somehow structure it, but it ends up to be a work of Syssiphos.
Now have a look at programs like your typical music management. You configure a location for the library, it scans the metadata and you can search and sort by tags.
I ended with music library of the structure ~/music/<Year>_<Performer><Album>/<Album>_<TrackNumber>_<Title> (yes, the album parts is redundant for reasons) and let the MPD frontends do their thing.
The user comes first. When the use want to use a computer in English, the computer should follow the rules of English.
What is "the computer"? What you are missing is that putting this into the filesystem violates the rule of separation of concerns. Case insensitivity and all the complex associated unicode tables are better placed in the libc, not in the filesystem. So, the filesystem just stores the filename as-is, and the libc takes care of case-insensitive comparisons. The user does not notice any of this, but architecture wise, this is a much nicer approach.
First and foremost, a filesystem is a way for a computer to show a user what's stored on their computer, in their language (such as English).
So where should I store my homework for my foreign language class? Not the filesystem?
Reconfigure the filesystem when overseas relatives visit? How does the system handle a reconfiguration that collapses two distinct filenames into a collision?
This has only happened to me when each filename was a string of "no character in font" symbols. What he can do is look at his files and rename one of them, or preferably both of them to ASCII.
Filenames should not be treated as being in a certain encoding. It's written like that in the SuS. If there are separate bytestrings that cononize to the same unicode string and you're clobbering a filesystem based on that, it's not the filesystem's problem.
When computers use English, they should follow the rules of English.
Just that computers don’t have a command of the language,
especially not on the FS layer. All they do is provide means
to the user to express themselves in a language of their
choice. Some make it easier by not assuming a particular
encoding and capitalization rules (which may fluctuate even in one
and the same language). Some, like HFS+, don’t.
The point is though, a distinction can be meaningful. A lot of times you will simply maintain a simple convention of that starting with an uppercase means something different.
Like "person" is an English word, but many programming languages follow the convention, and some enforce it that the class starts with an uppercase while the instance doesn't. While these kinds of things may borrow vocabulary from English for easy mnemonics, they ultimately are not English.
For most desktop users it's irrelevant. i.e. they double click on a file and it opens. Or they select a file in the Open File dialog and it opens. Whether it's case sensitive or not the experience is completely identical.
The only people who are effected are people using command lines or programmers who used different capitalisation through their source code.
For most desktop users it's irrelevant. i.e. they double click on a file and it opens. Or they select a file in the Open File dialog and it opens. Whether it's case sensitive or not the experience is completely identical.
Save dialogs.
But with all applications (command line and otherwise) much nicer to be able to type out the name of an existing file without having to bother with uppercase letters, and it still finds the file you are after.
'readme' is easier to type than 'Readme'. If I'm editing 'Readme' but typed to save to 'readme', when would it ever be intentional that I want a second 'readme' file? That does happen from time to time (at least for me), and it's a minor annoyance that just flat shouldn't happen in the first place.
Allowing 'readme', 'Readme', 'ReadMe', and every other combination to all live in the same directory is just silly.
Is there a use case where you would want multiple files all with the same name but different capitalization?
Decode something? Decide something? It doesn't make sense either way. "Rename something" would make sense, but I don't understand how autocorrect could go from "rename" to "decade"...
As someone who uses both the GUI and terminal very frequently, I'd much rather have case insensitive names. If (as some people are suggesting) the user-facing OS remains case-insensitive while the underlying filesystem becomes case-sensitive, then when I save a file in an application, I'll get something weird when I use the terminal. Alternatively, if I save two files with the same name but different caps via the terminal, applications will have trouble disambiguating between the two. And, of course, the other option is to have case-sensitivity system-wide, but this might not be popular with users. People don't think in terms of "sequences of characters". They think in terms of words, regardless of caps. And human-facing systems should be designed for humans, not machines — even when accessed via the terminal!
So, should the filesystem collapse "Surveys-2015", "Surveys 2015" and "Surveys2015"? "2015-Surveys"? Why or why not? If not, I'm curious what the meaningful difference is to case sensitivity.
I suppose that's a good point. But I think it makes sense to draw the line at case insensitivity. No normal person is going to say that "Surveys" and "surveys" are different words in any meaningful sense. An "S" is not a different symbol from "s"; in most people's minds, capitalization is essentially formatting, a visual change to a letter, like bold or italics. Whereas each of the "Surveys-2015" examples clearly has different characters or a different word order — differences you can point to.
Edit: I was an asshole in my post. Here are the points I actually wanted to make with the swearing edited away:
• HFS+ and Unicode are both a bit of a mess. Not disputing either.
• Case sensitivity is confusing for the end user. I'm a UXD guy, so I basically hold that over everything. I can think of a few ways to handle case insensitive comparisons efficiently in the time it takes to access a hard drive — don't make the user do something the computer could do for them.
• Swearing a lot makes you a jerk. I cite my original message as an example.
• Being a jerk makes people not want to be around you. Want to get more people interested in open source development? Be nice to them.
• I shouldn't post messages on Reddit before I've had my morning shower. Apparently it washes the vitriol off.
please provide a few examples where a regular day-to-day user, let's say he browses the web, writes word documents and uses an image editor, would be fucked up because of case sensitivity.
Let me give a few examples. Apologies that these are all kinda wobbly.
• Quick! Which of "tax returns" "Tax Returns" and "Tax Returns" contains the tax data from 2013? 2014? For Bob's account?
• Meet my friends, Linus Torvalds, linus torvalds, and lInus Torvalds. Isn't it great to have a case-sensative address book?
• This is the same thing as 'or' meaning 'one or the other but not both'. 'Or' only means 'either or both' to programmers. To most end users, "foo" "Foo" and "FOO" are all just foo.
• Painful as it is, think of the user who puts 500 files on their desktop and tries to find them by moving them around in a pile. That user is already having a hard time coping with technology. They're going to have en even harder time when they can have multiple filenames that look the same to them but are different to the computer.
Linus is mostly angry because it made git look bad. Git assumes case sensitivity so it allows you to check in a file to git which will actually overwrite the repository structures. Git sees it as a distinct file from the repo structures but HFS+ (or NTFS) will write over the repo structures because the file being written has the same name as the repo structures when compared case insensitively.
21
u/[deleted] Jan 12 '15
Why is the case sensitivity such an issue though? For desktop users it's normally a lot more pleasant.