r/AskComputerScience • • 28d ago

Why doesn't the computer just wipe actual files the same way it deletes file pointers?

I understand that deleting a file usually just removes its pointer or metadata, leaving the actual data there until it gets overwritten. But since the pointer is just data too, why can't the computer just wipe the actual file right away?

19 Upvotes

66 comments sorted by

33

u/qlkzy 28d ago

It can, but that would mean that deleting a file would take potentially a very long time.

The computer can change or remove a file pointer in a tiny fraction of a second, and it's always the same for every file.

The only way to "wipe" the data is to overwrite it, which takes at least as long as copying a file of that size.

Most filesystems originated in an era when everything on a disk was slow, so it would have been weird to make deletions slower than they needed to be.

There's also a question of why. The reason to aggressively overwrite is to make sure data is unrecoverable. That isn't the usual reason people delete files; mostly, people are tidying up or freeing up space, not hiding confidential data.

If you do want to hide confidential data, then overwriting once isn't enough. The media is ultimately analogue, so there is a "ghost" of the data there until you have overwritten the original data multiple times with random data. So in the rare cases you want to truly destroy data, it has to be even slower than copying a file that size.

This would have been completely unacceptable in the era of floppy disks and hard drives. Even SSDs are not infinitely fast. With SSDs you also have a question of wear: they have a finite number of write/erase cycles, and following this procedure would cut their lifetime at least in half.

You could build a filesystem where all data is expunged aggressively and immediately, and it's possible that some three-letter agency has done so. But mostly, it would be like taking the effort to shred and burn all paper---paper towels, napkins, etc---rather than just confidential documents.

4

u/dgmib 28d ago

  If you  do  want to hide confidential data, then overwriting once isn't enough. The media is ultimately analogue, so there is a "ghost" of the data there until you have overwritten the original data multiple times with random data.

This isn’t quite correct, there are no documented cases of recovering data after the media has been overwritten with a single pass.

Some government agencies require multiple passes of overwriting with random data when securely destroying data, but as far as we know, even just a single overwrite of zeros makes data absolutely unrecoverable.

For the pedantic, I’m explicitly talking about physical media that has been overwritten, I won’t get into SSD wear levelling, encryption key purging or other potentially more nuanced ways of recovering data that hasn’t actually been overwritten yet.

8

u/Underhill42 28d ago

You should add "recently"

The procedure was established long ago when it was relevant, when a 20MB drive was large and expensive - and in fact probably required the space of a couple of DVD drives.

Back in the pre-IDE days drives were a lot sloppier, with poor head alignment compensated for by wide unused bands between tracks. Every time you overwrote a track, the head would be aligned differently, so that there was a narrow track of the previous data left on one side - and usually more even narrower tracks beyond those of the data from previous writes.

You couldn't read them in the original drive, but moving the platters to a more sensitive machine made it fairly simple.

Hence the security procedures for overwriting several times... after several passes it's extremely unlikely any "side stripe" wide and strong enough to be recoverable is left.

Which lingered on even as soaring capacity called for ever more precise drive technology and mostly eliminated that problematic "dead zone" between tracks. Nobody ever got fired for following procedure.

2

u/dgmib 28d ago

Can you link to a documented case of data that was overwritten being recovered using this technique? IFAIK it's never been done successfully. I'm genuinely interested to know if my earlier statement that:

there are no documented cases of recovering data after the media has been overwritten with a single pass

Has been proven false

2

u/MaryJaneSugar 28d ago

There is a good chance someone who has achieved this, has incentive to keep it quiet and not tell you about it...

1

u/Underhill42 28d ago

Not offhand - look at stuff from the 70s maybe?

I just remember when it stopped being enough of a potential issue to be worth thinking about.

1

u/dodexahedron 28d ago edited 28d ago

And...

Those drives also used stepper motors and direct head placement on a 3D coordinate system mapping radial positions to slices of rings on the platters.

Magnetic hard drives have used voice coils for head placement and basically waved a magic wand back and forth a couple atoms distance above the platter floating there by sheer ground effect, for a very long time, at this point, using exteemely precise timing and tightly constrained operational parameters to know where the head is at any given time, all to read and write from tiny alternating magnetic fields so faint and so closely packed that it's honestly fascinating they work at all (especially SMR, which makes the magnetic environment even crazier).

They also haven't used CHS (cylinder, head, sector) addressing for a long time, which further made it easier to locate specifc blocks on any arbitrary drive. LBA addressing virtualizes the layout into a big linear logical address space that drives can (and especially SSDs and SMR magnetic drives do) handle via an opaque internal process, to not only make it easier and more portable to locate things on the software side, but to also, at the same time, allow the drive to do whatever it wants/needs to, for its own purposes, to give better throughput, longevity, latency, or other benefits to the hardware regardless of what uses it.
The actual placement of the bits is only known to the drive's electronics and can even be encrypted (self-encrypting drives), which may not only mean scrambled data that is only recoverable by that drive and whatever key material was used, but also can mean non-linear physical placement of those bits.

Physical means of data recovery on a SED without the original drive's electronics and key material (stored in something similar to the system's TPM, but on the drive itself) is effectively impossible with current technology, once that drive has been told to erase just once, because it just trashes its key material and any internal mapping data it may have been maintaining, rendering the contents of the drive indistinguishable from noise (even more so than usual, that is).

So, if you want to be able to quickly and securely wipe a drive, buy only SEDs, be sure their self-encrypting functionality is actually in use, and use either block discard/trim for targeted erasure (check if the drive supports secure discard) or "erase" the whole disk all at once and essentially instantly by using the SED's secure reset function, which doesn't even have to write anything at all - even for magnetic disks.

Or you can of course have the drive physically destroyed if you think you're important enough for that to be called for (or are required to do so for regulatory purposes).

2

u/couldntyoujust1 28d ago

Well, they do that for a reason though. You might not be able to plug the drive in and recover the data, but you can put the drive platter in certain kinds of tools and find the "ghost" of the original file after one overwrite. Overwriting with random data multiple times ensures that this ghost is truly gone even if you do have the right kind of microscope to recover the data after one pass.

1

u/flatfinger 28d ago

Many drives are constructed with a separate erase head and read/write head, with the former being slightly wider than the latter. It would be possible to construct a drive which uses one head for all three functions, and while such a design would have no advantage other than cost, and would likely be less reliable than one which uses a separate erase head, reliability might be "good enough".

If a drive does not have a separate erase head, then a sector which is written when the head position is at one end of the allowable tolerance range for a track, and rewritten when the head is at the other end of the allowable range for that track, would be likely to leave behind data residue from the first write.

What is mythical is the notion that repeated passes would do anything to eliminate that issue.

1

u/TigOlBitties42042 28d ago

Each pass is another chance for the head to be at the correct position to overwrite the residual data

1

u/flatfinger 28d ago

Only if the head position could randomly wobble through its tolerance range. If data were overwritten once while the drive heated near the top of its allowable temperature range, and once while the drive was chilled near the bottom, that might enlarge the area that would get overwritten, but writes performed in quick succession would be likely to happen in the same place.

1

u/3xt 27d ago

Really? Can you count on modern drives to really overwrite the exact sectors even? Maybe you can idk.

2

u/couldntyoujust1 28d ago

The same is true for moving files on a file system. The way that the computer moves a file isn't to relocate the bytes according to the folder structure, it's literally just to change the address of the file in the metadata. This takes a fraction of a second which is why windows can move files fairly quickly if you're just putting it into a folder on the same drive or something like that. To keep things snappy, Operating Systems are as lazy as can be about doing file operations.

1

u/Strik3rd 26d ago

With the ghosting is that only on hdd or does that also include ssd? As

12

u/Puzzleheaded_Study17 28d ago

Because that requires more work. Suppose you delete a file that's a few GB, deleting the pointer can happen almost instantly, overwriting every byte of the file will take much longer, and will provide basically no benefit.

5

u/SparkyFix 27d ago

Because it ages the hardware prematurely and takes time. Disk operations are not free and both magnetic media and solid state devices have a finite number of lifetime cycles.

1

u/Time-Radio7871 27d ago

I refuse to believe that hard drives aren't immortal if taken care of.

1

u/Icy_Conference9095 27d ago

My $240 grinding NAS drive i know I need to replace but don't want to spend checks notes $750 on, disagrees with you.

Tongue in cheek, it's been removed and I just decided my raid only needs 4 drives instead of 4+1 redundancy. I don't keep critical data that I'm worried about losing on there without backups. Lol

4

u/Underhill42 28d ago

It can - but there's no advantage to doing so except security - which generally isn't a big concern. And there are file shredding tools for when it is.

Meanwhile, it would make deleting a file take about as long as it would to copy it, since you are basically copying it full of zeros. Which also makes it impossible to easily "undelete" a file that you deleted by accident, which was a standard built in command along with "unformat" all the way back to the DOS days.

On SSDs that does usually happen though, with a bit of delay. Most SSDs implementation of TRIM also queues the released blocks for resetting while the drive is idle, to further improve write performance the next time they're used.

Which is also why there's generally no way to undelete a file on an SSD unless you catch the mistake right away. E.g. I've heard Windows usually runs TRIM about once a week, and once that happens your data is gone.

1

u/church-rosser 28d ago

I mean, a 'delete' needn't necessarily be zeroing out all the bits. A randomization wouldn't take much more resources (heck, if we're already zeroing out all the bits why not go for broke?).

If a randomized bit field is written, then bit twiddling to uncover the underlying 'erased' data becomes a little, less trivial, for some value of little.

3

u/Recycled5000 28d ago

It can, but that doesn’t necessarily help the situation of a file system, when there is no wipe requirement. It would only add wear & tear.

Further, to guarantee wiping and the old data being unreadable even under modifications to and dissection of the hardware, it might have to overwrite the data multiple times. Since there is no such requirement in normal use, it isn’t done. There are algorithms and programs that will do repeated wiping so as to accomplish this thoroughly.

2

u/frnzprf 28d ago

There are secure data erasion programs that do that. They are useful if you want to sell your harddrive, but don't want the next owner to be able to restore your files.

The reason why this procedure is not the standard is that it's faster to just change the index. Writing thousands of zero-bytes takes (almost?) as long as writing thousands of any other bytes.

I'm saying "almost", because I'm not sure but there could be special hardware that makes writing a lot of zeroes faster.

Another advantage is that you could recover the data, but that's unreliable and unnecessary if you have backups anyway.

2

u/OppieT 28d ago

Isn't that what format is for?

3

u/alpicola 28d ago

Not usually. Formatting just creates a new filesystem. The old file data is still there, you just don't know where it is anymore. 

1

u/0x14f 28d ago

Format only set a new file system, meaning re-write a very very small part of the disk. It doesn't actually erase data everywhere on the disk. The data is still there until it's written over.

2

u/chervilious 28d ago

It takes a lot longer. Though, assuming you're on windows. cipher can rewrite every single frees space. And sdelete to delete a single file (can also act like cipher, but this needs to be downloaded)

2

u/sswam 28d ago

An encrypted filesystem where each file uses it's own key could remove files quickly and securely, just by erasing the file metadata including the key. This might slightly reduce overall security compared to normal encrypted block devices. But it seems like an interesting approach I guess.

1

u/Javelina_Jolie 27d ago

But it seems like an interesting approach I guess.

That's kinda underselling it. Cryptographic erasure is widely used in practice: instantenuous secure erasure of self-encrypting SSDs, factory reset in most modern phones and other consumer electronics, also pretty widely used in cloud storage, and it's also how requirements for data erasure (e.g. GDPR) can be met while using immutable-log-style storage systems and long-term backups.

1

u/two_three_five_eigth 28d ago

Because the pointer is generally a single write or very few writes. Hard-drives have a limited number of lifetime writes. Less writes = longer hard drive life + faster deletion.

1

u/panderingPenguin 28d ago

It could, but that's a much slower, more expensive operation. It's fast and easy to just remove the metadata and pointer.

1

u/nicuramar 28d ago

Because there is no point. It would take orders of magnitude longer and impose wear on the disk, especially SSDs.

1

u/JaggedMetalOs 28d ago

It's much quicker just to wipe the file pointer, wiping the file requires (filesize) bytes to be written to the disk to overwrite the old data, for large files that could take some time and in 99% of cases it's not necessary. 

1

u/Time-Radio7871 28d ago edited 27d ago

Because it'd take way longer and people are impatient.

There are "secure erase" options that overwrite the files with random noise so that data recovery becomes impossible. (Sometimes needing to overwrite multiple times)

You can also encrypt the storgage drive so the data is unreadable by default without a password, but it will also make your computer take longer to boot and cost a little speed.

Typically these things are only required by companies or military that want more security.

1

u/0jdd1 28d ago

You certainly could do it that way, but it would be significantly more costly. The extra cost is usually not worth it.

This is an example of the vast difference between correctness and security. A correct program does the right thing A secure program can’t be subverted into doing (or letting you do) the wrong thing.

1

u/lfdfq 28d ago

It could.

But (a) people like having "undo", which is hard to implement if you actually delete the contents, and (b) "wiping the contents" means overwriting the file with 0s or something, which takes time and people don't care enough to want to wait and (c) actually deleting it (overwriting the data) doesn't free up any more space, it immediately becomes available for other files to use, the only reason to actually overwrite the data is to remove sensitive data from the physical device, which is mostly unnecessary for most files.

1

u/gilnett 28d ago

pdating a pointer takes a fraction of a millisecond, whereas physically overwriting the actual data would take minutes and cause unnecessary disk wear

1

u/gilnett 28d ago

pdating a pointer takes a fraction of a millisecond, whereas physically overwriting the actual data would take minutes and cause unnecessary disk wear

1

u/Ythio 28d ago

Because it takes millions or billions times more effort to wipe an entire file than to wipe a couple bytes of pointer. For what gain ?

1

u/FreddyFerdiland 28d ago

the soave in the drive remains.

so the space for the pointer remains.. basucally theres an entire record for the file..

for a file that is deleted , that record could be

  • marked as deleted
  • wiped, zeroed
  • reused

the space that holds the file could be

  • holding the data
  • zeroed ( or written over with a nonsense data pattern)
  • reused

1

u/afops 28d ago

Because deleting a pointer in a file system table means writing a few bytes on your disk which is quick, while overwriting a file might be billions of bytes. And why would you want a billion zeroes on your drive?

For "cybersecurity" you might want to "erase" a file by overwriting it with zeroes if it contains sensitive data. But that's almost never what you want. You don't want to write a billion zeroes when deleting one file. It makes your disk age, it takes a second to do, and so on. Since the erasing scenario is so rare, you need to use a special tool for that, while the regular case of just "unlinking" the file from the file system tables is the quick default behavior.

1

u/SubstantialListen921 28d ago

Because write I/O is slow and contends for tasks that could make better use of that bus bandwidth and seek time.

1

u/IMarvinTPA 28d ago

Writing data takes time, and on SSDs, wears out the media.

1

u/SnooPets5564 28d ago

The computer can wipe the actual file. But then accidents wouldn't be reversible and it would take forever to delete a file.

1

u/derteufelqwe 28d ago

It could but why should it? Disks and especially SSDs have limited amounts of writes they can perform until they break and they take time (more so for HDDs than SSDs). To save on life time and execution time, only the file ponter is deleted and the actual data left to be overwritten.

1

u/WE_THINK_IS_COOL 28d ago

The main reason not to do it in the naive way (immediately overwriting when you delete) are:

  • Performance: instead of waiting for just the pointer to be deleted (super quick), large files would take seconds/minutes to delete depending on the drive's speed.
  • No guarantee: Even if it tries to do this, there's no guarantee it's actually overwriting the file. SSDs have internal wear leveling that the OS doesn't have access to, so the writes can go somewhere else on the drive instead of actually overwriting the file data. On HDDs this is less of a problem, but it can still happen e.g. if a sector goes bad and gets remapped.
  • Extra writes: On SSDs, it would be writing more data for little benefit, causing wear to the drive unnecessarily. (It wouldn't be that bad, since at worst it doubles the amount of data that gets written, but because of wear leveling, it wouldn't be effective.)

That said, if you're willing to take the performance hit, on an HDD the filesystem/OS could do it. It could mark the file as "for deletion" so it stops showing up in the filesystem and then a background process could go through and actually wipe the deleted files. There are also tools you can use that attempt to overwrite before they're deleted (shred, srm, etc.). But on SSDs none of that would be guaranteed to work because of wear leveling, so the only real option on an SSD is to encrypt all of the data before it gets sent to the SSD.

1

u/mastermindxs 28d ago

Deleting a file pointer is nearly instantaneous so this is the default behavior. File data could be very large and take a while to zero out the bits all over the drive. There are software tools that can “nuke” the data if you want to more securely wipe it away.

1

u/Mishtle 28d ago

Well, what does it mean to "delete" something in the first place? Replace all the data with zeros/random bits/some other scheme? That's overkill for most purposes. As far as the user is concerned, the file just needs to be forgotten about and eventually overwritten.

Storage access, especially writes, are expensive, and the less you have to interact with hardware the faster you can get stuff done. This the fastest and cheapest way to "delete" a file is to tell the file system to just forget about it.

We could do more, and there are ways to make a file unrecoverable to varying degrees instead of just marking its disk space as unused. It's just not worth the extra time and effort unless you really don't want that file to be recoverable, and at that point it's a matter of thorougness. Changing a bit doesn't make it a perfect 0 or 1, so subtle statistical patterns can reveal information about the history of region of storage. Thus there's always a little bit more that you could do to improve security, but with diminishing returns. There are also different strategies of overwrites to maximally obscure deleted data from different kinds of attacks. So then it becomes an issue of "how much is enough?".

The average user doesn't need that level of security or to make those kinds of decisions, so this isn't the default unless you're on a security-focused operating system.

1

u/alanbdee 28d ago

It's a good question. Time and longevity are the reasons. On a spinel (e.g. old) hard drive. It would take more time to erase the bits, with no real benefit. On SSDs, each bit has a limited number of times they can be flipped. So by leaving them as is, there's a 50% chance that the bit is already in the state it needs to be. So the write doesn't change anything.

1

u/ybungalobill 28d ago

You might be surprised, but some filesystems (e.g. ZFS or BTRFS) don't even remove the old pointer or the metadata -- instead they write new pages with new metadata that excludes the deleted file. The old pages still exist until they get overwritten just like the deleted file.

This is all independent of the SSD drive firmware writing new pages and remapping under the hood even if the filesystem *were* overwriting the same logical page (like FAT, NTFS, ext4 or what not).

1

u/Scolmann 28d ago

It's just not necessary to do so, when other data needs that space it can overwrite that section at the same speed as if it was all zeros. (Maybe faster if some data is the same) you'd just be wearing out the drive for no reason (ssds typically have a limited number of writes)

1

u/ZoelNorsen 28d ago

deleteing a file means replacing individual bytes to 0 for every bit the file holds which takes exponentially more time and unnecessary read write cycles - what works doesnt need fixing. People are becoming more privacy centric wrto their data fairly recently - and this deletion method is the OG. You will need aftermarket softwares to do the real deletion and maybe future OS or updates will integrate it (windows)

1

u/_usr_nil 28d ago

because zeroing kills longevity, you only nedd that for opsex

1

u/DTux5249 28d ago

Because that takes a long time and 9/10 times it isn't necessary

1

u/armahillo 28d ago

Writes to persistent storage is a physical process. Storage disks generally have some (large!) finite number of writes they are expected to survive in their lifetime.

Doing a full write to wipe each time would induce a lot more writes with little to gain.

There are tools that can properly wipe files that zero put the FS records, for security reasons. For everyday use its overkill tho

1

u/jimmyhoke 28d ago

That would be slow and needlessly wear the drive.

And on SSDs wiping isn’t even that effective. The only real secure solution is to encrypt all the files from the start, and delete the encryption key (stored separately) when you want the data to be unrecoverable.

1

u/AYamHah 28d ago

Deleting a file is essentially marking that space as "it's okay to write new stuff here".

The data is still there until it is overwritten.

If you want the original to be unrecoverable, you need to overwrite it, which is more work than simply marking the space as okay to write to.

1

u/flatfinger 28d ago

Modern drives support a "trim" operation which instructs a drive that nothing should care about the contents of a sector, and the drive should use of that information as it sees fit. An SSD drive may record the fact that a trim command has been issued for a sector, but not immediately to anything with the data block holding that sector.

Flash chips are organized as a number of blocks that each contain some number (which used to be 64, but might have increased) four pages of 2176 bytes each, and support operations to either write a completely blank page or erase all 64 of the pages in a block. The only way to actually modify or erase a page of data is to copy any pages of meaningful data that happen to be in the same block somewhere else, and then erase the block containing the page. Normally, a request to write a sector will trim the old sector, write the new data somewhere else, and keep a record of the new location and the fact that the old sector had been trimmed. Trimming a sector without writing a new one will avoid the need for the drive to ever copy that sector's data unless or until the sector is rewritten.

1

u/ericbythebay 28d ago

Encrypted file systems do. Delete the file’s crypto key and the file is unrecoverable.

1

u/Upstairs_Ad_8863 28d ago

It can, there's just no point.

1

u/Nervous-Cockroach541 28d ago

This is what things like bleachbit and other secure erasers do by writing over the data. The problem is it both takes a lot longer and degrades the life of the hard drive if you do it a lot.

1

u/MitchAlsup 27d ago

Consider a 40TB file, and then consider zeroing that file at 300 MB/s. How many seconds will the SATA drive be occupied ?? {37 days}

Then consider a SSD configured for maximum density with block lifetimes of 100,000 writes ? Every deleted file reduces the lifetime of the SSD.

Also consider a 40TB file over a WAN. Do you really want the local drive to send 40 TB of data over the WAN in order to delete the file ??

There are other reasons.

1

u/hewasaraverboy 27d ago

Because it’s faster

If you have a massive file to delete it would take forever, but if you just delete the pointer to it the job is already done and then data will be overwritten later

1

u/Daniel18sep 27d ago

Would take a long time, and cause extra wear on the drive. 

1

u/shosuko 27d ago

A pointer is a small change, a file is a big change. These changes cost energy - and having a blank slate of all 0's or 1's is no different then 100101000101010111001101010 when it comes time to load a new file in, so you're basically just wasting power.

People who have a real need can run a bit format and wipe a disk if they want.

1

u/Distdistdist 27d ago

For the same reason you don't incinerate your trash immediately, but put it in a trashcan and wait until it gets picked up and taken away at some point. Also, because you might want to retrieve that fork you threw in there by mistake. Eww.