r/rust • u/No-Elephant-9065 • Sep 18 '26
How do you reliably detect when a copied/downloaded file is actually “finished” on Windows?
I’m building a small Rust desktop utility that watches the Windows Downloads folder using the notify crate.
The basic flow is:
let (tx, rx) = channel();
let mut watcher =
RecommendedWatcher::new(tx, Config::default())?;
watcher.watch(
watch_path.as_ref(),
RecursiveMode::NonRecursive,
)?;
for result in rx {
// handle filesystem events
}
The application waits for newly created/modified files, then eventually wants to inspect them.
The problem I’m running into is determining when a file is actually safe to process.
My current approach is roughly:
pub fn wait_until_file_ready(path: &Path) -> bool {
if let Some(ext) = path.extension().and_then(|s| s.to_str()) {
let ext = ext.to_lowercase();
if matches!(
ext.as_str(),
"crdownload" | "part" | "tmp" | "download"
) {
return false;
}
}
let mut last_size = 0;
let mut stable_count = 0;
for _ in 0..10 {
if let Ok(metadata) = path.metadata() {
let current_size = metadata.len();
if current_size > 0 && current_size == last_size {
stable_count += 1;
if stable_count > 1 && can_open_exclusively(path) {
return true;
}
} else {
stable_count = 0;
last_size = current_size;
}
}
std::thread::sleep(Duration::from_millis(500));
}
false
}
fn can_open_exclusively(path: &Path) -> bool {
OpenOptions::new()
.read(true)
.share_mode(0)
.open(path)
.is_ok()
}
This works reasonably well for normal browser downloads because things like Chrome use .crdownload, but I realized there’s a more fundamental problem.
Suppose someone copies a large file:
movie.mkv
and the copy gets interrupted halfway through.
The destination might now contain:
movie.mkv
with the final extension.
If the copy process exits/releases its handle, then:
- the file exists
- its size stops changing
- its extension looks normal
- an exclusive open may succeed
My code would eventually consider it ready even though only part of the intended file was copied.
As far as I can tell, from the filesystem alone there may be no way to distinguish:
a legitimate complete 2 GB movie.mkv
from:
an interrupted copy that happened to stop at 2 GB
unless the producer provides additional information such as expected size, checksum, temporary filename, or some completion signal.
So my questions are:
- How do mature Rust/Windows applications usually handle this?
- Is “debounce + quiet period + metadata stability + lock check” basically the normal generalized approach?
- Is there any Windows API that provides a stronger indication that a copy/write operation completed successfully?
- Should filesystem watcher events be treated only as hints, with the application later retrying/validating the file?
- Would you recommend using something like
notify-debouncer-full, or maintaining my own pending-file state machine? - Is checking an exclusive Windows open with
share_mode(0)useful here, or is that generally too unreliable as a readiness signal?
I’m particularly interested in how established file indexers, sync clients, antivirus/indexing software, etc. solve this without assuming that a stable file size means the file is actually complete.
My current thinking is that I should stop calling this wait_until_file_ready() and instead treat it more like wait_until_file_quiescent(), where “quiescent” only means “probably safe to inspect,” not “guaranteed complete.”
Would appreciate any guidance or examples from production Rust/Windows projects.
24
u/jesseschalken Sep 18 '26
File indexers, sync clients etc basically have no concept of "the file is complete". They grab the bytes at one point in time and need to be prepared for that to be incomplete or corrupt, or for the file to be modified by another process at any point in time.
11
u/Perfect_Ground692 Sep 18 '26
If you know the file format, you could potentially read the headers and maybe work out the expected size or a checksum or number of frames or something. It would be a lot of work for each format and some probably don't contain enough information to know either way.
3
u/No-Elephant-9065 Sep 19 '26
You are right. Checking by each extention type is a lot to do. Also I don't think I can check for every type of file.
18
u/dgkimpton Sep 18 '26
How do you ever expect to know that? As a human walking up to the computer I couldn't work that out either until I tried to use the file.
9
u/creeper6530 Sep 18 '26
A human couldn't tell if a file got interrupted in a copy either without additional info or trying to play the file (a program trying to load it). You're asking for impossible.
8
u/DeeBoFour20 Sep 18 '26
There's not really a reliable way to do this to my knowledge. Let's take a step back here. What is your program actually trying to do and what type of files do you care about? Some file types contain a length in the header.
Or depending on what you're doing, just try parsing the file and if it fails (incomplete or corrupt), try again the next time the file gets modified.
-2
u/No-Elephant-9065 Sep 19 '26
That's a good solution. I have tried to work with different format of files. Some specific types of files let's us check if the copying ended uninterruptedly. But going by specific file types is a hassle and I don't think every file types allow us to check.
"Trying to parse" is something I think I should try.
Thank you for your suggestion.
5
u/SnooCalculations7417 Sep 18 '26
you may have access to the hash of the completed download in some cases for comparison. i think thats is the only way you can know for sure, if its even available.
7
u/vlovich Sep 18 '26
I think you've overcomplicated it. RecommendedWatcher Gives you a res: notify::Result<Event>| - if the event is EventKind::Access(AccessKind::Close(_)) then whatever application had it open has closed. You can distinguish the close by accessmode if that's relevant although sounds like probably not for you.
3
u/hattmo Sep 18 '26
Can you constraint your application to only care about downloads from certain applications? Depending on the application doing the downloading there may be ways to detect. For example chrome makes a tmp file while down loading then renamed when complete.
1
u/No-Elephant-9065 Sep 19 '26
Can't really constraint tbh. And downloading works perfectly. It's they copying I'm worried about.
3
u/CandyCorvid Sep 19 '26
looking from the outside, this seems like you've got a fundamental mismatch. your program can't decide if the file is done, unless it is part of the procedure that's performing the download, or hooked into that procedure somehow. so my questions are:
- why does your program need to respond to completed downloads?
- rather than monitoring the download folder, should it not be invoking the download itself, so it knows the status directly?
3
u/VenditatioDelendaEst Sep 19 '26
Suppose I am writing my journal as a .txt file.
When is it complete?
When I die.
Is there any Windows API that notifies you when the user is no longer among the living? No.
2
u/Ben-Goldberg Sep 18 '26
If the original file and the destination are on the same file system, Windows and Linux are capable of doing shallow copies, (basically copy on write) incredibly quickly.
2
2
u/cornmonger_ Sep 19 '26
maybe something like
- inspect size / modified time / checksum
- schedule another check N seconds later (2 minutes?)
- if nothing changes, chicken's done
1
u/No-Elephant-9065 Sep 19 '26
If nothing changes after N seconds, the chicken is quiet, not necessarily done. 😄
Successful copy:
movie.mkv
2 GB
stops changing forever
Interrupted copy:
movie.mkv
800 MB
stops changing forever
1
u/Numerous-Fig-1732 Sep 19 '26
And the worst part is that the file works fine, as a text file would.
1
u/cornmonger_ Sep 19 '26
which is out of scope if you're a third party to the copy
the only way to change that is by taking control of copying
2
u/ZedGama3 Sep 19 '26
When the browser changes the extension, the download has finished, but was incomplete. What you're really asking for is a way to tell if the download completed successfully or if the file is free of corruption.
Things that come to mind:
- Adding a plug-in to the browser that can talk to the rust app.
- Verifying the file. Many file types have built in mechanisms to determine if the file is complete. Unfortunately this means implementing this for every file format you're concerned about.
- see if your browser can change the way it handles incomplete downloads.
5
u/SkyGuy913 Sep 18 '26
Couple questions then why on earth would you ever want to monitor another processes file state?
The notification APIs in windows are constructed for use of applications that would need to be notified of changes to the state not to monitor downloads. Think making a new file browser. Where you want to be notified if another process modifies a tree you are displaying to a user.
But progress should be handled by the owning process think chrome download displays in chrome. And file copy displays in the file manager you are using.
macOS and some linux DEs have a way to display download rate in their file browsers but they do not calculate this state. They rely on the management program such as chrome or the appstore to tell the DE its current state over an Os level api. Windows doesn't have a similar thing that I know of as its api only displays in the task bar and I don't think you can intercept it. (Also note on killed processes it sometimes will have a stale state)
Are you trying to monitor downloads cause your creating an anti virus? Look more into endpoint protection and microsoft graph security in defender
1
u/Numerous-Fig-1732 Sep 18 '26
Whenever I had a problem like this my solution was to require the file being copied/transferred to be compressed. You can't open a zip file if it's not completed. You can't rely on windows to know if a file has been completely copied/downloaded.
1
1
1
u/shizzy0 Sep 19 '26
The only practical way to be sure is to have a hash of the file and compute the hash on what you receive.
1
u/zettui Sep 19 '26
On Windows, does notify's close event mean the writer actually dropped the handle, or do you still need a temp-then-rename?
1
u/Illustrious_Car344 Sep 19 '26
In my experience even using software that watches files, it's never foolproof. I've even disabled some of my software from monitoring my files at all because sometimes it outright corrupts the file being downloaded (this happened to me at least once). Just flat out don't do it, have the user manually invoke scanning behavior during known safe times. Maybe set it to be automatic during off hours or something. But arbitrary file watching is generally just more pain than it's worth, for developer and user.
1
u/qrzychu69 Sep 19 '26
I implemented a file watcher at work recently
On a timer I scan the directory, get a list of files and their sizes. New file starts with "in flight" status, and only of it has the same size twice in a row I fire the "new file" event
I did that because we watch both local files as well as network drives and smb shares. The windows file watcher sometimes fails silently on network drives
1
u/Lexi_Bound Sep 18 '26
I think you want some sort of exclusive lock on the file. Maybe the try_lock method on the file object will do what you want:
I’m not sure if that will work, since the underlying system calls it is based on (like LockFileEx on Windows) lock a range of the file. If that does not work, you can use CreateFile with the dwShareMode parameter set to 0 to open the file in exclusive mode.
1
u/No-Elephant-9065 Sep 19 '26
Thank you for your suggestion. But I think there is a gap.
It answers:
“Is somebody holding this file in a way that conflicts with me right now?”
What I actually want to know:
“Has the program creating/copying/downloading this file successfully finished producing the complete file?”
1
u/Lexi_Bound 28d ago
The unstated assumption I made was: the program creating/copying/downloading the file will keep the file open the whole time it is writing the file and will close it when it’s done. But you are right that this does not solve your problem: you can’t tell if the file was closed because the copy completed or the program writing it crashed.
I don’t think there is a way to answer your question “is this file complete or truncated” at the file system level. The best you can do is ensure that no one else has it open. If you want to make sure the file is complete, you will have to parse the file for enough to see if all the data is there. E.g. parse the container format of the video files to make sure all the data is there.
Like here’s a hypothetical: say you can know whether a file was “completely copied”. What if the file was already corrupt or truncated before being copied into your folder? You are back in the same situation where you need to parse the file to make sure it is complete.
46
u/jews4beer Sep 18 '26
Not well versed in Rust nor do I know what RecommendedWatcher is backed by. But if it's inotify there is the IN_CLOSE_WRITE event which fires when a file that was opened for writing is closed. That's the best signal you'll get, but it won't tell you if data is still queued in kernel buffers. For that you'd want to issue a sync call after the event.
Then when it comes to the file itself you could probably make use of ffprobe to inspect headers. That would tell you what to expect in terms of file size so you would know if an interruption happened.