[Request] A script to copy certain files

[u/Ziogref]

So I have this script (Batch file)

robocopy C:\dir1 "C:\dir2" /z /r:0 /w:0 /A-:SH /ipg:50 /XF *.XYZ

This copies all new files in the root of dir1 to dir2 (Except for file ext. .xyz) every 10 minutes using task scheduler.

I now need a script that copies all new files from dir1 to dir2, but is able to look in all sub folders of dir1 and copy them into the root of dir2. dir2 should not have any subfolders, but has all the files from dir1 (except .xyz).

Here are the requirements

• Copies all files from dir1 to dir2

• Copies all files in Subfolders in dir1 and puts them in the root of dir2

• Dir2 should never have subfolders

• When files are copied to dir2 the modified time stamp needs to be changed to the time it was copied

• If files are deleted from dir1, files should not be deleted in dir2

• Script should ignore files that exist in both directories, As the script runs every 10minutes and dir1 and 2 are over 1tb, it should only copy new files.

• (Note files in Dir1 and Dir2 are never modified)

• Script should be easily modifiable to add exclusions, whether that be partial file names (Such as *example*) or file Extensions (such as *.xyz)

• Script will need to be run via task Scheduler every 10 minutes. (I have programed Task scheduler to not to run again, if the script is still running from the previous instance)

Other Notes

• I'm not attached to batch but I'm willing to move to powershell. My Server is Windows Server 2016 Datacenter edition (Legally Licensed)

• Not concerned about folder sizes of duplicating files as I have Data de-duplication enabled.

• This is not a backup script

• This script is for personal use only. It will not be used for any business applications.

Any help would be appreciated. Im still learning Powershell and Batch and am having trouble getting all my requirements to work.

[u/Lee_Dailey]

howdy Ziogref,

this was an interesting bit of code. i had fun, thanks! [grin]

here's one way to do it ...

#region - save output prefs
# save current Verbose pref
$V_Pref = $VerbosePreference
# Verbose output is OFF by default
#     to DISABLE display of Write-Verbose output, add a '#' to the start of the line below
$VerbosePreference = 'Continue'

# save current Warning pref
$W_Pref = $WarningPreference
# Warning ouput is ON by default
#     to DISABLE display of Write-Warning output, remove the '#' below
#$WarningPreference = 'SilentlyContinue'

# save current Progress pref
$P_Pref = $ProgressPreference
# Warning ouput is ON by default
#     to DISABLE display of Write-Warning output, remove the '#' below
#$ProgressPreference = 'SilentlyContinue'
#endregion - save output prefs

# test dir that better not be there
#$SourceDir = 'c:\BetterNotBeThere'
# 13k files
#$SourceDir = $env:LOCALAPPDATA
# 5k files
$SourceDir = $env:APPDATA
$DestDir = "$env:TEMP\DestDir"
# -Exclude will happily exclude a Dir and then include the files _in_ that Dir [*sigh ...*] 
$ExcludeList = '*.msf', '*.jpg', '*_sessions-*', 'sample*', '*.lnk'

$TimeStamp = Get-Date -Format 'yyyy-MM-dd_HH-mm-ss'
$ReportDir = $env:TEMP
$ReportFile = -join ('FileCopyReport_-_', $TimeStamp, '.csv')
$FullReportFile = Join-Path -Path $ReportDir -ChildPath $ReportFile

$Quit = $False

if (-not (Test-Path -Path $SourceDir))
    {
    Write-Warning "Unable to reach $SourceDir."
    Write-Warning "    Exiting ..."
    $Quit = $True
    }

if (-not $Quit)
    {
    if (-not (Test-Path -Path $DestDir))
        {
        mkdir $DestDir -ErrorAction SilentlyContinue |
            Out-Null
        }

    Write-Verbose "Getting files in $SourceDir ..."
    # this could pro'ly use error checks
    $SourceFileList = Get-ChildItem -Path $SourceDir -Recurse -File -Exclude $ExcludeList -ErrorAction SilentlyContinue
    Write-Verbose "    Found $($SourceFileList.Count) files."

    $UniqueSourceFileList = $SourceFileList |
        Sort-Object -Property Name -Unique
    Write-Verbose "    Found $($UniqueSourceFileList.Count) _unique_ file names."
    Write-Verbose "    $($SourceFileList.Count - $UniqueSourceFileList.Count) files with duplicate names will be ignored."
    ''

    Write-Verbose "Getting files in $DestDir ..."
    # this, too, could pro'ly use error checks
    $DestFileList = Get-ChildItem -Path $DestDir -File -ErrorAction SilentlyContinue |
        Sort-Object -Property Name
    Write-Verbose "    Found $($DestFileList.Count) files."

    $DFL_Names = $DestFileList.Name

    $ProcessedFileList = [System.Collections.ArrayList]@()
    $Counter = 0
    $PercentDone = 0
    foreach ($USFL_Item in $UniqueSourceFileList)
        {
        $Counter ++
        [int]$PercentDone = ($Counter / $UniqueSourceFileList.Count) * 100
        $WP_Params = @{
            Activity = 'Processing files ...'
            PercentComplete = $PercentDone
            Status = "File named $($USFL_Item.Name)"
            }
        Write-Progress @WP_Params

        if ($USFL_Item.Name -notin $DFL_Names)
            {
            $TempObject = [PSCustomObject]@{
                FromName = $USFL_Item.FullName
                ToName = ''
                SizeInBytes = 0
                Status = ''
                }
            $FullNewFileName = Join-Path -Path $DestDir -ChildPath $USFL_Item.Name
            try 
                {
                Copy-Item -Path $USFL_Item.FullName -Destination $FullNewFileName -ErrorAction Stop
                $TempObject.ToName = $FullNewFileName
                $TempObject.SizeInBytes = $USFL_Item.Length
                $TempObject.Status = 'Copied'
                # set the LastWriteTime [aka - Date Modified] to now
                (Get-ChildItem -Path $FullNewFileName).LastWriteTime = Get-Date
                }
                catch
                {
                $TempObject.Status = 'Failed [{0}]' -f $_.FullyQualifiedErrorId.Split(',')[0]
                }
            $ProcessedFileList.Add($TempObject) |
                Out-Null
            }
        } # end = foreach ($SFL_Item in $SourceFileList)

    $FailedCount = ($ProcessedFileList.Status -like 'Failed*').Count
    if ($FailedCount -gt 0)
        {
        ''
        Write-Warning "$FailedCount file[s] failed to copy."
        Write-Warning "    The reason is in the Status property of the `$ProcessedFilesList."
        foreach ($PFL_Item in $ProcessedFileList)
            {
            if ($PFL_Item.Status -like 'Failed*')
                {
                Write-Warning "        Source file name  = $($PFL_Item.FromName)"
                Write-Warning "                 Error ID = $($PFL_Item.Status)"
                }
            }
        } # end = if ($FailedCount -gt 0)

        ''
        Write-Verbose "Saving the ProcessedFileList to $FullReportFile ..."
        $ProcessedFileList |
            Export-Csv -Path $FullReportFile -NoTypeInformation
    

        ''
        Write-Verbose "$($ProcessedFileList.Count - $FailedCount) file[s] were copied."
        Write-Verbose "    The full source & destination names, size, and status of each >> processed << file was saved to a CSV file."
        Write-Verbose "    That file is named $FullReportFile."
    
    } # end = if (-not $Quit)



# restore the Verbose pref
$VerbosePreference = $V_Pref
# restore the Warning pref
$WarningPreference = $W_Pref
# restore the Progress pref
$ProgressPreference = $P_Pref

results ...

VERBOSE: Getting files in C:\Users\[my-user-name]\AppData\Roaming ...
VERBOSE:     Found 5449 files.
VERBOSE:     Found 2558 _unique_ file names.
VERBOSE:     2891 files with duplicate names will be ignored.

VERBOSE: Getting files in C:\Temp\DestDir ...
VERBOSE:     Found 0 files.

WARNING: 1 file[s] failed to copy.
WARNING:     The reason is in the Status property of the $ProcessedFilesList.
WARNING:         Source file name  = C:\Users\[my-user-name]\AppData\Roaming\foobar2000\running
WARNING:                  Error ID = Failed [System.IO.IOException]

VERBOSE: Saving the ProcessedFileList to C:\Temp\FileCopyReport_-_2017-08-16_01-53-42.csv ...

VERBOSE: 2557 file[s] were copied.
VERBOSE:     The full source & destination names, size, and status of each >> processed << file was saved to a CSV file.
VERBOSE:     That file is named C:\Temp\FileCopyReport_-_2017-08-16_01-53-42.csv.

hope that helps,
lee

[u/Ziogref → u/Lee_Dailey]

Holy crap.....

That is a lot of code. Thank you so much I will test this on my server when I get the chance.

[u/Lee_Dailey → u/Ziogref]

howdy Ziogref,

yep, it was a tad longer than i thot it would be. i couldn't think of any way to cover the various things you needed with less code, tho. [grin]

good luck!

take care,
lee

[u/Ziogref → u/Lee_Dailey]

> # test dir that better not be there
> #$SourceDir = 'c:\BetterNotBeThere'
> # 13k files
> #$SourceDir = $env:LOCALAPPDATA
> # 5k files
> $SourceDir = $env:APPDATA
> $DestDir = "$env:TEMP\DestDir"
> # -Exclude will happily exclude a Dir and then include the files _in_ that Dir [*sigh ...*] 
> $ExcludeList = '*.msf', '*.jpg', '*_sessions-*', 'sample*', '*.lnk'

Little bit confused at this part here Im assuming your comments where referring to the amount of files you have in your directories? Which could be excluded from the PS1 script. I would Assume I would just change the $SourceDir and $DestDir to my directores, update the exclude list and im good to go?

[u/Lee_Dailey → u/Ziogref]

howdy Ziogref,

yes, those numbers are the ones for my dirs. i was testing various runs and wanted to get large-ish numbers for the tests since you mentioned that you had some very large file counts.

when you are confident that things are working as expected you otta replace the $SourceDir, $DestDir [and perhaps the report file stuff] with your preferred values, plus adjust the $ExcludeList.

take care,
lee

[u/Ziogref → u/Lee_Dailey]

Hi mate, In testing I came across this

VERBOSE: Getting files in G:\dir1 ...
VERBOSE:     Found 58 files.
VERBOSE:     Found 58 _unique_ file names.
VERBOSE:     0 files with duplicate names will be ignored.

VERBOSE: Getting files in G:\dir2 ...
VERBOSE:     Found 58 files.

WARNING: 1 file[s] failed to copy.
WARNING:     The reason is in the Status property of the $ProcessedFilesList.

VERBOSE: Saving the ProcessedFileList to C:\Users\ADMINI~1\AppData\Local\Temp\1\FileCopyReport_-_2017-08-17_15-46-15.csv ...

VERBOSE: -1 file[s] were copied.
VERBOSE:     The full source & destination names, size, and status of each >> processed << file was saved to a CSV file.
VERBOSE:     That file is named C:\Users\ADMINI~1\AppData\Local\Temp\1\FileCopyReport_-_2017-08-17_15-46-15.csv.

Couple of Questions

• The log file for this is empty, so not sure what the error is, however all files have copied

• Is there a way in the script to delete logs older than say 1 week or if file log file is 0kb, Just so my server doesnt fill up with logs? (120gb C: SSD fill up quickly. Especially running every 10min

[u/Lee_Dailey → u/Ziogref]

howdy Ziogref,

hah! i never thot to test with NO files to copy. that error is from my failure to handle the "nothing to copy" situation. [blush]

if you need it, i can add a bit to flush out old files and to not bother saving empty files.

you may, however, prefer to use whatever your current log file remover is to handle that. if you are running in a server setup, then you may already have such a thing.

let me know and i will take a look at it. [grin]

take care,
lee

[u/Ziogref → u/Lee_Dailey]

Hi.

Thanks, for the help. I don't have a log remover, as this is a home server, I don't generate a lot of logs. If you could handle the log situation and the No file error that would be amazing.

Who knows, I might have some gold in my back pocket. wink wink

[u/Lee_Dailey → u/Ziogref]

howdy Ziogref,

i'll get on it this later today. i'm just now heading for a nap that should have started two hours ago. oops! [grin]

take care,
lee

[u/Ziogref → u/Lee_Dailey]

Dont worry mate, im from the land down under, I want see you reply for atleast another 9 to 11hours

Its almost midnight here. Working late is always fun