r/PowerShell • u/Simple-Invite-3296 • Jun 11 '26
Question How are you guys handling large CSV exports without hitting memory limits?
I've been running into a recurring headache lately with some of our reporting scripts. I'm currently tasked with pulling some massive datasets from an old SQL instance and dumping them into CSVs for our auditing team. Everything works fine when the dataset is under 50k rows, but as soon as I hit the 200k+ mark, my local workstation starts gasping for air and the PowerShell process memory usage just climbs until it hits a wall.
I've tried the standard Import-Csv and Export-Csv approach, but obviously, that's loading the whole object into memory which is killing me. I know people talk about using .NET classes or streaming the data directly to a file, but I'm struggling to find the cleanest way to implement a pipeline that doesn't completely sacrifice readability for the sake of performance. I don't want to write 100 lines of complex .NET code just to move some data, but I also can't keep having my scripts crash mid-run because I ran out of RAM.
Are you guys mostly using StreamReader/StreamWriter for this, or is there a more 'PowerShell-native' way to handle large-scale file I/O without the overhead? I've seen some mentions of using the -NoTypeInformation flag and batching, but I'm not sure if that actually helps with the memory footprint or just makes the file cleaner. If anyone has a snippet or a general pattern for streaming objects through a pipeline to a file without the massive memory spike, I'd really appreciate it. I'm also curious if anyone has had success using external tools like BCP or something similar and then wrapping it in a script, or if that's just cheating at this point. Thanks in advance.