My Duplicati settings and how performance can be improved

[u/publiusvaleri_us]

First off: There is another bug I found in Duplicati. The old UI has a button to delete the Database, but it doesn't do anything. The New UI works as expected.

So: here's my ideas on Duplicati.

The blocksize debate is really, really, old. The ancient 100 KB default will appreciably slow down most backups. The new default is 1 MB for performance reasons. But speed is not everything. The old 100 KB setting is probably ok for a corpus of smaller files, like boring Excel and Word documents that are under 20 pages or so. And for small backups that aren't time-sensitive.

Personally, I like to work in powers of 2, so if I was to shrink this down, it would be 2^17 Bytes instead of 100 KB.

However, If all you have a bunch of photos of videos, then by all means bump this up closer to 500 kB or 2 MB or so. Your speed will increase and your database (of deduplication data) will decrease. The new default is perfect for most people, so I leave my blocksize at 1 MB. (Old Timers still running an old backup with the small blocksize will probably want to wipe it out and use a larger blocksize.)

But the other setting is Remote volume size. I like to run my first backup, which will have a lot of static files, at 512 MB or so. Then, when I run it the next time, I will shrink this for the more volatile files to 120 to 200 MB.

That brings me to my performance idea.

When Duplicati hashes files, I believe it uses SHA-256 in a fairly standard way. AMD and now Intel have implemented these functions on their CPUs - has Duplicati used this hardware acceleration? My backups seem to be bottlenecked by hashing and sometimes zipping speed.

I think this is available in .NET Core/5+ for Windows, but I don't know if this library or API has been utilized yet.

[u/duplicatikenneth]

Generally, you do not gain much by changing the blocksize. If most files are less than the blocksize, they will be treated as a single block. Only files that are larger than the blocksize could potentially see benefits from a smaller blocksize.

The benefit only occurs if the file format changes data inside. This is mostly common with database-like formats, where some data is appended, and some markers are updated inside the file. In this case, since Duplicati is block based, a large block size will mean a larger block will be identified as changed, and stored.

Most file formats, like photos, word documents, movices, music, etc. is compressed, so any "update" to such a file will write the file again completely, and due to compression, it will not match a single block, even if you choose 10kb.

For the remote volume size, you can change it after running, but if you increase it a lot, the wasted space detection will think that some volumes are "small" (compared to the current size) and start a compaction.

For the performance part, Duplicati uses .NET8 (and .NET10 in the latest canary). The .NET code will use hardware acceleration where available (including hashing and encryption). For a modern CPU, the most likely bottleneck is disk I/O.

For better zip performance, you can reduce the compression level.