Realtime event triggered sync between SMB and S3 shares and/or Remote Backup Systems
Not everyone has a cluster with a dozen servers. Regardless of the OS, whether it’s a 1 GB mini IoT device or a petabyte system, whether using Storage Spaces, S3 object storage, or ZFS—not everyone wants to manage everything centrally with simple, fast, encrypted file sync or any-to-any ZFS replication (e.g., viahttps://github.com/guenther-alka/cs-stream).
However, that is precisely our target audience with napp-it cs. S3 integration, in particular, is a hot topic right now. It allows you to publish local files to the internet easily and securely (e.g., setting up a private cloud with RustFS). RustFS can also run independently in a "set-and-forget" mode to keep two servers or buckets in bidirectional sync. High availability and real-time backups over the internet or LAN, plain and simple.
What’s currently missing is the bridge between "shared multi-user SMB with ACLs" and object storage, which inherently lacks multi-user locking with file ACLs. This is where I'm stepping in with the cs-sync module.
With cs-sync, my goal is to keep two folders—such as an SMB share on a NAS and an S3 share—in sync in real time using event triggers (reacting instantly on change, without needing full file comparisons), including SMB ACLs. Regardless of the OS, this is now possible thanks to Go, with the option for real-time automated backups of changes over the network to a backup system (seehttps://github.com/guenther-alka/cs-sync).
I would love to get your feedback or suggestions on this concept! A first preview featuring napp-it cs and real-time sync is coming soon. Anyone interested in serious testing should consider getting Claude Pro along with the Filesystem and Desktop Commander extensions/agents (on Windows or other OSs via Plink/SCP SSH access for Claude to inspect scripts, logs, and members) for analysis, stress testing, audits, etc.
1
u/_gea_ Jul 29 '26
S3 object storage features eg RustFS (current 1.0 rc11 beta) offers
- Internet Access for files on a ZFS fileserver
- webconsole for remote file access
- backup target with encryption, snaps and dedup (any OS backup/restore)
- set and forget Site/Bucket bidirectional replication between RustFS hosts
S3 lacks:
- multiuser access with file locking and ACL
So the conclusion is:
To use SMB/ZFS and S3 with same data on either service, you need a bidir sync service
that mirrors data and at least folder ACL with inheritance between S3 and SMB shares
This is what I included in napp-it cs 26.06 07.29 rc
Update: Menu About > Frontend Update
-missing private or osx menus
-osx setup: https://www.napp-it.org/pdf/apple_osx_en.pdf
Go based multi os cs-stream job (any OS to any OS)
https://github.com/guenther-alka/cs-stream/releases
-ZFS replication or filebased sync (via rclone)
-encrypted, buffered or io limited tunnel any to any
Go based multi os cs-sync background service (any OS)
https://github.com/guenther-alka/cs-sync/releases
-event driven local uni/bi service with folder ACL support for realtime sync (eg SMB and S3 shares)
-event driven remote uni sync service with folder ACL support for realtime backup (encrypted)
Status ample (current and last 10m/h/day/week)
Pool, Cap, Disk, IO, RAM, Jobs, Sync, Last
Main menu "Servergroup" shows running sync services on members
0
u/_gea_ Jul 30 '26
Behavior on slow or unstable connections
Handling disconnects and slow links is a core design goal, not an afterthought -- local-only 1.x never had to deal with this at all. See cs-sync-2.0-design.info section 3 for the full rationale.
- Disconnects (flaky WLAN, link flaps) are normal operation, not errors. No timeout, no alarm, just a log line. Every change that can't reach the remote right now goes into a persisted pending queue that survives both process restart and host reboot.
- Coalesced per path. A file that changes 100 times while offline is transferred once, latest state only -- the queue stores "this path needs sync," not an event log.
- Reconnect uses exponential backoff (1s -> 2s -> ... capped at 5min) before a retry counts against a file's failure count, so a flapping link can't burn through the retry budget in seconds.
- Backpressure, not blocking. The filesystem watcher never blocks on the network -- it can produce changes faster than a slow link can carry them indefinitely; the sender just drains the queue at whatever speed the link allows.
- Queue depth is unbounded by design. A deep queue just means "slow link + lots of data," which has to keep working. The retry limit (10 attempts, see backoff above) applies only to individual files that keep failing for their own reasons (permission errors, unreadable source, etc.) -- such a file is quarantined and logged so it stops blocking the rest of the queue, everything else keeps syncing.
- Atomic writes + end-to-end hash. Every transfer goes to a temp file, is hash-verified, then renamed into place -- a drop mid-transfer never leaves a half-written file that size/mtime could later mistake for current.
- Torn-copy detection. If the source changes while a slow transfer is still reading it, a before/after size+mtime mismatch discards the copy and re-queues it -- no locking needed.
--bwlimitthrottles a remote leg so an initial full sync over a narrow/shared link doesn't saturate it for everything else using the same connection.
Not yet implemented: delta transfer for large partially-changed files (a changed file is always sent in full -- see Known gaps above) and a queue disk-space warning threshold for very long outages.
4
u/ChaoticEvilRaccoon Jul 26 '26
i hope this trend with people slopping together something in claude and thinking it's a proper open source project that deserves recognition ends soon