Syncthing
A good, easy, peer-to-peer, free Dropbox-replacement system for the modern paranoid age
January 3, 2019 — April 11, 2024
My favourite file synchronisation tool. Think Dropbox, but without a server, or a third party, or any need to spend money on anything apart from hard drives.
Choose this if… You have a collection of various folders that you need shared to various different machines, and you would like many of the different machines to be able to edit them. You don’t need a server and thus you are happy for syncing to happen if and when the peers are online. And you don’t care about iOS. And you don’t mind wasting a few CPU cycles (not many by modern standards) on making it decentralised. I use Syncthing for synchronising my music production files across my studio machines, studio backup machines and gig machines.
Syncthing has an elegant decentralised peer-to-peer design. It is mostly simple and friendly to use in practice, although I spent a long time feeling intimidated by the manual before just diving in and finding that out.
Granularity is per-folder-per-machine — each shared folder (and all subfolders) is a separate share. It doesn’t support iOS. It doesn’t support archiving stuff to USB keys or semi-offline stores, or multiple copies of the same folder on one machine.
Stated design criteria:
Private. None of your data is ever stored anywhere else than on your computers. There is no central server that might be compromised, legally or illegally.
Encrypted. All communication is secured using TLS. The encryption used includes perfect forward secrecy to prevent any eavesdropper from ever gaining access to your data.
Authenticated. Every node is identified by a strong cryptographic certificate. Only nodes you have explicitly allowed can connect to your cluster.
Editorial note re: privacy: Note that, if any of your machines are compromised your attacker still gets the data on that machine. Syncthing is not magic. It’s simply that you don’t have to worry about a whole other copy of your data sitting unencrypted on a server owned by some faceless third party who may or may not have acceptable confidentiality practice.
1 Useful config
I find it is better to set the following files ignored in the .stignore, otherwise it synchronises a little too aggressively.
// generic
*.part
*.tmp
// From Windows
$RECYCLE.BIN
System Volume Information
$WINDOWS.~BT
pagefile.sys
desktop.ini
// From OS X
Icon?
.Spotlight-V100
.Trashes
(?d).DS_Store
.fseventsd
(?d)._*
// From Linux
lost+found
.gvfs
.local/share/trash
.Trash-*
.fuse_hidden*In fact, it is better to put these in another file and to #include that file from .stignore; then the other file can be shared between various peers to ensure they have similar sync lists.
If you are synchronising to an external drive or SD card which due to its OS or its file system has limited metadata support then some configuration is required to keep things tidy. Useful settings include setting modWindowS=2 and ignorePerms="true" which keeps the conflicts minimal even when you use e.g. an ExFAT external HDD.
2 UIs
There is a CLI Syncthing manager for remote cloud instances, syncthingmanager. It has a macOS client. Revised recommendation: don’t bother with platform-specific clients on desktop. The built-in web UI is really good.
3 Nitty gritty
Syncthing also has file versioning and such, but cryptographic signing of versions and guaranteeing consistent snapshots and so on is not a front-and-centre feature.
3.1 Case sensitivity
The major gotcha is that syncing between case insensitive and case sensitive file systems is broken and can delete data. That’s right, this app works beautifully, smoothly and easily except that the moment you use it to sync between Linux and macOS or Linux and Windows (which have different case sensitivity per default, although it’s a long story) stuff goes horribly wrong. This works as of version 1.9.0, sort of. It could be better documented. This solution is clunky — for me at least, often if one machine renames a file changing only the case, then other machines no longer delete the data, which is a necessary improvement. Rather, some machines become confused about the rename and demand that you manually “fix” it, by renaming rather than syncing the rename across the machines. It seems you need to do this on each of the machines. I’m sure that there is some sensible reason that this happens, to do with the intrinsic difficulties of distributed something-or-other, but it does lead to much manual labour to do something that looks automatic and trivial. Actually at some point this became usually seamless and OK.
3.2 Metadata storage
A lesser problem is that Syncthing uses a non-trivial amount of disk space in a central cache folder (i.e. outside where the files of interest are). This is something like 1% of the size of the data being backed up, which if you are syncing many terabytes does add up.
3.3 File conflicts
The exact model that this software uses to synchronise things is a little hard to explain. I found the following introduction from the forums where someone gets it wrong and a core developer corrects them, to be helpful: sync-conflicts.
3.4 Truncation bitrot
My most common way of losing data with Syncthing is that it will sometimes truncate files; I think that this is probably not Syncthing per se but rather the underlying filesystems; Once it has happened, though, the truncated file is obediently replicated to all the other machines, and the original file is lost. I can restore the original from the Syncthing backup, of course, but if it is not a file I use often I won’t notice that it is truncated until the backup has long since been purged.
4 Windows
On Windows with Chocolatey, Syncthing will need to be turned into a startup service to run all the time.
New-Service Syncthing $env:ChocolateyInstall/bin/syncthing.exe -StartupType Automatic
Start-Service SyncthingNB, the Syncthing docs suggest this is a security hole, and indeed that a better task manager might be required. Perhaps running it inside a wrapping app via synctrayzor would be less overall messing about?
5 Cloud deployment
So I just said that it can create a self-healing serverless swarm. What if we want to create a server to join the swarm, to provide a little cloud redundancy? TBC.
I would really like it if official Syncthing sponsor Kastelo Inc. would host cloud deploys, but sadly their main service is technical support, which is less helpful for me personally.
There is now a facility to share data to untrusted hosts as encrypted blobs: