A good, easy, peer-to-peer, free Dropbox-replacement system for the modern paranoid age

January 3, 2019 — April 11, 2024

computers are awful
computers are awful together
concurrency hell
Figure 1

My favourite file synchronisation tool. Think dropbox, but without a server, or a third party, or any need to spend money on anything apart from hard drives.

Choose this if… You have a collection of various folders that you need shared to various different machines, and you would like many of the different machines to be able to edit them. You don’t need a server and thus you are happy for syncing to happen if and when the peers are online. And you don’t care about iOS. And you don’t mind wasting a few CPU cycles (not many by modern standards) on making it decentralised. I use syncthing for synchronising my music production files across my studio machines, studio backup machines and gig machines.

Syncthing has an elegant decentralised peer-to-peer design. It is mostly simple and friendly to use in practice, although I spent a long time feeling intimidated by the manual before just diving in and finding that out.

Granularity is per-folder-per-machine — each shared folder (and all sub folders) is a separate share. It doesn’t support iOS. It doesn’t support archiving stuff to USB keys or semi-offline stores, or multiple copies of the same folder on one machine.

Stated design criteria:

  • Private. None of your data is ever stored anywhere else than on your computers. There is no central server that might be compromised, legally or illegally.

  • Encrypted. All communication is secured using TLS. The encryption used includes perfect forward secrecy to prevent any eavesdropper from ever gaining access to your data.

  • Authenticated. Every node is identified by a strong cryptographic certificate. Only nodes you have explicitly allowed can connect to your cluster.

Editorial note re: privacy: Note that, if any of your machines are compromised your attacker still gets the data on that machine. Syncthing is not magic. It’s simply that you don’t have to worry about a whole other copy of your data sitting unencrypted on a server owned by some faceless third party who may or may not have acceptable confidentiality practice.

1 Useful config

I find it is better to set the following files ignored in the .stignore, otherwise it synchronises a little too aggressively.

// generic

// From Windows
System Volume Information

// From OS X

// From Linux

in fact, it is better to put these in another file and to #include that file from .stignore; then the other file can be shared between various peers to ensure they have similar sync lists.

#include _stignore

If you are synchronising to an external drive or SD card which due to its OS or its file system has limited metadata support then some configuration is required to keep things tidy. Useful settings include setting modWindowS=2 and ignorePerms="true" which keeps the conflicts minimal even when you use e.g. an ExFAT external HDD.

2 UIs

There is a CLI syncthing manager for remote cloud instances, syncthingmanager. It has a macOS client. Revised recommendation: don’t bother with platform-specific clients on desktop. the built in web UI is really good.

3 Nitty gritty

Syncthing also has file versioning and such, but cryptographic signing of versions and guaranteeing consistent snapshots and so on is not a front-and-centre feature.

3.1 Case sensitivity

The major gotcha is that syncing between case insensitive and case sensitive file systems is broken and can delete data. That’s right, this app works beautifully, smoothly and easily except that the moment you use it to sync between linux and macOS or Linux and Windows (which have different case sensitivity per default, although it’s a long story) stuff goes horribly wrong. This works as of version 1.9.0, sort of. It could be better documented. This solution is a clunky — for me at least, often if one machine renames a file changing only the case, then other machines no longer delete the data, which is a necessary improvement. Rather, some machines become confused about the rename and demand that you manually “fix” it, by renaming rather than syncing the rename across the machines. It seems you need to do this on each of the machines. I’m sure that there is some sensible reason that this happens, to do with the intrinsic difficulties of distributed something-or-other, but it does lead to much manual labour to do something that looks automatic and trivial. Actually at some point this became usually-seamless and OK.

3.2 Metadata storage

A lesser problem is that syncthing uses a non-trivial amount of disk space in a central cache folder (i.e. outside where the files of interest are). This is something like 1% of the size of the data being backed up, which if you are syncing many terabytes does add up.

3.3 File conflicts

The exact model that this software uses to synchronise things is a little hard to explain. I found the following introduction from the forums where someone gets it wrong and a core developer corrects them, to be helpful: sync-conflicts.

3.4 Truncation bitrot

My most common way of losing data with syncthing is that it will sometimes truncate files; I think that this is probably not syncthing per se but rather the underlying filesystems; Once it has happened, though, the truncated file is obediently replicated to all the other machines, and the original file is lost. I can restore the original from the syncthing backup, of course, but if it is not a file I use often I won’t notice that it is truncated until the backup has long since been purged.

3.5 Syncing is per-server, per-share not per-drive or per-file

One bit of friction I did not think about at the start is that this is a swarm of servers but not actually a swarm of drives. The non-obvious (to me) result is that if I want to have two copies of a thing, I need to have two servers syncing the thing. This is usually sort-of what I want, but bear this in mind when seeing if it fits your needs.

The other corollary of this model is that it is quite hard to get one file from the swarm; the machine really wants to give me all of them. That is not fast; if the folder is a 1tb folder then it might be a terabyte worth of download before I can get the 10kb file that I actually want. There are ways around this, but they are all swimming against the flow of the app.

The android client at one time supported partial sync, but I have not checked that it still does recently.

Pre-seeding syncs is possible, but do check the manual first.

3.6 Moving shares

It is not terribly convenient to move the location of a share on a given server.

4 Windows

On windows with Chocolatey, Syncthing will need to be turned into a startup service to run all the time.

New-Service Syncthing $env:ChocolateyInstall/bin/syncthing.exe -StartupType Automatic
Start-Service Syncthing

NB, the syncthing docs suggest this is a security hole, and indeed that a better task manager might be required. Perhaps running it inside a wrapping app via synctrayzor would be less overall messing about?

5 Cloud deployment

So I just said that it can create a self-healing serverless swarm. What if we want to create a server to join the swarm, to provide a little cloud redundancy? TBC.

I would really like it if official Syncthing sponsor Kastelo Inc. would host cloud deploys, but sadly their main service is technical support, which is less helpful for me personally.

There is now a facility to share data to untrusted hosts as encrypted blobs: