Syncthing

A good, easy, peer-to-peer free Dropbox-like system for the modern paranoid age

January 3, 2019 — July 4, 2023

computers are awful
computers are awful together
concurrency hell
distributed
diy
p2p
Figure 1

My favourite file synchronsation tool. Think dropbox, but without a server, or a third party, or any need to spend money on anything.

Choose this if… You have a collection of various folders that you need shared to various different machines, and you would like many of the different machines to be able to edit them. You don’t need a server and thus you are happy for syncing to happen if and when the peers are online. And you don’t care about iOS. And you don’t mind wasting a few CPU cycles (not many by modern standards) on making it decentralised. I use syncthing for synchronising my music production files across my studio machines, studio backup machines and gig machines.

Syncthing has an elegant decentralised peer-to-peer design. It is mostly simple and friendly to use in practice, although I spent a long time feeling intimidated by the manual before just diving in and finding that out.

Granularity is per-folder-per-machine — each shared folder (and all sub folders) is a separate share. It doesn’t support iOS. It doesn’t support archiving stuff to USB keys or semi-offline stores, or multiple copies of the same folder on one machine.

Stated design criteria:

  • Private. None of your data is ever stored anywhere else than on your computers. There is no central server that might be compromised, legally or illegally.

  • Encrypted. All communication is secured using TLS. The encryption used includes perfect forward secrecy to prevent any eavesdropper from ever gaining access to your data.

  • Authenticated. Every node is identified by a strong cryptographic certificate. Only nodes you have explicitly allowed can connect to your cluster.

Editorial note re: privacy: Note that, if any of your machines are compromised your attacker still gets the data on that machine. Syncthing is not magic. It’s simply that you don’t have to worry about a whole other copy of your data sitting unencrypted on a server owned by some faceless third party who may or may not have acceptable confidentiality practice.

1 useful config

I find it is better to set the following files ignored in the .stignore, otherwise it synchronises a little too aggressively.

// generic
*.part
*.tmp

// From Windows
$RECYCLE.BIN
System Volume Information
$WINDOWS.~BT
pagefile.sys
desktop.ini

// From OS X
Icon?
.Spotlight-V100
.Trashes
(?d).DS_Store
.fseventsd
(?d)._*


// From Linux
lost+found
.gvfs
.local/share/trash
.Trash-*
.fuse_hidden*

Pro tip: in fact, it is better to put these in another file and to #include that file from .stignore; then the other file can be shared between various peers to ensure they have similar sync lists.

#include _stignore

Additional pro-tip: If you are synchronising to an external drive or SD card which due to its OS or its file system has truncated metadata then some configuration is required to keep things tidy. Useful settings include setting modWindowS=2 and ignorePerms="true" which keeps the conflicts minimal even when you use e.g. an ExFAT external HDD.

2 UIs

There is a CLI syncthing manager for remote cloud instances, syncthingmanager. It has a macOS client. Revised recommendation: don’t bother with platform-specific clients on desktop. the built in web UI is really good.

3 Nitty gritty

Syncthing also has file versioning and such, but cryptographic signing of versions and guaranteeing consistent snapshots and so on is not a front-and-centre feature.

The major gotcha is that syncing between case insensitive and case sensitive file systems is broken and can delete data. That’s right, this app works beautifully, smoothly and easily except that the moment you use it to sync between linux and macOS or Linux and Windows (which have different case sensitivity per default, although it’s a long story) stuff goes horribly wrong. This works as of version 1.9.0, sort of. It could be better documented. This solution is a clunky — for me at least, often if one machine renames a file changing only the case, then other machines no longer delete the data, which is a necessary improvement. Rather, some machines become confused about the rename and demand that you manually “fix” it, by renaming rather than syncing the rename across the machines. It seems you need to do this on each of the machines. I’m sure that there is some sensible reason that this happens, to do with the intrinsic difficulties of distributed something-or-other, but it does lead to much manual labour to do something that looks automatic and trivial. Actually at some point this became usually-seamless and OK.

A lesser problem is that syncthing uses a non-trivial amount of disk space in a central cache folder (i.e. outside where the files of interest are). This is something like 1% of the size of the data being backed up, which if you are syncing many terabytes does add up.

The exact model that this software uses to synchronise things is a little hard to explain. I found the following introduction from the forums where someone gets it wrong and a core developer corrects them, to be helpful: sync-conflicts.

4 Windows

On windows with Chocolatey, Syncthing will need to be turned into a startup service to run all the time.

New-Service Syncthing $env:ChocolateyInstall/bin/syncthing.exe -StartupType Automatic
Start-Service Syncthing

NB, the syncthing docs suggest this is a security hole, and indeed suggest that a better task manager might be required. Perhaps running it inside a wrapping app via synctrayzor would be less overall messing about?

5 Cloud deployment

So I just said that it can create a self-healing serverless swarm. What if we want to create a server to join the swarm, to provide a little cloud redundancy? TBC.

I would really like it if official Sycnthing sponsor Kastelo Inc. would host cloud deploys, but sadly their main service is technical support, which is less helpful for me personally.

There is now a facility to share data to untrusted hosts as encrypted blobs: