Designing a Local-First Filesystem Sync Engine

A filesystem that talks to cloud storage has two audiences: the remote service and every local program that assumes files behave normally.

Calling an API is the easy part. The hard part is deciding what `open`, `read`, `write`, `rename`, and `delete` mean when the network is slow, the remote tree is stale, or the process restarts halfway through a change. The local interface expects immediate, durable behavior; the cloud side offers eventual observations and fallible requests.

Local-first design begins by choosing which guarantees the local system can make on its own. It should not pretend a remote write is complete when it is only queued, but it also should not freeze an editor until an upload finishes. That distinction shapes the metadata model, cache, sync engine, CLI, and error reporting.

A remote file can exist in the mounted tree without its content being present locally.

Persistent metadata records the path, identity, type, size, timestamps, and hydration state. A local mirror cache stores bytes only after a download or local write. Directory listings and metadata calls can therefore answer from local state while an actual open triggers hydration when needed.

This split keeps cold starts and traversal practical, but it creates an invariant: the metadata database must never imply that cached content is valid when it is absent or belongs to a different remote version. Hydration is a state transition, not just a download function.

Convenience features become dangerous when they invent a second meaning of the same state.

A bulk “download this subtree” command could call the cloud API directly, but then it might bypass the driver’s cache rules and metadata updates. A safer design scans for eligible unhydrated entries, respects the same allow and exclusion paths, then opens each file through the mounted filesystem.

The command gets progress, dry-run, interruption, and rerun behavior, while the system keeps one hydration implementation. Reusing the ordinary file-open path is not the fastest shortcut to code, but it prevents two components from disagreeing about when a file is truly local.

For local-first storage, “saved” and “uploaded” are related but different facts.

The local operation can update the mirror and persistent metadata first. Once that state is durable, the calling application can continue. A background worker later pushes the remote change, records success, or retains enough context to retry. The user should be able to see pending and failed convergence rather than receiving false certainty.

This makes restarts part of the design. A queued operation that only exists in memory is not local-first; it is a race between the network and process lifetime. Persistent identifiers, operation state, and tombstones let the sync loop reconstruct what remains to be done without guessing from filesystem timestamps alone.

The remote tree can change while local work is waiting to synchronize.

A blind refresh may overwrite a local modification. A blind local preference may resurrect a remote delete. The sync engine needs enough evidence to distinguish a new remote object, an observed update, a local pending write, and an explicit deletion.

There is no universal conflict rule that makes this disappear. When one side cannot be proven authoritative, preserving data and surfacing the disagreement is safer than silently selecting a winner. Sync is mostly the work of carrying intent across two incomplete timelines.

A technically correct state machine is still unusable if nobody can tell what it is doing.

Mount status, service state, hydration progress, recent errors, pending work, and logs should be available through one CLI. Configuration should define mount, cache, sync, and exclusion paths once. Error messages should point toward an inactive service, missing state, expired authentication, or inaccessible mount instead of exposing an internal traceback as the product.

This is where systems and product engineering meet. The state model defines what is true; the operator interface makes that truth visible enough to trust.