SSSync

When syncing data to a client, you can express what the client needs by either:

1. Live queries (arbitrary, requires a running server)
2. A pre-calculated subset of data (user’s working set)

Live queries are in most ways the superior way to do things. They combine the freedom of “just make an API request” with the performance a built-in local cache and the consistency of transactional updates. There’s only 4 (small) downsides with existing libraries:

1. They involve stateful servers for satisfying queries (usually pretty scalable read replicas of the main database), which have RAM requirements proportional to O(row results) instead of using the client as the data source.
2. Are usually tied to specific databases (often Postgres) or are a database too (Convex, Jazz). This is mostly a good idea, just constraining.
3. None of the existing solutions have support for SSR yet.
4. Only a couple of them support offline mutations.

I expect that LLMs will make the first two problems smaller over time, by rewriting the server code in more memory-efficient languages and make it trivial to add new database adapters.

Meanwhile, the downsides of a pre-calculated subset are many:

1. Obviously, it only works for apps where the data a user sees is a neat rarely-changing subset.
2. Data, once fetched, continues to build up on the client even if none of the UI is requesting it anymore.
3. Your subset-calculating code can contain bugs.
4. Most existing solutions use JSON document models instead of relational.
5. Most existing solutions require loading the user’s working subset the first time they open the app/site. This is untenable past a certain size of data.

So why work on it?

Linear is (to my knowledge) the only subset-based sync solution that also has efficient data loading. They do so by combining a fetch-when-needed (instead of load upfront) protocol with a change stream. My library will try to commoditise this capability. With support for SSR, unlike Linear.

In addition, the way my approach handles data fetching allows it to be used for arbitrary apps, not just ones that want a persistent local cache. Github, for example, is a mixture of arbitrary, temporarily-used data (when you visit a random repo) and well-defined subsets that could be cached locally (your own repos, your org’s repos) and something like that would be possible to build with SSSync. (Doing so would worsen a few consistency guarantees in the local database, but that’s what HTTP-based web apps do anyway).

Some other reasons:

1. Be able to make apps that have zero idle cost by running on serverless compute and storage.
2. Build things like fuzz testing, network failure simulation, etc. into the library dev tools.
3. Have an open source library of my own.

System components

Client query API

My requirements are:

• Ability to clearly know that something is a “fetch-by-id + relations” that can be sent to the server if not found locally.
• Easy to make performant on IndexedDB.
• Easy to build an efficient subscription system for.

This suggests two simple yet distinct query patterns:

// Get a single item, with nested relations
sssync.db.posts.byId([id], ["authors"])

// Get a range, filtered by a Javascript function
sssync.db.posts.all(() => a.date < TODAYS_DATE)

Client storage layer

IndexedDB has the most browser compatibility and least need for custom tooling or bundler complications. Especially for a v1, there’s no reason to reinvent storage yet. I will also avoid using indexes (Javacript-only filtering) and migrations (just blow away old databases and remake a new one in a new namespace each time).

I need to be careful about serialization though.

Data-fetching protocol

Fetch by id + relations

Most UI in apps is relational. The nice thing about “get by id, and relations of that item” is that it makes for a simple protocol. The server knows exactly what kind of requests to expect since they all look like:

{type: "posts", id: "uuid", related: ["authors", "comments"]}

It’s also super easy to cache. A request for an item can be conclusively satisfied just be checking if it exists in the local database, no invalidation needed. And relations can be tracked too, each time a server satisfies a relational fetch.

Named loader functions

But how do you build a cache around things fetched in arbitrary bunches, in truncated lists, or that use arbitrary server logic to put together?

I propose a simple system of named functions, producing cache keys that are a combination of the function name and args. These functions can be run once, on a debounce, or manually whenever the developer wants — and their results dumped into the local database.

// An example of a simple fetch()-based loader
sssync.loaders = {
  latestPosts: (user) => fetch("/posts/" + user)
}

// Component using the loader
function Posts (){
  const loaded = sssync.load.latestPosts({refresh: 3000})
  const posts = sssync.db.posts.all()

  return (
    <div>
      {posts.slice(0, 50).map(post => <h2>{post.title}</h2>)}
      {!loaded && (<div> This list might be stale! </div>)
    </div>
  )
}

Sync logic

The only sync path is an ordered stream of updates that the server is responsible for fanning out to clients. Given the last_sync_id from any client, it must return updates and a fresh last_sync_id in a single response. If it wants to batch the data into multiple responses, it may add a more_updates: true parameter to the JSON payload to tell the client to make more requests.

I need to finalise the shape of updates that must come back from a sync request. It should be possible to implement this protocol over Websockets and SSE too.

Write path

I prefer semantically named and versioned change objects because they force best practices while also solving the schema migration problem. Credit to Livestore for the inspiration.

postCreated: {
   name: 'v1.PostCreated',
   schema: object({ id: string(), text: string() }),
 })

sssync.write(
  "v1.postCreated",
  {id: NEW_UUID, title: "New post about SSSync"}
)

When triggered, these named changes will be immediately converted into actual state updates using projector functions on the client and broadcasted to the server in the background.

On the server, you can write whatever code you like to handle them. This approach to conflict management is called server reconciliation.

If you feel like commenting on this post, send me an e-mail at judah@joodaloop.com. I check all mail and respond within 2 days.