J

Local-first Software

Since late 2023, I’ve worked on at least four apps of different complexity, all of them based on using browser-based storage as a local cache for data. Like others, I respect the ideals of the unofficial Local-first manifesto.

But I have developed reservations around just how “correct” it’s claims are.

Browser limitations

Storage limits

LocalStorage, with it’s 5MB limit can only be used by apps that are certain to have small data requirements. 5 megabytes is just around a million words of text. Enough for a low-use writing app, enough for a journal app where you write ~500 word posts everyday over a period of 5 years. Decent, but not enough for serious apps.

IndexedDB has browser-specific (and maybe OS-specific) limits that are not well-defined. Popular concensus is that you can rely on it for about 1GB worth of data. It’s generally not a great idea to dump so much data on in the client’s browser (especially if they’re going to be using mobile devices).

Speed

To get over the slowness of IndexedDB queries, this guy built AbsurdSQL. It really is absurd, you ship SQLite as a WebAssembly binary to the browser (5MB worth of code!) and use it to run SQLite on top of IndexedDB (which in turn runs on top of SQLite!).

For anything less than many hundreds of items though, you should be fine with using idb, or Dexie.js (I use the latter).

SQLite

It’s such good technology man. The original local storage.

You can now do fun things like replicate your SQLite database with Litestream, or use Fly’s LiteFS. The guys at TursoDB even created a fork called LibSQL that supports HTTP transactions.

CRDTs

“Local-first” does not mean giving up on collaboration, that’s not a tradeoff worth making anymore. And so collaboration through CRDTs is one of the primary research areas for the Ink & Switch team, and they’re doing lots of good work.

They work great for text-editing, especially plain text.

Yjs is faster, but Automerge is catching up quickly.

Despite all this, I don’t think CRDTs as the solution. I’ll be the first to admit that I don’t understand them well enough, but it feels like there are some fairly “basic” things that they just can’t do yet. Moving elements around in a list is still an open problem).

“My theory is that the big blocker to a smarter local technology is the same as the reason we’ve largely lost the independent web: a deep-rooted human desire for convenience. People are lazy, and although they like ideas like longevity and privacy in the abstract, we’re willing to sacrifice them for systems that make things easy. Big cloud platforms are more than happy to take advantage of this defect in human mentality because it’s a win-win from their perspective – it’s better for them if they own your data, so unless users are demanding an alternative at deafening volume, they’d never go about it any other way.” Brandur

A local database

I think all sync engines are moving toward a world where you don’t need separate in-memory state management. If you have a local reactive database and it’s on the main thread there’s no reason for separate state management. The db can efficiently maintain queries that are sorted, filtered, grouped for you.

So to the extent that Solid wants to manage state it’s going to be in conflict with these systems. To the extent that it wants to very efficiently update views, it’s going to be a great match.

We often find ourselves fighting react and its invalidation model. I still think there is need for a “it’s just views” ui library that does granular updates very well.

Aaron Boodman

Migrations

Your users have different versions of your data model on different devices. How do you write a migration script to get each of them up to speed? You have to set things up do older versions of the app can always work with changes to the schema.

Safety

People don’t trust themselves to manage their own data. I know I don’t. My writing app of choice works perfectly offline, offers multiple exporting options, but I still pay $15/year for syncing it to my iCloud account. Because there’s no way I want to risk losing 2 years worth of writing by losing my laptop.

It feels like a charisma trap.

The tech has limits. The original Ink & Switch post has hit HackerNews half a dozen times, and yet, we’re yet to see any of the ideas actually brought to production. Except in the Muse app, which itself is an I&S offshoot.

Every non-trivial usecase needs it’s own custom CRDT implementation.

Compromise

There’s two main features of local-first software that make them noticeably better than apps that rely on the cloud:

  1. Offline usage
  2. Data ownership

In developer discourse, the term CRDT sometimes gets thrown around as a synecdoche for a broader set of techniques to enable Figma-like collaborative features. But when we started talking to dozens of companies building ambitious browser-based apps, we found it rare for apps to use true CRDTs to power multiplayer collaboration. You might not need a CRDT

If a user logs in a week later, how bad would it be if we overwrote their local storage (where necessary)?

What Figma does.

References

You might not need a CRDT and it’s Hackernews discussion.

Are CRDTs suitable for shared editing?

https://marknotfound.com/posts/reverse-engineering-linears-sync-magic/