Offline-First Support

[b-barry]

Hey TanStack team! 👋

I've been loving TanStack DB so far - the reactive collections and optimistic mutations are exactly what I need. I'm wondering if you have any plans to support offline-first functionality?

It would be amazing to have:
Local persistence for collections
Ability to work completely offline
Auto-sync when connection returns

This seems like a natural extension of the existing optimistic updates system that's already so well implemented.
Is this something on your roadmap?

Would love to contribute to discussions around this if helpful!

Thanks for the awesome work so far! 🙌

[KyleAMathews]

Hey yeah! We definitely want to support offline use cases. Our next milestone is stabilizing all the APIs and fixing all the bugs so we're production ready but offline is definitely of interest past that.

My general thinking about offline is that it should always be opt-in and each app needs to do specific design work around how they want offline support to work. As there's a lot of nasty edge cases you need to design against e.g. an app comes back online after a few weeks w/ some old changes and overwrites a bunch of new work.

Even local persistence isn't "free" as it can add a lot of extra IO for all the writes & there are cases where reading from a local store can be slower than reading from the network — so it's again not obviously better for most apps unless they really need it.

---

To add offline support there's two parts — collections can load data from some sort of offline store and transactions (w/ their mutations) are persisted until they can be written to the network.

If you use QueryCollection — you can already do persistence of data w/ https://tanstack.com/query/latest/docs/framework/react/plugins/persistQueryClient — it probably makes sense to rely on the loaders/sync tools to do the persistence so I'm not sure we need to do anything beyond document this.

For working offline & syncing — this is already doable as well w/ manually controlled transactions — https://github.com/TanStack/db/blob/main/docs/index.md#useoptimisticmutation — I can imagine a useOfflineMutation that creates the transactions for mutations, checks if offline, if so — stores it in indexeddb, detects when you're online, has hooks for letting the dev precisely control any checks/controls needed to ensure safe writing to the server of the offline work. E.g. there's a series of dependent transactions and an early one has a conflict that the user has to resolve. How does the dev detect this, populate a UI for the user to make a decision on what to do, etc. etc.

I personally don't have enough experience w/ offline work to design that sort of framework. So I'd love to hear more about people's experiences and pointers to frameworks that work really well around handling offline mutations — particularly the syncing part when back online..

[thruflo]

It’ll be nice to have strategies for persistence that are independent of the sync/load config aka collection type.

A bit like the VFS layer for PGlite — many strategies and ignorant of the sync plugin.

Perhaps any collection could be passed to a persister which ties into the collection reactivity?

[KyleAMathews]

It’ll be nice to have strategies for persistence that are independent of the sync/load config aka collection type.

A bit like the VFS layer for PGlite — many strategies and ignorant of the sync plugin.

Perhaps any collection could be passed to a persister which ties into the collection reactivity?

yeah perhaps — the problem though is that different ways of loading/syncing data have their own way of re-syncing after coming back online. So e.g. with Electric, we can do an incremental sync of updates when we come back online — so Electric would want to directly control the persistence so it knows there's data on the device already & does an efficient catch-up call.

So I don't think you can decouple persistence from collection type. It seems like the right split of concerns that TanStack DB doesn't know how/where the data comes from, just that it's there.

[thruflo]

Yup, I can see that.

Unless perhaps the state that the sync/loader needs is just part of what’s persisted locally. So separate local persistence from external data loading. They would need to talk but perhaps that’s some standard lifecycle events for the collection (“synced” -> so persist, “hydrated” -> so resume sync).

[samwillis]

In our planning for persistence it's important to note there are three reasons for doing storage:

1. Persistence between sessions for faster startup
2. offline support so that an app can work offline
3. datasets that are larger than you may want to load fully into memory.

The first two have a lot of cross over and are quite "simple", but the third needs a little more consideration as it needs indexing in some way that would be hooked into by the query system.

In my mind persistence would be pluggable between collections, and have an interface for storing metadata that the sync provider needs to resume. I think it would be a mistake to jump to tightly integrate the collections with persistence.

Finally, with both the Browser and potentially React Native as targets, we would want alternative storage for the two - there is a clear matrix of sync x persistence backend.

[KyleAMathews]

Deep Research surveying how various libraries handle offline. https://chatgpt.com/share/682b2043-0fc0-800c-9c07-f0479ec4b2c9

I'd love to hear people's experiences with any of these tools!

[andr-ec]

Hey I can give a little feedback about how some libraries handle offline. I've built offline apps in web and in native ios and android.

GraphQL:
In web the cache is fine, it wasn't built with offline in mind, at least that's what it feels like.
In iOS and android the cache can be in memory or in sqlite or a custom implementation.
The big problem with both is that graphQL takes a lot of set up on the client side. There's some cool tools that help with codegen, but it become bulky very quickly. So this is treaded more like a cache than a sync.

What I've seen most often on ios and android is that the offline is rolled out manually. meaning sqlite or some other embedded db is used, and most of the time the schema is unrelated to the server db schema. The bad part is that it's a lot of work to maintain separate schemas and migrations, etc. But on the plus side, it's very flexible, Especially when using something like the repository pattern. This can be used as a cache or a sync.

1. Persistence between sessions for faster startup
2. offline support so that an app can work offline
3. datasets that are larger than you may want to load fully into memory.

I would add that reason 1 and 2 are going to be the most common.

but for 3 I would mention that a more common use case there is for client side search.

For most of my use cases just having persistence between sessions for speed and offline support covers the majority of the needs I've had.

The use case I'm evaluating currently is using something like tanstack db with electric in a webview to support offline mode in ios and android apps. From the previous messages it looks like this is already possible but I still need to test it out.

[xmsz-stu]

+1. The two features of offline data synchronization and multi - person collaboration synchronization are really practical. We have developed a PWA application for travel. Users often encounter offline situations (such as on airplanes or on roads without signal).

[Muromi-Rikka]

I'm excited about the recent release of @tanstack/db and its potential to simplify database caching and querying in web applications. I'd like to request support for integrating @tanstack/db with @electric-sql/pglite, a lightweight Postgres-compatible database designed for web environments.

[KyleAMathews]

We're not sure it makes much sense to integrate the two as there's a lot of overlap in functionality. More likely is TanStack DB will be persist directly to indexeddb like pglite does.

[KyleAMathews]

A related exploration livestorejs/livestore#366

[xmsz-stu]

+1. The two features of offline data synchronization and multi - person collaboration synchronization are really practical. We have developed a PWA application for travel. Users often encounter offline situations (such as on airplanes or on roads without signal).

btw，
I've tried many existing solutions (seen from here https://localfirstweb.dev/). Currently, I'm using "instant" because more or less, there are problems with the others.
However, instantDB also has drawbacks. For example, it can't be self - deployed.

[ignatz]

Exciting! Would love to join the conversation. What I think I understood:

1. There should be storage interface with the browser probably defaulting to index DB 👍
2. There also seems to be consensus that there's not a single perfect reconciliation strategy, since it will depend on the specific use-case as well as backend capabilities.

Looking at the ChatGPT output above, I was genuinely surprised how many solutions use last-write-wins (LWW) to reconcile. On the bright side, this means we can start very simple. The next level would probably be a compare-and-swap (CAS), and CRDTs we can leave to the academics 🙃

I'm probably over-simplifying, did I miss anything? What's the move?

[KyleAMathews]

My thinking atm is the only thing the core library should do is provide an API for persisting synced data. Everything else (optimistic mutations) is ephemeral unless the user decides to persist them or do anything else. Afaik the Transaction class is already flexible enough to enable persistence.

For persisting data, @samwillis is busily adding index support which is a necessary first step as any persistance would be built on that as the index would be loaded first and then data loaded as needed.