Ability to create collections with parameterized query functions

[zerdalu]

After reading Tanstack DB documentation, I was not able to find an example that fetches data from server with parameters. All examples show how to fetch all the data. I need to use project ID to fetch that project's data, not everything else.

Fow example, this creates a collection. queryFn: async () has no parameter.

const todoCollection = createCollection(
  queryCollectionOptions({
    queryKey: ['todos'],
    queryFn: async () => {
      const response = await fetch('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/api/todos')
      return response.json()
    },
    getKey: (item) => item.id,
    // Handle all CRUD operations
    onInsert: async ({ transaction }) => {
      const { modified: newTodo } = transaction.mutations[0]
      await fetch('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/api/todos', {
        method: 'POST',
        body: JSON.stringify(newTodo),
      })
    },
    onUpdate: async ({ transaction }) => {
      const { original, modified } = transaction.mutations[0]
      await fetch(`/api/todos/${original.id}`, {
        method: 'PUT', 
        body: JSON.stringify(modified),
      })
    },
    onDelete: async ({ transaction }) => {
      const { original } = transaction.mutations[0]
      await fetch(`/api/todos/${original.id}`, { method: 'DELETE' })
    },
  })
)

Since, todoCollection is being used in a component like below, I can not put it in a useEffect to either.

function TodoList() {
  // Basic filtering and sorting
  const { data: incompleteTodos } = useLiveQuery((q) =>
    q.from({ todo: todoCollection })
     .where(({ todo }) => eq(todo.completed, false))
     .orderBy(({ todo }) => todo.createdAt, 'desc')
  )

  // Transform the data
  const { data: todoSummary } = useLiveQuery((q) =>
    q.from({ todo: todoCollection })
     .select(({ todo }) => ({
       id: todo.id,
       summary: `${todo.text} (${todo.completed ? 'done' : 'pending'})`,
       priority: todo.priority || 'normal'
     }))
  )

  return <div>{/* Render todos */}</div>
}

What I want is something like below:

const todoCollection = createCollection(
  queryCollectionOptions({
    queryKey: ['todos'],
    queryFn: async (id: string) => {
      const response = await fetch('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/api/todos/' + id);
      return response.json();
    },
    getKey: item => item.id,
  })
);

Is there a way to do this?

[giladv]

exactly what i'm not getting about this library. are we supposed to load everything?
what happens if theres a /ORG_ID/todos ?

[KyleAMathews]

Collections are just Maps that sync basically — you can create as many of them as you want in whatever fashion you want.

I've been meaning to document a collection factory pattern which is what it sounds like you two want. I'll do this soon. This is what we see a lot of people doing something like this:

const cache = new Map<string, ReturnType<typeof createCollection>>();

const getTodoCollection = (id: string) => {
  if (!cache.has(id)) {
    const collection = createCollection(
      queryCollectionOptions({
        queryKey: ['todos', id],
        queryFn: async () => {
          const response = await fetch('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/api/todos/' + id);
          return response.json();
        },
        getKey: item => item.id,
      })
    );
    
    collection.on('status:change', ({ status }) => {
      if (status === 'cleaned-up') {
        cache.delete(id);
      }
    });
    
    cache.set(id, collection);
  }
  return cache.get(id)!;
};

We're also working on "query-driven sync" which will let a collection load data on demand based on what live queries you write — #612 — we'll have more on this soon.

[litzebauer]

@KyleAMathews I am following your collection factory pattern but the collection never fires 'status:change' with the status 'cleaned-up'. It looks like on clean up the event listeners are cleared before the status is changed to 'cleaned-up'. https://github.com/TanStack/db/blob/main/packages/db/src/collection/lifecycle.ts#L254-L269

[KyleAMathews]

@litzebauer oh heh — that's less than useful. Here's a PR w/ a fix #714

[litzebauer]

Thanks a lot! Looks good! Wasn't sure if I should submit a PR or call issue out first.

[JinNgVN]

hi, i tried the factory pattern but then i realized how can you get that typed?

[KyleAMathews]

yes sorry — the first example didn't cover that. Please try this:

// Step 1: Create a factory function
const createUserTodoCollection = (userId: string) => {
  const options = queryCollectionOptions({
    // ... your configuration here
  })
  
  return createCollection(options)
}

// Step 2: Extract the type using ReturnType
type UserTodoCollection = ReturnType<typeof createUserTodoCollection>

// Step 3: Use the type in your cache
const cache = new Map<string, UserTodoCollection>()

// Step 4: Create a getter function
export const getUserTodoCollection = (userId: string): UserTodoCollection => {
  if (!cache.has(userId)) {
    const collection = createUserTodoCollection(userId)
    
    // Optional: Auto-cleanup when collection is disposed
    collection.on("status:change", ({ status }) => {
      if (status === "cleaned-up") {
        cache.delete(userId)
      }
    })
    
    cache.set(userId, collection)
  }
  
  return cache.get(userId)!
}

[radoslavkarlik]

@KyleAMathews is it possible to achieve such behavior with react-db as well? The cleanup is not being called when a new collection with different params is created and used.

const collection = createCollection(accountId);

const { data } = useLiveQuery((q) =>
    q.from({ collection }).findOne()
);

I have also tried creating one in useEffect with manually calling collection.cleanup in effect cleanup which yields an error as the collection was still considered to be used.

[KyleAMathews]

You can set gcTime as an option to control when cleanup happens. By default is five mins after the collection is no longer used anywhere.

[KyleAMathews]

Completed by the typed business-scoped collection factory and memoization guidance in #1671. Runtime and business parameters belong in the collection factory, query key, and query function; relational predicates, ordering, limits, and offsets remain on-demand LoadSubsetOptions. See the Query Collection guide and RFC #1643.