Fix handling of storage that has multiple shared mounts - #271
Fix handling of storage that has multiple shared mounts#271christoph-bessei wants to merge 2 commits into
Conversation
bd9f527 to
546a21e
Compare
…le shared mounts Signed-off-by: Christoph Bessei <28066477+christoph-bessei@users.noreply.github.com>
…le shared mounts Signed-off-by: Christoph Bessei <28066477+christoph-bessei@users.noreply.github.com>
546a21e to
0c91783
Compare
|
Confirming this on a large instance, in case another data point helps it along. We hit exactly the symptom from #254 on Nextcloud 34.0.3 with context_chat 5.4.0 and context_chat_backend 5.4.1: roughly 1.5 M eligible files, heavy use of shares and group folders, and a steady stream of on What made it costly for us is what happens next to those entries. if ($queueMapper->lock($document->getId())) {
try {
$files[$document->getId()] = $this->getFileSource(...);
} catch (\Exception $e) {
$this->logger->warning($e->getMessage(), ['exception' => $e]);
$queueMapper->delete($document);
}
}so the queue drained quickly while almost nothing reached the vector database — from the outside it looked like indexing had stopped, when in fact files were being consumed and discarded. Because the warning goes to the Nextcloud log rather than the backend container, it is easy to spend a long time looking in the wrong place. With this patch applied, indexing ran to 32,879 documents and 490,387 chunks before we paused it for unrelated reasons, and the remaining per-file failures are the ordinary ones — One note on Thanks for writing it — happy to test further revisions on an instance of this size if that is useful. |
Fixes #254
Description
As described in #254 there are some edge cases where
$mounts[0]->getUser()->getUID()returns a user ID that has no access to a given file. This PR fixes this by iterating through the mounts until it finds a user with access to the given file.There might be cleaner solutions, but I wanted to reduce the changes as much as possible for an easier review. Assuming
$mounts[0]is working in most cases, the performance impact should be minimal.Notes
🤖 AI (if applicable)