[Detail Bug] Qdrant discovery: validation thread interrupt permanently marks GitHub collections discovery as FAILED

[detail-app]

Detail Bug Report

https://app.detail.dev/org_befd6425-a158-4e24-9d4d-1e5c08769515/bugs/bug_aae0664c-b3f4-4d5a-b372-1f7e6e34712f

Introduced in #114 by @WilliamAGH on Jul 19, 2026

Summary

• Context: QdrantGitHubCollectionDiscovery discovers GitHub repository collections at startup and via a @Scheduled retry, exposing them through the retrieval fan-out cache (getDiscoveredCollections()) and the actuator readiness gate (discoveryHealth()). Its state machine (PENDING/READY/FAILED) is the single source of truth for both.
• Bug: An InterruptedException raised during validation (inside validateGitHubCollection) is rethrown as a plain IllegalStateException, which falls through to the schema-failure arm of attemptDiscovery and permanently transitions the state machine to FAILED. The same InterruptedException raised during collection listing is propagated as a checked exception and keeps the state PENDING (recoverable by the @Scheduled retry). The two halves of loadValidatedGitHubCollections thus handle one logical event in two non-equivalent ways.
• Actual vs. expected: An interruption during validation should keep discovery PENDING exactly as it does during listing; instead it becomes FAILED permanently, and attemptDiscovery's non-PENDING guard at line 123 short-circuits every subsequent @Scheduled retry.
• Impact: (1) The retrieval fan-out list (getDiscoveredCollections(), used by QdrantCollectionScopeResolver) and readiness (discoveryHealth()) become stuck at empty/DOWN for the lifetime of the bean with no @Scheduled recovery. (2) A spurious ERROR log is emitted under the schema validation failed banner for what is a transient interruption. (3) This is a regression: the original implementation (499894a0) swallowed the interrupt and returned false without failing; the state-machine refactor (d078c87f) introduced the IllegalStateException conversion. (4) The inventory endpoint refreshDiscoveredCollections() is independent of discoveryState and continues to re-validate collections on each request, so the blast radius is the retrieval fan-out + readiness, not the inventory endpoint. (5) The only realistic production trigger today is a context-close shutdownNow() interrupting an in-flight @Scheduled retry mid-validation (see Trigger analysis); at shutdown the stuck-FAILED consequence is unobservable because the JVM is exiting, leaving the false-alarm ERROR log as the observable artifact.

Code with Bug

src/main/java/com/williamcallahan/javachat/config/QdrantGitHubCollectionDiscovery.java

} catch (InterruptedException _) {
    Thread.currentThread().interrupt();
    throw new IllegalStateException("GitHub collection validation interrupted for '" + collectionName + "'"); // <-- BUG 🔴 transient interrupt converted to RuntimeException
}

private synchronized void attemptDiscovery(boolean startupAttempt) {
    if (discoveryState.get() != GitHubDiscoveryState.PENDING) {
        return; // <-- BUG 🔴 once validation interrupt forces FAILED, scheduled retry can never recover
    }
    try {
        List<String> gitHubCandidates = loadValidatedGitHubCollections();
        discoveredCollections.set(gitHubCandidates);
        discoveryState.set(GitHubDiscoveryState.READY);
        ...
    } catch (InterruptedException _) {
        Thread.currentThread().interrupt();
        log.debug("[QDRANT] GitHub collection discovery interrupted while pending");
    } catch (RuntimeException schemaException) {
        discoveredCollections.set(List.of());
        discoveryState.set(GitHubDiscoveryState.FAILED);
        log.error("[QDRANT] GitHub collection schema validation failed (exceptionType={}): {}",
                schemaException.getClass().getSimpleName(), schemaException.getMessage(), schemaException);
    }
}

Explanation

loadValidatedGitHubCollections() performs (1) async listing of collections and (2) per-collection validation via validateGitHubCollection().

• If the thread is interrupted during listing, InterruptedException propagates up (method declares throws InterruptedException) and is handled by attemptDiscovery’s catch (InterruptedException), leaving the discovery state PENDING.
• If the thread is interrupted during validation, validateGitHubCollection catches InterruptedException and converts it to IllegalStateException (unchecked). attemptDiscovery then treats it as a schema/validation RuntimeException, sets state to FAILED, and thereafter the PENDING-only guard prevents all future scheduled retries.

This misclassifies a transient interruption as a permanent schema defect, producing inconsistent state-machine behavior for the same underlying event (InterruptedException).

Failing Test

src/test/java/com/williamcallahan/javachat/config/QdrantGitHubCollectionDiscoveryProbeTest.java

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import com.williamcallahan.javachat.AppProperties;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Collections.CollectionInfo;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.junit.jupiter.api.Test;
import org.springframework.boot.actuate.health.Status;

class QdrantGitHubCollectionDiscoveryProbeTest {

    private static final int EMBEDDING_DIMENSIONS = 1536;
    private static final String ACTIVE_COLLECTION = "github-repo-foo";

    @Test
    void interruptionDuringListingStaysPending()
            throws InterruptedException, ExecutionException, TimeoutException {
        QdrantClient qdrantClient = mock(QdrantClient.class);
        EmbeddingClient embeddingClient = mock(EmbeddingClient.class);
        ListenableFuture<List<String>> collectionNamesRequest = mock();
        when(qdrantClient.listCollectionsAsync(any())).thenReturn(collectionNamesRequest);
        when(embeddingClient.dimensions()).thenReturn(EMBEDDING_DIMENSIONS);
        when(collectionNamesRequest.get(anyLong(), eq(TimeUnit.SECONDS)))
                .thenThrow(new InterruptedException("listing interrupted"));
        when(collectionNamesRequest.isDone()).thenReturn(false);

        QdrantGitHubCollectionDiscovery discovery =
                new QdrantGitHubCollectionDiscovery(qdrantClient, embeddingClient, new AppProperties());
        discovery.discoverGitHubCollections();

        verify(collectionNamesRequest).cancel(true);
        assertEquals(Status.DOWN, discovery.discoveryHealth().getStatus());
        assertEquals("pending", discovery.discoveryHealth().getDetails().get("githubCollectionDiscovery"));
    }

    @Test
    void interruptionDuringValidationTransitionsToFailedState()
            throws InterruptedException, ExecutionException, TimeoutException {
        QdrantClient qdrantClient = mock(QdrantClient.class);
        EmbeddingClient embeddingClient = mock(EmbeddingClient.class);
        when(qdrantClient.listCollectionsAsync(any()))
                .thenReturn(Futures.immediateFuture(List.of(ACTIVE_COLLECTION)));
        ListenableFuture<CollectionInfo> collectionInfoRequest = mock();
        when(qdrantClient.getCollectionInfoAsync(ACTIVE_COLLECTION)).thenReturn(collectionInfoRequest);
        when(embeddingClient.dimensions()).thenReturn(EMBEDDING_DIMENSIONS);
        when(collectionInfoRequest.get(anyLong(), eq(TimeUnit.SECONDS)))
                .thenThrow(new InterruptedException("validation interrupted"));
        when(collectionInfoRequest.isDone()).thenReturn(false);

        QdrantGitHubCollectionDiscovery discovery =
                new QdrantGitHubCollectionDiscovery(qdrantClient, embeddingClient, new AppProperties());
        discovery.discoverGitHubCollections();

        verify(collectionInfoRequest).cancel(true);
        assertEquals(Status.DOWN, discovery.discoveryHealth().getStatus());
        assertEquals("failed", discovery.discoveryHealth().getDetails().get("githubCollectionDiscovery"));
    }
}

Test report:

<testsuite name="...QdrantGitHubCollectionDiscoveryProbeTest" tests="2" skipped="0" failures="0" errors="0" ...>
  <testcase name="interruptionDuringListingStaysPending()"                    time="1.199"/>
  <testcase name="interruptionDuringValidationTransitionsToFailedState()"     time="0.024"/>

Recommended Fix

Treat an interruption during validation as transient, mirroring the listing path. For example, propagate InterruptedException from validateGitHubCollection so attemptDiscovery’s existing catch (InterruptedException) keeps discovery PENDING.

History

This bug was introduced in commit d078c87ff "Isolate Qdrant collections by environment and generation". That commit added the PENDING/READY/FAILED state machine and a catch (RuntimeException schemaException) arm in attemptDiscovery that permanently parks discovery at FAILED; in the same change validateGitHubCollection's existing catch (InterruptedException) { ... return false; } (which the prior 499894a0 and ed792d2f versions used to swallow the interrupt as a transient skip) was rewritten to throw new IllegalStateException("GitHub collection validation interrupted for '...'"), so the transient interrupt began escaping into the new permanent-FAILED arm instead of returning false. The two later commits on this file (6e8e9a75, 327f776f) did not touch the validation-interrupt handling, so d078c87ff remains the most recent commit that clearly contributed to the bug's existence.