[bug] Unable to upgrade to v3

[gaeljw]

Describe the bug

Hello folks,

I just tried to upgrade to v3 (from 2.8.4) following https://docs.sourcebot.dev/self-hosting/upgrade/v2-to-v3-guide but upon startup of the v3, I've got errors related to Postgres.

I haven't had time to investigate yet, this may be a permission issue on our end.

In doubt, I thought I'd share it :)

Logs during startup:

[Info] Waiting for the database to be ready...
postgres: could not access the server configuration file "/data/.sourcebot/db/postgresql.conf": No such file or directory
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response

To reproduce

N/A

Sourcebot deployment information

Sourcebot version (e.g. v3.0.1): v3.0.2

Additional information

No response

[msukkari]

The one time I've seen this happen is when postgres is unable to initialize on the initial startup. What is the setup you have with mounting the disk to the container?

[gaeljw]

What is the setup you have with mounting the disk to the container?

It's a regular volume mount through Kubernetes. Volume is NFS though. I've had issues in the past with NFS volumes and Postgres, so that could be it. But there's not much logs for me to investigate.

I'll give it locally on my laptop to troubleshoot

[gaeljw]

This definitely look's like a permission issue.

On my laptop, I'm able to make it work but I see that the postgres folder is owned by another user:

ll .sourcebot   
total 16K
drwx------. 19   100069   100069 4,0K  7 avril 13:06 db
drwxr-xr-x.  2 gael gael 4,0K  7 avril 13:06 index
drwxr-xr-x.  2 gael gael 4,0K  7 avril 13:07 redis
drwxr-xr-x.  3 gael gael 4,0K  7 avril 13:06 repos

I know NFS is a bit crappy for such things. I've had a look at another setup where we have a NFS volume mount for a Postgres container and we've had to add a couple of configurations to make it work (runAsUser and runAsGroup on the postgres container with same uid than the owner of NFS files).

Thus I guess it's something related to our infra rather than an issue with Sourcebot.

But I would expect Sourcebot to raise a more explicit error that it's an issue with permissions.

That being said, for now, I don't have a workaround on my end to make it work. I'll probably switch to another type of volume anyway.

[gaeljw]

It's the following line that raises the error https://github.com/sourcebot-dev/sourcebot/blob/bf8b28252d1aacdf9d54e0f5feb8f8994ecb9006/entrypoint.sh#L133C5-L133C52

Even after giving all possible rights to the folder. From inside the container:

/app # su postgres -c "postgres -D $DATABASE_DATA_DIR"
postgres: could not access the server configuration file "/data/.sourcebot/db/postgresql.conf": No such file or directory
/app # ls -al /data/.sourcebot/db/
total 8
drwxrwxrwx    2 postgres postgres      4096 Apr  7 11:15 .
drwxr-xr-x    4 nobody   nobody        4096 Apr  7 11:15 ..

[msukkari]

Thanks for investigating! Yeah seems like the internal postgres deployment doesn't like certain volumes. I don't have too much experience with postgres, so not exactly sure why without further investiagation

We should definitely add a better error message when this happens. In the meantime, could you try a different mount? Alternatively, you could spin up an external postgres server and connect sourcebot to it using DATABASE_URL

[belaw]

I've had this issue as well and have found a workaround.
The issue is that Sourcebot can't chown the DB directory:

[Info] Initializing database at /data/.sourcebot/db...
chown: /data/.sourcebot/db: Operation not permitted

This might be related to NFS root squashing? No idea.
This message will only be logged in the first container Kubernetes creates, which immediately crashes and gets replaced. If you don't have a log aggregation system, it is very easy to miss

Workaround

1. Connect to your NFS server and navigate to the .sourcebot directory of the container and chown the db directory:

chown 70:70 $path_to_db_directory # 70 = postgres

2. Open a shell inside the Sourcebot container and execute the following command:

su postgres -c "initdb -D $DATABASE_DATA_DIR"

3. Restart the deployment and it should work.

The line su postgres -c "initdb -D $DATABASE_DATA_DIR" is taken directly from the entrypoint.sh script, it just gets skipped if the db directory already exists.

[msukkari]

Thanks for finding a workaround @belaw ! We'll try to get this logic backed into the script

[gaeljw]

The workaround was not enough for me ; I ended up spinning a dedicated external Postgres DB.

[komapa]

We had to give our container the CHOWN capability:

          securityContext:
            runAsNonRoot: false
            readOnlyRootFilesystem: false
            capabilities:
              add: [ "SETGID", "SETUID", "CHOWN" ]

[chedabob]

I ended up creating a volume and then mounting both that and the config, but this was on a fresh install. I'm only experimenting with Sourcebot on my laptop so not sure if this is actually a sensible fix or an outright hack.

For reference I'm running it on Mac OS 15.6 in Rancher, using QEMU and 9p (I haven't changed back to Apple VZ and virtiofs since switching to using a Docker volume, so they may also work).

docker volume create sourcebot-vlm

docker run \
    -p 3000:3000 \
    --pull=always \
    --rm \
    -v sourcebot-vlm:/data \
    -v $(pwd)/config.json:/data/config.json \
    -e CONFIG_PATH=/data/config.json \
    --name sourcebot \
    ghcr.io/sourcebot-dev/sourcebot:latest

[msukkari]

going to reopen this @brendan-kellam but we should probably create separate issues to track underlying issues here