Wait for coordinator to finish recovery before connecting in gpstart

[reshke]

In CBDB3 after the upstream PG16 rebase on commit 7ff23c6
PMSIGNAL_RECOVERY_STARTED is now sent during crash recovery too.

With hot_standby=off (the default), this causes postmaster to write PM_STATUS_STANDBY to the pidfile as soon as recovery begins, and pg_ctl -w treats standby as success. gpstart then hits
FATAL "Hot standby mode is disabled".

Add a shared _waitForCoordinatorRecovery helper that polls dbconn.connect with a 5s interval up to 300s, retrying only on recovery-related FATAL messages (not accepting connections / not yet accepting connections /

reshke@yezzey-cbdb-bench:~/cloudberry$ gpstart -a
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting gpstart with args: -a
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Gathering information and validating the environment...
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Cloudberry Binary Version: 'postgres (Apache Cloudberry) 3.0.0-devel+dev.20675.g51967d33cb2 build dev'
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Cloudberry Catalog Version: '302606111'
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting Coordinator instance in admin mode
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility " start
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Obtaining Cloudberry Coordinator catalog information
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Obtaining Segment details from coordinator...
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Coordinator still recovering, waiting 5 seconds (waited 0/300): connection to server at "localhost" (::1), port 7000 failed: FATAL:  the database system is not accepting connections
DETAIL:  Hot standby mode is disabled.

20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Setting new coordinator era
20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Coordinator Started...
20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Shutting down coordinator
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Process results...
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Successful segment starts                                            = 6
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Failed segment starts                                                = 0
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Successfully started 6 of 6 segment instances
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting Coordinator instance yezzey-cbdb-bench directory /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=850d57def3e5611f_260818042511 $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=dispatch " start
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Command pg_ctl reports Coordinator yezzey-cbdb-bench instance active
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting standby coordinator
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Checking if standby coordinator is running on host: yezzey-cbdb-bench  in directory: /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby
20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=850d57def3e5611f_260818042511 $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby/log/startup.log -t 600 -o " -p 7001 -c gp_role=dispatch " start
20260818:04:25:13:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Database successfully started

[reshke]

To trigger issue you need to dirty your buffers and run immediate stop:

CREATE TABLE z(i int); INSERT INTO z SELECT generate_series(1, 1000000); 

and then gpstop -a -i && gpstart -a

[leborchuk]

LGTM, 300 seconds for recovery wait is normal. And also for large installations we should have option to increase it

[reshke]

Another option is to change pg_ctl wait semantic for CBDB. But I consider this as undesirable discrepancy with upstream.

[yjhjstz]

LGTM