Area Dispatcher missing Unix socket after unplanned restart (No such file or directory /db/db//osm3s_areas Unix_Socket::7)

[bill-sabo-ia]

Hello, I am encountering an issue when using area queries (also shows up when the areas update cycle executes). For both scenarios I am seeing the following exception: 'runtime error: open64: 2 No such file or directory /db/db//osm3s_areas Unix_Socket::7'

This started after an unplanned restart of our dev environment (long story, bad k8s health probe configuration). What really struck me as strange is that I found the PID for the areas dispatcher and killed it, then all of the sudden the osm3s_areas socket showed up in /db/db. So, I deleted that file, then attempted to restart the areas dispatcher and got the following error: 'File_Error File exists 17 /osm3s_areas Dispatcher_Server::1'. This is despite the fact that I had deleted the one that showed up, very odd.

This server had been running for close to a month straight with no issues before this, including the init process for the North American portion of the OSM dataset, so don't know what to make of it. I would imagine that it's mostly my lack of in-depth knowledge of the Overpass API architecture, which brings me here.

Below are my outputs of ls -lsa for /db and /db/db:

/db:

4 drwxr-xr-x 5 root root 4096 Sep 20 08:46 .
4 drwxr-xr-x 1 root root 4096 Oct 27 15:12 ..
7632 -rw-r--r-- 1 overpass overpass 7807320 Oct 28 01:24 changes.log
4 -rw-r--r-- 1 overpass overpass 114 Oct 28 01:24 cookie.jar
12 drwxr-xr-x 3 overpass overpass 12288 Oct 27 23:24 db
4 drwxr-xr-x 2 overpass overpass 4096 Oct 28 01:24 diffs
0 -rw-r--r-- 1 overpass overpass 0 Sep 20 08:46 init_done
16 drwx------ 2 overpass overpass 16384 Sep 19 14:49 lost+found
4 -rw-r--r-- 1 overpass overpass 4 Oct 27 23:13 replicate_id
4 -rw-r--r-- 1 overpass overpass 4 Oct 28 01:24 replicate_id.backup

/db/db (Note the lack of osm3s_areas socket):

12 drwxr-xr-x 3 overpass overpass 12288 Oct 27 23:24 .
4 drwxr-xr-x 5 root root 4096 Sep 20 08:46 ..
3528516 -rw-r--r-- 1 overpass overpass 3613196288 Oct 27 22:04 area_blocks.bin
36 -rw-r--r-- 1 overpass overpass 33880 Oct 27 22:04 area_blocks.bin.idx
294404 -rw-r--r-- 1 overpass overpass 301465600 Oct 27 22:04 area_tags_global.bin
4 -rw-r--r-- 1 overpass overpass 3790 Oct 27 22:04 area_tags_global.bin.idx
62756 -rw-r--r-- 1 overpass overpass 64258048 Oct 27 22:04 area_tags_local.bin
12 -rw-r--r-- 1 overpass overpass 10883 Oct 27 22:04 area_tags_local.bin.idx
4 -rw-r--r-- 1 overpass overpass 21 Oct 27 21:45 area_version
61700 -rw-r--r-- 1 overpass overpass 63176704 Oct 27 22:04 areas.bin
4 -rw-r--r-- 1 overpass overpass 200 Oct 27 22:04 areas.bin.idx
251568 -rw-rw-rw- 1 overpass overpass 257590266 Oct 28 01:54 database.log
192 -rw-r--r-- 1 overpass overpass 196608 Oct 27 23:15 node_frequent_tags.bin
4 -rw-r--r-- 1 overpass overpass 28 Oct 27 23:24 node_frequent_tags.bin.idx
448 -rw-r--r-- 1 overpass overpass 458752 Oct 27 23:14 node_keys.bin
4 -rw-r--r-- 1 overpass overpass 24 Oct 27 23:24 node_keys.bin.idx
1461092 -rw-r--r-- 1 overpass overpass 1496154112 Oct 27 23:15 node_tags_global.bin
984 -rw-r--r-- 1 overpass overpass 1006000 Oct 27 23:24 node_tags_global.bin.idx
957236 -rw-r--r-- 1 overpass overpass 980205568 Oct 27 23:14 node_tags_local.bin
940 -rw-r--r-- 1 overpass overpass 962529 Oct 27 23:24 node_tags_local.bin.idx
16865140 -rw-r--r-- 1 overpass overpass 17269899264 Oct 27 23:14 nodes.bin
4772 -rw-r--r-- 1 overpass overpass 4884056 Oct 27 23:24 nodes.bin.idx
8074340 -rw-r--r-- 1 overpass overpass 8268120064 Oct 27 23:14 nodes.map
1584 -rw-r--r-- 1 overpass overpass 1619136 Oct 27 23:24 nodes.map.idx
11080868 -rw-r--r-- 1 overpass overpass 11346804736 Oct 27 23:14 nodes_meta.bin
9868 -rw-r--r-- 1 overpass overpass 10103000 Oct 27 23:24 nodes_meta.bin.idx
0 srw-rw-rw- 1 overpass overpass 0 Oct 27 15:12 osm3s_osm_base
4 -rw-r--r-- 1 overpass overpass 21 Oct 27 23:13 osm_base_version
192 -rw-r--r-- 1 overpass overpass 196608 Sep 22 02:22 relation_frequent_tags.bin
4 -rw-r--r-- 1 overpass overpass 33 Oct 27 23:24 relation_frequent_tags.bin.idx
128 -rw-r--r-- 1 overpass overpass 131072 Oct 27 23:23 relation_keys.bin
4 -rw-r--r-- 1 overpass overpass 24 Oct 27 23:24 relation_keys.bin.idx
128 -rw-r--r-- 1 overpass overpass 131072 Oct 22 22:44 relation_roles.bin
4 -rw-r--r-- 1 overpass overpass 24 Oct 27 23:24 relation_roles.bin.idx
96192 -rw-r--r-- 1 overpass overpass 98500608 Oct 27 23:24 relation_tags_global.bin
48 -rw-r--r-- 1 overpass overpass 49071 Oct 27 23:24 relation_tags_global.bin.idx
90980 -rw-r--r-- 1 overpass overpass 93159424 Oct 27 23:24 relation_tags_local.bin
52 -rw-r--r-- 1 overpass overpass 53123 Oct 27 23:24 relation_tags_local.bin.idx
236864 -rw-r--r-- 1 overpass overpass 242548736 Oct 27 23:24 relations.bin
16 -rw-r--r-- 1 overpass overpass 15544 Oct 27 23:24 relations.bin.idx
22848 -rw-r--r-- 1 overpass overpass 23396352 Oct 27 23:24 relations.map
4 -rw-r--r-- 1 overpass overpass 2424 Oct 27 23:24 relations.map.idx
39972 -rw-r--r-- 1 overpass overpass 40927232 Oct 27 23:24 relations_meta.bin
12 -rw-r--r-- 1 overpass overpass 9768 Oct 27 23:24 relations_meta.bin.idx
4 drwxr-xr-x 2 overpass overpass 4096 Sep 20 07:04 rules
332 -rw-r--r-- 1 overpass overpass 334340 Oct 28 01:54 rules_loop.log
71208 -rw-rw-rw- 1 overpass overpass 72871489 Oct 28 01:54 transactions.log
256 -rw-r--r-- 1 overpass overpass 262144 Oct 27 23:24 user_data.bin
4 -rw-r--r-- 1 overpass overpass 24 Oct 27 23:24 user_data.bin.idx
5412 -rw-r--r-- 1 overpass overpass 5537792 Oct 27 23:24 user_indices.bin
4 -rw-r--r-- 1 overpass overpass 312 Oct 27 23:24 user_indices.bin.idx
192 -rw-r--r-- 1 overpass overpass 196608 Oct 27 23:22 way_frequent_tags.bin
4 -rw-r--r-- 1 overpass overpass 30 Oct 27 23:24 way_frequent_tags.bin.idx
708 -rw-r--r-- 1 overpass overpass 720896 Oct 27 23:19 way_keys.bin
4 -rw-r--r-- 1 overpass overpass 24 Oct 27 23:24 way_keys.bin.idx
5114612 -rw-r--r-- 1 overpass overpass 5237358592 Oct 27 23:22 way_tags_global.bin
2372 -rw-r--r-- 1 overpass overpass 2427420 Oct 27 23:24 way_tags_global.bin.idx
2827204 -rw-r--r-- 1 overpass overpass 2895052800 Oct 27 23:19 way_tags_local.bin
2024 -rw-r--r-- 1 overpass overpass 2068914 Oct 27 23:24 way_tags_local.bin.idx
11516980 -rw-r--r-- 1 overpass overpass 11793383424 Oct 27 23:19 ways.bin
4476 -rw-r--r-- 1 overpass overpass 4580216 Oct 27 23:24 ways.bin.idx
1415908 -rw-r--r-- 1 overpass overpass 1449885696 Oct 27 23:19 ways.map
176 -rw-r--r-- 1 overpass overpass 176480 Oct 27 23:24 ways.map.idx
1627380 -rw-r--r-- 1 overpass overpass 1666433024 Oct 27 23:19 ways_meta.bin
760 -rw-r--r-- 1 overpass overpass 776328 Oct 27 23:24 ways_meta.bin.idx

Please let me know what else you need to help me in debugging this issue. Any help I can get in familiarizing myself with the remediation of potential operational issues like this would be invaluable. Thank you and have a great evening!

[arnesetzer]

I had this problem as well when the the docker instance got killed without proper notice. iirc correctly I used the command from the overpass-api bin/dispatcher --areas --db-dir="db/" inside the docker container to restart the area service. Also have a look at #54 or #128

[bill-sabo-ia]

I had this problem as well when the the docker instance got killed without proper notice. iirc correctly I used the command from the overpass-api bin/dispatcher --areas --db-dir="db/" inside the docker container to restart the area service. Also have a look at #54 or #128

Thanks for that additional info. I was able to get everything back up and running last night, and along the way figured out a couple of important details.

I also tried the 'dispatcher --areas' command like you posted, but mine would fail because I already had an areas lockfile in /dev/shm. This is confusing because the lockfile has the same name as the socket in /db/db (osm3s_areas), so the error message about that file existing made me think it was talking about the socket file not the lock file. At the time I did not realize that so it was all very confusing. The 'terminate' command for dispatcher was also not happy about it, so I wasn't able to use that. I had to find the PID of the 'areas' process and kill it manually, then supervisor respawned it and all was good.

So, I think the sequence of events for my case went as follows:

1. Unplanned restart (k8s pod deletion / recreation) due to false positive from Kubernetes health probe.
2. On restart the old 'areas' file in /db/db was not cleaned up.
3. Dispatcher tried to restart the 'areas' listener, and failed after cleaning up the old 'areas' file. This is where things are a bit fuzzy for me; not sure why the process failed to restart but it cleaned up the old 'areas' file. One thought I have is that my /db/ folder is on a persistent mount, so it was out of sync with my lockfiles in /dev/shm on the container's main volume after the container got refreshed. This is total speculation, need to research Overpass documentation more for better understanding.
4. I killed the old 'areas' process, supervisorctl automatically restarted it (didn't know this was built-in behavior).
5. New 'osm3s_areas' file created in /db/db, and lockfile overwritten in /dev/shm.

The first time I tried step 4, to kill the process, I could not figure out why I got a new 'osm3s_areas' file in /db/db, but now I know that it's because supervisorctl is configured to auto respawn processes that get killed. So, at this point my service was probably up and running again and I didn't know it. Still a little fuzzy on what caused the original areas dispatcher to start properly the first time (steps 2-3), but so far the sequence above is my best guess.

This can be closed, but if anyone else runs across it and has more insights please feel free to share anything that you think is important. At the least, I hope this can help someone else who runs into anything similar.