[gmpassos]

Summary

bones_api_concurrency_memory_test.dart failed under the macOS default limit of open files (256) with Too many open files (errno = 24).

The cause is in DBObjectDirectoryAdapter: selectAll and selectByIDs read all their object files in parallel, one async readAsString per file. Concurrent selects of a large table therefore opened thousands of files at once. This is not a leak: every file is closed afterwards. It is an unbounded burst.

Investigation

• Running each adapter group alone under ulimit -n 256: only the DBObjectDirectoryAdapter group fails. SQL memory, SQLite, object memory and fake GCS all pass.
• lsof sampling during that group:
  • 3875 of the adapter's object files open at once, and 0 at the end;
  • 30 concurrent selectAlls of 1000 objects: 24000 open at once.

Fix

• DBObjectDirectoryAdapter: object reads go through a FileLimited semaphore (default 50), the same mechanism APIServerResponseCache already uses. The peak above drops from 3875 to 38.
• FileLimited.readAsString: like File.readAsString, under the semaphore. This adds one method to an exported class, matching its existing readAsBytes/writeAsBytes/stat/delete/exists.

Review fixes (third commit)

A /code-review pass found 7 issues, and all are fixed:

1. Read race. _readObject checked existsSync before waiting for the semaphore, so an object deleted while its read waited threw PathNotFoundException and failed the whole select.
   • The check is gone: a missing file now reads as null.
   • This also removes one blocking syscall per row (finding 6).
2. Delete race. _doDeleteImpl now reads synchronously instead of waiting behind the reads of concurrent selects. A concurrent delete of the same object now deletes nothing instead of throwing.
3. FileLimited semaphore swap. Every method released whatever semaphore _semaphore pointed to at that moment, so an operation still running when setSemaphoreLimit replaced it released the new semaphore, pushing it above its limit.
   • The acquire/release now lives in one _run helper that releases the semaphore it acquired.
4. Pool not configurable. DBObjectDirectoryAdapter.fileLimited exposes the shared semaphore so it can be tuned with setSemaphoreLimit, and FileLimited is now exported by bones_api_db_directory.dart.
5. One queued read per object. Selects now read in batches of the fileLimited limit, so each concurrent select queues at most one batch of reads instead of one read per object.
6. Fragile regression test. It now:
   • resolves the package root, so it works from any working directory;
   • runs the script directly with dart --packages instead of dart run, so no package resolution or build hooks run under the low limit;
   • is skipped when ulimit -n 256 can't be set.

Tests

• Regression test (bones_api_entity_db_directory_test):
  • It runs concurrent selectAlls in a child process limited to 256 open files (ulimit -n 256), through the helper script test/bones_api_entity_db_directory_open_files.dart.
  • It fails with errno = 24 without the fix and passes with it.
  • A subprocess is needed because the open-file count can't be measured reliably from inside the process: on macOS, /dev/fd lists at most 128 entries, against 24000 really open.
  • Verified against a master worktree: on master it fails with Too many open files; on this branch it completes OK: 10 x 1000 objects.
• Concurrent deletes and selects:
  • 5 selectAlls run while 150 objects are each deleted twice at the same time;
  • no exceptions;
  • each object is deleted exactly once.
• fileLimited: the default limit, and reads in batches of a tuned limit (selectAll and selectByIDs).
• FileLimited.readAsString unit test: encoding, concurrency above the limit, release after an error.
• FileLimited.setSemaphoreLimit while operations are running: all of them complete, and the new semaphore keeps working.
• bones_api_concurrency_memory_test.dart now passes with ulimit -n 256.

Run locally:

• dart format --set-exit-if-changed: clean
• dart analyze --fatal-infos --fatal-warnings: clean
• CI unit suite (dart test --platform vm -x docker -x concurrency -j 4): all passed
• bones_api_version_test: passed
• dart pub publish --dry-run: 0 warnings

Version

1.24.1, since v1.24.0 is tagged: pubspec.yaml, BonesAPI.VERSION and the CHANGELOG.

#186 (static files .aliases) is at 1.25.0. Whichever PR merges second will conflict on the version lines and the top of the CHANGELOG. If this one merges first, #186 stays 1.25.0, with the 1.24.1 section below it.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.59459% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.28%. Comparing base (66ca087) to head (0ab2da8).

Files with missing lines	Patch %	Lines
lib/src/bones_api_entity_db_object_directory.dart	91.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #187      +/-   ##
==========================================
- Coverage   95.29%   95.28%   -0.01%     
==========================================
  Files          68       68              
  Lines       23771    23774       +3     
==========================================
+ Hits        22653    22654       +1     
- Misses       1118     1120       +2     

Flag	Coverage Δ
unittests	95.28% <94.59%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.