Hide the bundled OpenSSL's symbols from other libraries

[excid3]

openssl.so links OpenSSL statically but exports all of it without symbol versions (about 1,800 symbols on 2.7.8). Two things go wrong when a system library that uses OpenSSL 3 is loaded in the same process, such as libpq through the pg gem, which every Rails app on Postgres loads:

• Ruby's openssl loaded first: libpq's SSL_new, SSL_CTX_new and other OpenSSL 3 calls bind to Ruby's static copy (confirmed with LD_DEBUG=bindings).
• pg loaded first: openssl.so's own calls go to libssl.so.3, and Net::HTTPS segfaults in peer_cert.

On the OpenSSL 1.x series (1.8 through 3.1), this breaks pg together with HTTPS: segfaults, munmap_chunk(): invalid pointer, and failed TLS handshakes to Postgres even with sslmode=disable. The 3.2+ series only work by luck, because both copies are OpenSSL 3.

Changes

• Linker flag: Ruby is now linked with -Wl,--exclude-libs,libssl.a:libcrypto.a. It can't be ALL, because libruby is static and its rb_* symbols have to stay exported. rbconfig keeps the flag, so native gems that link the bundled OpenSSL (puma) also stop exporting it.
• New check: check_openssl_hidden! fails a Linux build if any .so exports SSL_new or CRYPTO_malloc. It flags the current 2.3.8-3 and 2.7.8-3 releases and passes the new builds.

package.rb is part of the build fingerprint, so each version gets a new revision the next time it's released.

Testing

I built arm64 packages locally with bin/package-linux and tested each in ubuntu:24.04 with the system's libpq (OpenSSL 3), a local Postgres, and HTTPS to rubygems.org. Every combination was tested: both load orders, with sslmode=require and sslmode=disable.

Ruby	OpenSSL	Current release	This branch
2.3.8	1.0.2u	segfault / munmap_chunk in all 4	all 4 pass
2.7.8	1.1.1w	3 of 4 fail (peer_cert segfault, TLS error)	all 4 pass, plus puma 6 MiniSSL
3.4.11	3.5.5	passes, but libpq binds to Ruby's copy	all 4 pass, plus puma 6 MiniSSL
1.8.7	1.0.2u	not tested with pg	builds, no symbols exported, 2**64 still correct

https://claude.ai/code/session_0146HthzF4yTdERCfixDX2Mh