Repository navigation
[Bug] DeepH loses H-only edges and cannot read current ABACUS CSR/get_s output #7871
Description
Describe the bug
The current ABACUS–DeepH interface has two independent classes of failure:
- Correctness: the atom-pair support of the ABACUS Hamiltonian is not always a subset of the overlap support. DeepH uses overlap support to construct the inference graph, so valid Hamiltonian edges can be absent from the graph and prediction output.
- Current-version I/O compatibility: current ABACUS
developoutput cannot be consumed by either public DeepH0.2.2or the current documented DeepH1.0.6path through DeepH-dock0.9.12.
Versions checked on 2026-08-28:
- ABACUS
develope461ffd4246e6be41189d35a242bb44c4c50eca9, runtime bannerv3.11.0-beta8. - Public DeepH-pack
141faaaba1d5e17248283ef939bcd2ead2d8e677, version0.2.2. - DeepH-pack documented wheel
deepx-1.0.6+light. - DeepH-dock
6fc20bb4ec00c8d4de1d192338ba116527772f7a, version0.9.12.
1. Reproduced H/S support mismatch
ABACUS creates overlap pairs only for direct orbital overlap:
distance(i,j) < r_phi(i) + r_phi(j)
However, the nonlocal Hamiltonian can couple two orbitals through a third atom's projector:
<phi_i|beta_I> D_I <beta_I|phi_j>
Therefore, S_ij(R) == 0 does not imply H_ij(R) == 0. This follows directly from the different pair construction in overlap.cpp and nonlocal.cpp.
In the four-Si reproducer below, the NAO cutoff is 6 Bohr and the nonlocal-projector cutoff is 1.94 Bohr. Atoms 1 and 3 are 13.52296 Bohr apart, so they have no direct overlap, while each is 7 Bohr from atom 2 and couples through its projector.
At an atom-block threshold of 1e-8, current ABACUS produces:
H atom-block keys: 16
S atom-block keys: 14
H-only [0, 0, 0, 1, 3]: max|H|=5.48772265e-05 Ry, max|S|=0
H-only [0, 0, 0, 3, 1]: max|H|=5.48772265e-05 Ry, max|S|=0
These overlap blocks are structurally absent, not merely filtered numerical noise, so lowering the sparse threshold does not solve the problem.
This is deterministically harmful in the current converter: DeepH-dock first derives atom_pairs from overlap nonzeros, then extracts Hamiltonian blocks only for those overlap-derived pairs (source). Thus H-only pairs are discarded during conversion. Current DeepH documentation also permits only S/Sap graph construction during inference and states that overlap sparsity is indispensable, while training defaults to H and also supports HS (inference, training).
Possible consequences are silently missing predicted Hamiltonian blocks, incorrect bands/eigenvalues, and downstream generalized-eigenproblem failures.
2. Current ABACUS output is not readable by the documented DeepH path
Current ABACUS SCF output is:
running_scf.log
hrs1_nao.csr
hrs2_nao.csr # nspin=2
sr_nao.csr
DeepH-dock 0.9.12 still hard-codes:
running_scf.log
data-HR-sparse_SPIN0.csr
data-HR-sparse_SPIN1.csr
data-SR-sparse_SPIN0.csr
It also parses the legacy CSR header, while ABACUS v3.9.0.25+ writes the rich CSR header with spin metadata, UnitCell data, and labeled CSR sections. The current DeepH-dock documentation explicitly targets ABACUS 3.10 LTS and shows only the legacy layout (documentation).
For overlap-only inference, an actual calculation get_s run on e461ffd produced:
OUT.<suffix>/running_get_s.log
OUT.<suffix>/sr_nao.csr
DeepH-dock and public DeepH 0.2.2 expect case-sensitive running_get_S.log and SR.csr. The latest DeepH prediction guide also still says that ABACUS dumps SR.csr (documentation).
Public DeepH 0.2.2 has additional incompatibilities: it searches case-sensitively for lattice constant (Bohr) while current ABACUS prints Lattice constant (Bohr), rejects nspin=2, and applies a separate 1e-8 atom-block filter (parser).
This conflicts with the current ABACUS documentation claim that “DeepH v1.0.0+” reads the v3.9.0.25+ CSR format (ABACUS documentation). I could not find a current released converter that implements that contract.
3. Other interface limitations that should be part of the supported-version contract
- DeepH-dock cannot distinguish two ABACUS species of the same element that use different basis/pseudopotential settings; it requires standard element-symbol labels.
- Public DeepH
0.2.2does not support collinearnspin=2. DeepH-dock added support for the legacy two-file layout, but currenthrs1_nao.csr/hrs2_nao.csrcannot reach that path. - Current ABACUS native-binary and NPZ
out_hsrmodes have no documented DeepH conversion path; only text CSR should be advertised as supported. - Public DeepH
0.2.2and the new JAXdeepx-1.0.6are separate implementations. The compatibility documentation should name exact ABACUS, DeepH-dock, and DeepH versions.
Expected behavior
- The graph used for training and inference must contain every Hamiltonian target/prediction edge:
H_target_keys <= graph_keys
- Current ABACUS SCF and
get_soutput should be readable by the converter advertised in the ABACUS documentation. - The two projects should publish one exact compatibility matrix covering ABACUS version/commit, CSR format, DeepH-dock version, DeepH version, and
nspin=1/2/4.
A robust fix would export or construct a dedicated graph-support/atom-pair artifact containing the union of direct-overlap and nonlocal-projector-mediated Hamiltonian pairs. This should be independent of numerical overlap values; changing the physical overlap matrix or only lowering a threshold is not sufficient.
To Reproduce
- Build ABACUS
developate461ffd4246e6be41189d35a242bb44c4c50eca9with LCAO, MPI, and OpenMP enabled. - Create a case directory and copy these files from the ABACUS repository:
cp ../../tests/PP_ORB/Si_dojo_nsoc.upf .
cp ../../tests/PP_ORB/Si_dojo_6au_sz.orb .- Use the following input files.
INPUT:
INPUT_PARAMETERS
suffix deeph_bridge4
calculation scf
nbands 10
symmetry 0
pseudo_dir ./
orbital_dir ./
gamma_only 0
ecutwfc 20
scf_thr 1e-8
scf_nmax 100
basis_type lcao
smearing_method gauss
smearing_sigma 0.002
mixing_type broyden
mixing_beta 0.7
mixing_gg0 0.0
out_hsr 1 8
ks_solver scalapack_gvx
STRU:
ATOMIC_SPECIES
Si 14 Si_dojo_nsoc.upf upf201
NUMERICAL_ORBITAL
Si_dojo_6au_sz.orb
LATTICE_CONSTANT
1.0
LATTICE_VECTORS
40.0 0.0 0.0
0.0 40.0 0.0
0.0 0.0 40.0
ATOMIC_POSITIONS
Direct
Si
0.0
4
0.325000000000 0.500000000000 0.500000000000 1 1 1
0.500000000000 0.500000000000 0.500000000000 1 1 1
0.651554445663 0.587500000000 0.500000000000 1 1 1
0.500000000000 0.500000000000 0.675000000000 1 1 1
KPT:
K_POINTS
0
Gamma
1 1 1 0 0 0
- Run with one OpenMP thread:
OMP_NUM_THREADS=1 /path/to/abacus- Compare 4-by-4 atom blocks in
hrs1_nao.csrandsr_nao.csrat1e-8. The two H-only pairs and values shown above are obtained. - Change the INPUT to
calculation get_sand run again. The actual output names arerunning_get_s.logandsr_nao.csr, which the current DeepH-dock ABACUS validator does not recognize.
Environment
- OS: Ubuntu 22.04
- Compiler: GCC 11.4.0 through Open MPI
mpicxx - MPI: Open MPI 4.1.2
- Linear algebra: OpenBLAS, LAPACK, ScaLAPACK
- ABACUS build: MPI ON, OpenMP ON, LCAO ON, ELPA ON
- Runtime: CPU, one MPI rank,
OMP_NUM_THREADS=1
The SCF converged in 84 iterations. The get_s reproduction completes without SCF in about 0.08 seconds.
Additional Context
The old warning in the public DeepH repository already states that the DeepH–ABACUS interface has an overlap sparsity-pattern bug (warning). The reproducer above identifies a concrete physical/source-code cause and shows that the warning is still applicable to the latest documented S/Sap inference workflow.
Related issue #3691 is not a duplicate. That issue concerned a spin/SOC dimension mismatch (50-by-50 H versus 25-by-25 S) and was resolved by setting the converted data as spinful. This report concerns missing atom-pair keys even when H and S have the same orbital dimension.
Suggested regression coverage:
- Assert
H_target_keys <= graph_keysfor the four-Si projector bridge case. - End-to-end SCF and
get_sconversion tests fornspin=1,nspin=2, andnspin=4. - Pin exact ABACUS, DeepH-dock, and DeepH versions in CI and documentation.
Activity
goodchong commentedon Sep 7, 2026
Following up with a source-level comparison of OpenMX and ABACUS, and a proposal covering both existing output formats and a possible OpenMX-compatible export.
The required invariant is shared graph/block indices, not identical numerical nonzeros in H and S. A physically zero overlap block must remain zero, while the corresponding nonzero Hamiltonian block must be retained.
Scope: this follow-up inspected the local ABACUS checkout
83e5099087154be986135db536f52922340edcf9, OpenMX 4.0 sources (banner dated 26 April 2026), and DeepH-dockc988eaafc1ec41fe8d1bef4f7a2b50dccd4af682. The ABACUS checkout is older thane461ffdused in the original reproducer. Its legacyout_mat_hs/out_mat_hs2and NPZ flags should not be confused with the newerout_hsk/out_hsrmodes described above. This is an implementation proposal, not a claim that a fix or new format has been tested.1. What OpenMX does
OpenMX
.scfoutis a custom binary format with variable-size dense atom-pair blocks and shared connectivity metadata, rather than scalar CSR.FNAN,natn, andncn, together with per-atom orbital counts and lattice translations, determine the block layout for both H and S.In
SCF2File.c, both writers iterate over the same neighbor list and write every element of each block. Thus, for a stored pair with 3 and 5 orbitals, an all-zero S block still occupies 15 doubles. There is no independent numerical-nonzero filter in these H/S write loops. A pair absent from the connectivity list has neither an H block nor an S block.There is also a separate Hamiltonian construction difference:
truncation.cselects pairs within the sum of PAO cutoff radii, andSet_Nonlocal.cmultiplies the nonlocal contribution bydampingF(rcutA + rcutB, distance).dampingF.csmoothly takes this contribution to zero over the final 1 Bohr before that cutoff. This is more than an output-layout choice. The OpenMX author's explanation of the neighbor criterion is consistent with the inspected code.An ABACUS exporter should reuse the shared-block representation without copying this truncation/damping into ABACUS or deleting its projector-mediated H-only pairs.
2. Handling the existing formats
mat_i_j_Rx_Ry_Rzkeys and fill missing arrays with zeros. Apply a consistent Hermitian reconstruction convention.The inspected NPZ MPI writer omits atom pairs with
i > j; readers requiring directed edges must reconstruct(j,i,-R)consistently for both matrices. NPZ here is an archive of dense atom-block arrays, not SciPy's scalar-CSRsave_npzlayout. See the inspected NPZ writer.For newer native-binary CSR output, the same independent-read/union rule applies: changing text encoding to binary does not itself fix structural support.
Explicit zeros in scalar CSR alone are insufficient as a graph contract: downstream
nonzero, thresholding, oreliminate_zerosoperations may discard them again. Consumers must read structural keys independently of S's numerical values. Also, unioning exported files cannot recover values already discarded by an output threshold.The immediate converter fix is to replace the S-only
matrix_infoconstruction in the ABACUS converter. In contrast, the OpenMX converter already builds its block indices from connectivity metadata.3. The SCF/training fix must also cover overlap-only inference
When H is available, its block-key union with S is sufficient to preserve the exported H targets. During
calculation get_sinference, however, H is unavailable, so this union cannot be computed from output matrix values.ABACUS should therefore expose structural graph support constructed from geometry, basis/projector cutoffs, and the enabled Hamiltonian terms. For the reproducer, this must include both direct-overlap pairs and pairs connected through a common nonlocal projector. It should be available without SCF and shared by SCF and
get_sworkflows. Additional Hamiltonian terms require an explicitly supported graph-construction policy rather than assuming that these two pair classes cover every feature.The consumer must preserve these structural edges even when their S blocks are entirely zero. Merely adding zero blocks on the producer side does not fix an inference graph builder that re-filters them by overlap nonzeros.
4. Adding an OpenMX-compatible output
ABACUS
HContaineralready provides the atom-block organization needed for a new exporter. Two compatibility levels should be specified separately:.scfoutfile..scfoutcompatibility: target a specific binary format version and validate with the unmodified OpenMX reader. Beyond H/S, the reader expects position/momentum matrices, density matrices, metadata, and an input-file section. Missing physical quantities must not be silently replaced with zeros and advertised as fully supported.A real
.scfoutexporter must convert H from Ry to Hartree, preserve dimensionless S, map orbital ordering and real-spherical-harmonic phases, and implement the collinear/noncollinear spin and complex-number conventions. It should serialize the ABACUS structural graph, including its longer-range H-only pairs, rather than recreate OpenMX's distance cutoff. Each target postprocessor must be checked for assumptions about that neighbor list and about OpenMX-specific PAO/basis data. File readability does not establish compatibility with every OpenMX postprocessor or imply identical ABACUS/OpenMX basis functions.Suggested implementation order and acceptance checks
get_s, with shared-block NPZ output if useful; preserve existing format defaults..scfoutexport as a separately scoped feature once the required consumers and supported observables are defined.Regression checks should retain both H-only pairs in the four-Si reproducer, verify zero S blocks and all H spin channels, preserve directed/Hermitian partner keys, and check converted H(k), S(k), and generalized eigenvalues against the ABACUS matrices within the stated output precision. Test both SCF conversion and inference graph construction without H, including
nspin=1/2/4and MPI. For.scfout, additionally require unmodified-reader round trips and explicit tests of every advertised observable.