DoPhys Core Architecture & API Reference

1. Installation & Hardware Determinism Check

Precompiled manylinux_2_31 wheels are published for Python 3.10–3.12 against CUDA 12.2+ and ROCm 6.1 runtimes. The binary distribution bundles libdophys_core.so, PTX fatbins for $\text{SM\_80}, \text{SM\_86}, \text{SM\_89}, \text{SM\_90a}$, and XLA custom call targets.

# Install CUDA 12.4 wheel from the DoPhys PEP-503 package index
pip install dophys-core==2.4.2+cu124 --extra-index-url https://pkg.dophys.top/simple/

# Run hardware roofline & Kulisch super-accumulator bit-reproducibility verification
python -m dophys.diagnostics --check-determinism --test-gpudirect

2. Multi-Node SLURM & Apptainer Deployment

For distributed InfiniBand NDR/HDR clusters, pull our reproducible ORAS Apptainer image compiled with OpenMPI 5.0.3, UCX 1.16 (--with-gdrcopy --with-verbs), and NVSHMEM 2.11 for device-initiated halo exchange.

# Pull signed SIF image via ORAS
apptainer pull dophys_2.4.2_cuda12.4.sif oras://registry.dophys.top/production/core:2.4.2-cuda12.4

# Launch 16-GPU (4 nodes x 4 H100) job with GPUDirect RDMA enabled
export UCX_TLS=rc_x,sm,cuda_copy,gdr_copy,cuda_ipc
export NVSHMEM_SYMMETRIC_SIZE=4G
srun -N 4 --ntasks-per-node=4 --cpus-per-task=16 --gpus-per-task=1 --gpu-bind=closest \
    apptainer exec --nv dophys_2.4.2_cuda12.4.sif \
    dophys-exec --deck decks/stator_cht_8.9m.toml --checkpoint-interval 500

3. Hardware & Driver Support Matrix

4. Dual-Graph Partitioning & $\text{AoSoA}$ Memory Layout

Unstructured finite-volume solvers are bottlenecked by irregular indirect memory accesses u[owner[f]] and u[neighbor[f]] across interior faces $f \in \mathcal{F}_{\text{int}}$. When loading CGNS (HDF5) or Gmsh v4 meshes, dophys.Mesh performs a three-stage preprocessing pipeline:

1. $k$-way Dual-Graph Partitioning: Constructs the cell adjacency graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ weighted by element polynomial order $(p_i + 1)^3$ and minimizes edge-cut communication volume across MPI ranks via ParMETIS.
2. Bandwidth-Minimizing Permutation ($\text{RCM} + \text{Morton } Z_{64}$): Within each rank, cells are reordered to minimize the matrix profile $\beta(\mathbf{A}) = \max_{i} \max_{j: A_{ij} \ne 0} |i - j|$, ensuring that $\ge 91\%$ of face neighbor lookups hit the GPU L2 cache ($128\text{-byte}$ cache lines).
3. $\text{AoSoA-}W$ Vectorized Storage: Conserved variables $\mathbf{U} = [\rho, \rho u_x, \rho u_y, \rho u_z, \rho E]^T \in \mathbb{R}^{N_c \times 5}$ are tiled into warp-coalesced blocks of width $W = 16$ (for FP64, $16 \times 8\text{ B} = 128\text{ B}$ sector alignment):

5. Matrix-Free Newton-Krylov & $\text{PMIS}$ Algebraic Multigrid

At each implicit time step, DoPhys solves the nonlinear residual system $\mathcal{R}(\mathbf{u}) = \mathbf{0}$ via Newton-FGMRES with Eisenstat-Walker adaptive inexact forcing $\eta_k$:

The right-preconditioned Krylov subspace $\mathcal{K}_m(\mathcal{J} \mathcal{M}_{\text{AMG}}^{-1}, \mathbf{r}_0)$ uses a smoothed-aggregation Algebraic Multigrid preconditioner $\mathcal{M}_{\text{AMG}}^{-1}$ built entirely on GPU:

• Strength of Connection & PMIS Coarsening: Edges $(i,j)$ are classified as strongly coupled if $-a_{ij} \ge \theta_{\text{str}} \max_{k \ne i} (-a_{ik})$ with $\theta_{\text{str}} = 0.25$. Aggregates $\mathcal{A}_I$ are formed in parallel via Luby's Parallel Modified Independent Set ($\text{PMIS}$) algorithm.
• Smoothed Prolongation Operator $P_l$: Tentative piece-wise constant prolongators $\tilde{P}_l$ are smoothed via one damped Jacobi sweep $P_l = (I - \omega D_l^{-1} A_l^F) \tilde{P}_l$ with spectral radius bound $\omega = \frac{4}{3 \hat{\rho}(D_l^{-1} A_l)}$, followed by Galerkin triple-product coarse assembly $A_{l+1} = P_l^T A_l P_l$.
• Sync-Free $\ell_1$-Jacobi Chebyshev Polynomial Smoother: To avoid global reductions inside V-cycles, level smoothing applies a 4th-degree Chebyshev polynomial on the interval $[\alpha \hat{\lambda}_{\max}, \beta \hat{\lambda}_{\max}]$ ($\alpha = 0.06, \beta = 1.1$) preconditioned by the unconditionally convergent $\ell_1$-diagonal $M_{ii}^{\ell_1} = a_{ii} + \sum_{j \ne i} |a_{ij}|$.

6. $\text{ESDIRK}$ Time Integration & Discrete Adjoints

For stiff transient multiphysics, DoPhys implements a 6-stage, 4th-order $L$-stable, stiffly accurate Explicit Singly Diagonally Implicit Runge-Kutta scheme ($\text{ESDIRK4(3)6L[2]SA}$). Because all implicit stages share the identical diagonal Butcher coefficient $\gamma = 0.25$, the AMG preconditioner for the stage Jacobian $\mathbf{G}_i = \mathbf{M} - \Delta t \, \gamma \frac{\partial \mathbf{F}}{\partial \mathbf{u}}$ can be factored once and reused across stages $i = 2, \dots, 6$:

During reverse-mode differentiation (solver.step_with_adjoint), stage adjoint variables $\boldsymbol{\Psi}^{(i)}$ are solved in reverse stage order $i = 6, \dots, 2$ using the transposed stage Jacobian $\mathbf{G}_i^T$:

7. C++20 Hard Real-Time HIL Runtime & SPSC RingBuffer

When compiled with -DDOPHYS_NO_ALLOC=ON, operator new and cudaMalloc are poisoned after initialization. Telemetry frames from EtherCAT or FPGA DMA are ingested via a cache-line-padded, lock-free Single-Producer Single-Consumer ($\text{SPSC}$) ring buffer:

#include <atomic>
#include <sys/mman.h>
#include <dophys/hil/solver_context.hpp>

namespace dophys::hil {

template <typename T, size_t CapacityPow2 = 1024>
struct alignas(64) WaitFreeRingBuffer {
    static_assert((CapacityPow2 & (CapacityPow2 - 1)) == 0, "Power of 2 required");
    alignas(64) std::atomic<size_t> write_idx_{0};
    alignas(64) std::atomic<size_t> read_idx_{0};
    alignas(64) T slots_[CapacityPow2];

    [[nodiscard]] bool try_push(const T& frame) noexcept {
        const size_t w = write_idx_.load(std::memory_order_relaxed);
        const size_t next_w = (w + 1) & (CapacityPow2 - 1);
        if (next_w == read_idx_.load(std::memory_order_acquire)) return false;
        slots_[w] = frame;
        write_idx_.store(next_w, std::memory_order_release);
        return true;
    }
};

} // namespace dophys::hil

8. Production Solver Configuration Deck (.toml)

Standalone batch jobs executed via dophys-exec are configured via strongly-typed TOML input decks. Below is the complete reference configuration for the $8.92 \times 10^6\text{-DoF}$ micro-channel cold plate benchmark:

# decks/stator_cht_8.9m.toml — DoPhys v2.4.2 Schema
[mesh]
file = "grids/stator_cooling_v4_8.9m.cgns"
precision = "fp64"
partitioner = "parmetis_dual"
reordering = "rcm_morton_64"
aosoa_width = 16
halo_layers = 2

[physics.fluid]
model = "navier_stokes_low_mach"
convective_flux = "hllc_lm_weno5z"
viscous_flux = "nipg_symm_interior_penalty"
prandtl = 6.91

[physics.solid]
model = "anisotropic_fourier_conduction"
kappa_tensor_w_mk = [385.0, 385.0, 385.0]
interface_coupling = "monolithic_robin_schur"

[solver.nonlinear]
time_scheme = "esdirk4_3_6L2sa"
dt_seconds = 2.5e-4
newton_rtol = 1.0e-9
newton_atol = 1.0e-13
eisenstat_walker = true

[solver.linear]
krylov_type = "pipelined_fgmres"
orthogonalization = "cgs2_kulisch_2048"
max_restart = 48
amg_coarsening = "pmis"
amg_strength_theta = 0.25
amg_smoother = "chebyshev_deg4_l1_jacobi"
amg_cycle = "v_cycle_fp32_refinement"

[adjoint]
enabled = true
checkpoint_strategy = "revolve_binomial"
vram_snap_budget_mb = 16384

9. Runtime Environment Variables