============ Architecture ============ This document describes the software architecture of Cocoa. High-Level Overview ------------------- Cocoa follows a modular architecture with clear separation of concerns, built on the Kokkos [Edwards2014]_ [Trott2022]_ performance portability framework within the Trilinos [Heroux2005]_ ecosystem: .. graphviz:: :align: center :caption: High-level architecture showing the simulation pipeline digraph architecture { rankdir=TB; node [shape=box, style="filled,rounded", fontname="Helvetica", fontsize=11]; edge [color="#555555"]; bgcolor="transparent"; // Application layer subgraph cluster_app { label=""; style=invis; cocoa [label="Main Application\ncocoa.cpp", fillcolor="#E8F5E9", color="#43A047"]; } // Simulation layer subgraph cluster_sim { label=""; style=invis; sim [label="Simulation\nSimulation.hpp", fillcolor="#E3F2FD", color="#1E88E5"]; } // Orchestration layer subgraph cluster_orch { label=""; style=invis; ts [label="TimeStepper", fillcolor="#E3F2FD", color="#1E88E5"]; fm [label="ForcingManager", fillcolor="#FFF3E0", color="#FB8C00"]; bcm [label="BoundaryCondition\nManager", fillcolor="#FFF3E0", color="#FB8C00"]; cm [label="Communication\nManager", fillcolor="#F3E5F5", color="#8E24AA"]; om [label="OutputManager", fillcolor="#FFF3E0", color="#FB8C00"]; } // Solver layer subgraph cluster_solvers { label=""; style=invis; gwce [label="GwceSolver\n(continuity)", fillcolor="#E3F2FD", color="#1E88E5"]; mom [label="MomentumSolver\n(velocity)", fillcolor="#E3F2FD", color="#1E88E5"]; wd [label="WetDry\n(flooding)", fillcolor="#E3F2FD", color="#1E88E5"]; } // Data layer subgraph cluster_data { label=""; style=invis; mf [label="ModelFields\n(centralized data)", fillcolor="#FFEBEE", color="#E53935"]; } // Edges cocoa -> sim; sim -> ts; sim -> fm; sim -> bcm; sim -> cm; sim -> om; ts -> gwce; ts -> mom; ts -> wd; gwce -> mf; mom -> mf; wd -> mf; fm -> mf [style=dashed]; cm -> mf [style=dashed]; // Rank alignment {rank=same; ts; fm; bcm; cm; om} {rank=same; gwce; mom; wd} } Source Library Dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The source libraries have a clear dependency hierarchy: .. graphviz:: :align: center :caption: Library dependencies (solid = internal, dashed = external) digraph libraries { rankdir=TB; node [shape=box, style="filled,rounded", fontname="Helvetica", fontsize=11]; edge [color="#555555"]; bgcolor="transparent"; // Internal libraries cocoa [label="cocoa\n(executable)", fillcolor="#E8F5E9", color="#43A047"]; io [label="cocoa_io\n(I/O library)", fillcolor="#FFF3E0", color="#FB8C00"]; kernel [label="cocoa_kernel\n(compute library)", fillcolor="#E3F2FD", color="#1E88E5"]; meteo [label="cocoa_meteo\n(meteorological I/O)", fillcolor="#BBDEFB", color="#1565C0"]; dt [label="cocoa_datetime\n(datetime library)", fillcolor="#F3E5F5", color="#8E24AA"]; ct [label="cocoa_compute_types\n(math value types)", fillcolor="#E1F5FE", color="#0288D1"]; consts [label="cocoa_constants\n(constants, ordinals)", fillcolor="#E1F5FE", color="#0288D1"]; // External dependencies node [shape=box, style="dashed,rounded", fillcolor="#F5F5F5", color="#9E9E9E"]; trilinos [label="Trilinos\n(Kokkos, Tpetra, Belos)"]; netcdf [label="NetCDF-C"]; mpi [label="MPI\n(optional)"]; yaml [label="yaml-cpp"]; // Internal edges cocoa -> io; cocoa -> kernel; io -> kernel; io -> dt; kernel -> dt; kernel -> meteo; kernel -> ct; kernel -> consts; ct -> consts; meteo -> dt; // External edges kernel -> trilinos [style=dashed]; kernel -> mpi [style=dashed]; io -> netcdf [style=dashed]; io -> yaml [style=dashed]; meteo -> netcdf [style=dashed]; {rank=same; io; kernel} } Data Container Hierarchy ^^^^^^^^^^^^^^^^^^^^^^^^ ``ModelFields`` is the centralized data container passed through the simulation: .. graphviz:: :align: center :caption: ModelFields data container hierarchy digraph containers { rankdir=TB; node [shape=record, style="filled", fontname="Helvetica", fontsize=10]; edge [color="#555555"]; bgcolor="transparent"; mf [label="{ModelFields|Centralized data container\lpassed to all solvers}", fillcolor="#FFEBEE", color="#E53935"]; hydro [label="{HydrodynamicState|elevation (3 time levels)\lvelocity (3 time levels)\lflux (3 time levels)}", fillcolor="#E3F2FD", color="#1E88E5"]; physics [label="{PhysicsData|bathymetry}", fillcolor="#FFF3E0", color="#FB8C00"]; friction [label="{FrictionFields|tkm_xx, tkm_xy, tkm_yy}", fillcolor="#FFF8E1", color="#F9A825"]; lateral [label="{LateralStressFields|sigma_xx, sigma_xy\lsigma_yx, sigma_yy}", fillcolor="#FFF8E1", color="#F9A825"]; tidepot [label="{TidePotentialFields|potential (2 levels)}", fillcolor="#FFF8E1", color="#F9A825"]; meteoff [label="{MeteoForcingFields|pressure, wind\lwind stress}", fillcolor="#FFF8E1", color="#F9A825"]; wetdry [label="{WetDryData|node_status (3 arrays)\lelement_active\lslope_limiter}", fillcolor="#E8F5E9", color="#43A047"]; geom [label="{DeviceMesh|coordinates, connectivity\lbasis function derivatives\lnode-to-element CSR}", fillcolor="#F3E5F5", color="#8E24AA"]; rotation [label="{OptionalRotationData|rotation matrices\l(global meshes only)}", fillcolor="#F5F5F5", color="#9E9E9E"]; mf -> hydro; mf -> physics; mf -> wetdry; mf -> geom; mf -> rotation [style=dashed]; physics -> friction; physics -> lateral; physics -> tidepot; physics -> meteoff; } Directory Structure ------------------- .. code-block:: text src/ ├── cocoa/ # Main application │ ├── cocoa.cpp # Entry point │ └── CommandLineArgs.hpp # CLI argument parsing │ ├── cocoa_datetime/ # Date/time library │ ├── DateTime.hpp # Date/time representation │ └── TimeDelta.hpp # Time duration │ ├── cocoa_compute_types/ # Small math value types │ ├── Vec2.hpp # 2D vector │ ├── Mat2.hpp # 2x2 matrix │ ├── Mat3.hpp # 3x3 matrix │ └── MathUtils.hpp # Math helpers │ ├── cocoa_constants/ # Shared constants and scalar/index types │ ├── Constants.hpp # Physical constants │ ├── Defaults.hpp # Default parameter values │ ├── Thresholds.hpp # Centralized numerical thresholds │ └── Ordinals.hpp # Scalar/index types │ ├── cocoa_io/ # I/O utilities library │ ├── ConfigurationReader.hpp # YAML config parsing │ ├── Logger.hpp # Logging facade │ ├── NetcdfReader.hpp # NetCDF input │ ├── NetcdfWriter.hpp # NetCDF output │ └── NetcdfCommon.hpp # Shared NetCDF types │ ├── cocoa_meteo/ # Meteorological I/O library │ ├── core/ # Base classes and data types │ │ ├── MeteoReaderBase.hpp # Abstract reader interface │ │ ├── MeteoReaderConcept.hpp # C++20 concept for readers │ │ ├── MeteoReaderConfig.hpp # Format-agnostic configuration │ │ ├── MeteoFormat.hpp # Format enum │ │ ├── MeteoGrid.hpp # Regular/irregular grid types │ │ ├── MeteoField.hpp # 2D field container │ │ ├── MeteoFieldSnapshot.hpp # Time snapshot bundle │ │ └── CfTimeUtils.hpp # CF time unit parsing │ ├── readers/ # Format-specific readers │ │ ├── cf/CfNetcdfReader.hpp │ │ ├── owi_ascii/OwiAsciiReader.hpp │ │ └── owi_netcdf/OwiNetcdfReader.hpp │ ├── interpolation/ # Spatial interpolation │ │ ├── MeteoSpatialInterpolator.hpp │ │ ├── MultiDomainInterpolator.hpp │ │ ├── FieldInterpolation.hpp │ │ └── InterpolationWeight.hpp │ └── MeteoReaderFactory.hpp # Reader instantiation │ └── cocoa_kernel/ # Core computational library ├── core/ # Domain-meaningful primitives │ ├── Execution.hpp # Execution space selection │ ├── KokkosProfileRegion.hpp # Performance profiling │ ├── ModelConfiguration.hpp # Configuration struct │ ├── NarrowCast.hpp # Checked numeric casts │ ├── config/ # Per-domain config sub-structs │ │ ├── FlowBoundaryConfig.hpp │ │ ├── ForcingConfig.hpp │ │ ├── GwceConfig.hpp │ │ ├── HotStartConfig.hpp │ │ ├── MeshConfig.hpp │ │ ├── ModelConfigurationFactory.hpp # YAML -> config (ex-io/) │ │ ├── OutputConfig.hpp │ │ ├── PhysicsConfig.hpp │ │ ├── SimulationConfig.hpp │ │ └── TidalConfig.hpp │ └── types/ # Generic infrastructure types │ ├── KokkosAliases.hpp # View type aliases │ ├── LinearAlgebraTypes.hpp # Trilinos type aliases │ ├── Precision.hpp # Float/double storage precision │ ├── RingBuffer.hpp # Ring buffer type │ └── TemporalField.hpp # Multi-level temporal storage │ ├── data/ # Field data structures │ ├── ModelFields.hpp # Master data container │ ├── HydrodynamicState.hpp # Elevation/velocity state │ ├── PhysicsData.hpp # Physics parameters │ └── WetDryData.hpp # Wet/dry algorithm state │ ├── geometry/ # Mesh and geometry │ ├── Mesh.hpp # Top-level mesh: HostMesh + DeviceMesh + dist context │ ├── HostMesh.hpp # Host-side topology (nodes, elements, boundaries) │ ├── DeviceMesh.hpp # Device-side FE cache (gradients, areas, CSR) │ ├── Element.hpp # Triangle element │ ├── Node.hpp # Mesh node │ ├── Point.hpp # 2D/3D point │ ├── NeighborTable.hpp # Mesh connectivity │ ├── BasisFunctions.hpp # FE shape functions │ ├── GlobalMeshHandler.hpp # Global mesh operations │ ├── CoordinateRotation.hpp # Coordinate rotation │ ├── RotationData.hpp # Rotation matrices │ ├── VelocityTransform.hpp # Velocity transformations │ ├── ProjectionScaleFactor.hpp # Map projection scaling │ ├── ProjectionTransformer.hpp # Coordinate transforms │ ├── NodalAttribute.hpp # Nodal attribute type │ ├── NodalAttributeData.hpp # Attribute storage │ ├── NodalAttributeRegistry.hpp # Attribute registry │ ├── partition/ # Mesh partitioning (ex-top-level) │ │ ├── MeshPartitioner.hpp # Zoltan2 partitioning │ │ ├── PartitionCache.hpp # Cached partitions │ │ └── PartitionInfo.hpp # Partition metadata │ └── boundaries/ # Boundary data structures │ ├── BoundaryData.hpp │ ├── BoundaryRawData.hpp │ ├── BoundaryType.hpp │ └── BoundaryView.hpp │ ├── simulation/ # Simulation control │ ├── Simulation.hpp # Main simulation driver │ ├── TimeStepper.hpp # Time stepping logic │ ├── TimestepLogger.hpp # Per-step status logging (ex-io/) │ └── Diagnostics.hpp # Solution monitoring │ ├── numeric/ # Numerical algorithms │ ├── continuity/ # GWCE solver │ │ ├── GwceSolver.hpp # Solver interface │ │ ├── GwceVectorAssembler.hpp # RHS assembly │ │ ├── GwceVectorAssemblyKernels.hpp │ │ ├── GwcePreprocessingKernels.hpp │ │ ├── GwceCommonKernels.hpp # Shared GWCE kernels │ │ ├── OpenBoundaryCoefficients.hpp │ │ ├── NodePositionSorter.hpp │ │ │ │ │ ├── consistent/ # Implicit solver │ │ │ ├── GwceSolverConsistent.hpp │ │ │ ├── GwceMatrixAssemblerConsistent.hpp │ │ │ ├── GwceMatrixAssemblyConsistentKernels.hpp │ │ │ ├── ConjugateGradientSolver.hpp │ │ │ └── JacobiPreconditioner.hpp │ │ │ │ │ └── lumped/ # Explicit solver │ │ ├── GwceSolverLumped.hpp │ │ ├── GwceMatrixAssemblerLumped.hpp │ │ ├── GwceMatrixAssemblyLumpedKernels.hpp │ │ └── GwceSolverLumpedKernels.hpp │ │ │ ├── momentum/ # Momentum solver │ │ ├── MomentumSolver.hpp │ │ ├── MomentumSolveKernels.hpp │ │ └── MomentumRhsKernels.hpp │ │ │ └── wetdry/ # Wet/dry algorithm │ ├── WetDry.hpp │ └── WetDryKernels.hpp │ ├── forcing/ # Boundary and body forcing │ ├── ForcingManager.hpp # Forcing orchestration │ ├── RampFunction.hpp # Time ramping │ ├── meteorological/ # Meteorological forcing │ │ ├── MeteoForcingProvider.hpp # Ring buffer + interpolation │ │ ├── MeteoForcingConfig.hpp # Configuration │ │ ├── MetSnapshot.hpp # Device-side snapshot slot │ │ └── DragLaw.hpp # Wind drag formulations │ └── tide/ # Tidal forcing │ ├── boundary/ # Tidal boundary conditions │ │ ├── TideBoundaryForcing.hpp │ │ └── TideBoundaryConstituent.hpp │ └── potential/ # Tide potential forcing │ ├── TidePotentialInterface.hpp │ ├── astronomical/ # Astronomical tide potential │ │ ├── AstronomicalTidePotential.hpp │ │ ├── AstronomicalTidePotentialAdapter.hpp │ │ ├── AstronomicParameters.hpp │ │ ├── AstronomicConstants.hpp │ │ ├── MoonPosition.hpp │ │ ├── SunPosition.hpp │ │ ├── MoonSunPositionCalculator.hpp │ │ └── SiderealTime.hpp │ └── harmonics/ # Harmonic tide potential │ ├── TidePotentialHarmonics.hpp │ └── TidePotentialConstituent.hpp │ ├── physics/ # Physics kernels │ ├── BottomFriction.hpp # Manning's friction │ ├── LateralStress.hpp # Lateral stress tensor │ ├── InlineLateralStress.hpp # Inline stress computation │ ├── PressureGradient.hpp # Pressure gradient terms │ ├── NonConservativeAdvection.hpp # Advection terms │ └── Smagorinsky.hpp # Smagorinsky turbulence │ ├── distributed/ # MPI distributed computing │ ├── DistributedContext.hpp # MPI context management │ ├── DistributedConfig.hpp # Configuration │ ├── CommunicationManager.hpp # Communication orchestration │ ├── GhostExchange.hpp # Ghost node exchange │ └── MapFactory.hpp # Tpetra map creation │ └── io/ # Kernel I/O (split by direction) ├── async/ # Background writer thread │ ├── OutputWriterThread.hpp # Bounded writer thread pool │ └── OutputSnapshot.hpp # Recycled write buffers ├── checkpoint/ # Restart/hot-start │ ├── HotStartReader.hpp │ └── HotStartWriter.hpp ├── input/ # Read-side │ ├── MeshReader.hpp │ ├── MeshGather.hpp │ ├── NodalAttributeReader.hpp │ ├── NodalAttributeInitializer.hpp │ └── SalDataReader.hpp └── output/ # Write-side ├── OutputManager.hpp ├── OutputFile.hpp ├── OutputVariable.hpp ├── OutputVariableRegistry.hpp ├── NetcdfFileSetup.hpp # Shared UGRID file setup ├── DistributedIO.hpp # Parallel field gather (ex-distributed/) └── DistributedPartitionWriter.hpp # Partition output (ex-distributed/) Namespace Organization ---------------------- .. graphviz:: :align: center :caption: Namespace hierarchy within the Cocoa project digraph namespaces { rankdir=TB; node [shape=box, style="filled,rounded", fontname="Helvetica", fontsize=10]; edge [color="#555555", arrowsize=0.7]; bgcolor="transparent"; compound=true; cocoa [label="Cocoa", fillcolor="#E0E0E0", color="#616161", fontsize=12, penwidth=2]; // Top-level namespaces types [label="Types\n(aliases, views)", fillcolor="#F5F5F5", color="#9E9E9E"]; core [label="Core\n(execution, config)", fillcolor="#F5F5F5", color="#9E9E9E"]; constants [label="Constants", fillcolor="#F5F5F5", color="#9E9E9E"]; defaults [label="Defaults", fillcolor="#F5F5F5", color="#9E9E9E"]; thresholds [label="Thresholds", fillcolor="#F5F5F5", color="#9E9E9E"]; data [label="Data\n(field containers)", fillcolor="#FFEBEE", color="#E53935"]; // Geometry geometry [label="Geometry\n(mesh, elements)", fillcolor="#F3E5F5", color="#8E24AA"]; boundaries [label="Boundaries", fillcolor="#F3E5F5", color="#8E24AA"]; // Simulation simulation [label="Simulation\n(driver, timestepper)", fillcolor="#E3F2FD", color="#1E88E5"]; // Numeric numeric [label="Numeric", fillcolor="#E3F2FD", color="#1E88E5"]; continuity [label="Continuity\n(GWCE)", fillcolor="#E3F2FD", color="#1E88E5"]; consistent [label="Consistent\n(implicit)", fillcolor="#BBDEFB", color="#1565C0"]; lumped [label="Lumped\n(explicit)", fillcolor="#BBDEFB", color="#1565C0"]; momentum [label="Momentum\n(velocity solver)", fillcolor="#E3F2FD", color="#1E88E5"]; wettingdrying [label="WettingDrying", fillcolor="#E3F2FD", color="#1E88E5"]; // Forcing forcing [label="Forcing", fillcolor="#FFF3E0", color="#FB8C00"]; tide [label="Tide", fillcolor="#FFF3E0", color="#FB8C00"]; tideboundary [label="Boundary", fillcolor="#FFE0B2", color="#E65100"]; tidepotential [label="Potential", fillcolor="#FFE0B2", color="#E65100"]; meteorological [label="Meteorological\n(wind, pressure)", fillcolor="#FFE0B2", color="#E65100"]; // Meteo library meteons [label="Meteo\n(cocoa_meteo)", fillcolor="#BBDEFB", color="#1565C0"]; // Physics / Distributed / IO physics [label="Physics\n(friction, stress)", fillcolor="#E8F5E9", color="#43A047"]; distributed [label="Distributed\n(MPI, ghost exchange)", fillcolor="#FCE4EC", color="#C62828"]; partition [label="Partition\n(ParMETIS)", fillcolor="#FCE4EC", color="#C62828"]; io [label="IO\n(mesh reader, output)", fillcolor="#FFF3E0", color="#FB8C00"]; // Edges cocoa -> types; cocoa -> core; cocoa -> constants; cocoa -> defaults; cocoa -> thresholds; cocoa -> data; cocoa -> geometry; cocoa -> simulation; cocoa -> numeric; cocoa -> forcing; cocoa -> physics; cocoa -> distributed; cocoa -> partition; cocoa -> io; geometry -> boundaries; numeric -> continuity; numeric -> momentum; numeric -> wettingdrying; continuity -> consistent; continuity -> lumped; forcing -> tide; forcing -> meteorological; tide -> tideboundary; tide -> tidepotential; cocoa -> meteons; } Meteorological Forcing Pipeline -------------------------------- The meteorological forcing subsystem moves atmospheric data from files on disk through a staged pipeline into device-resident Kokkos views consumed by the GWCE and momentum kernels. The ``cocoa_meteo`` library handles file reading and spatial interpolation on the host; the ``cocoa_kernel`` forcing subsystem manages temporal buffering and device transfer. .. graphviz:: :align: center :caption: Meteorological forcing data pipeline digraph meteo_pipeline { rankdir=LR; node [shape=box, style="filled,rounded", fontname="Helvetica", fontsize=10]; edge [color="#555555"]; bgcolor="transparent"; files [label="Meteo Files\n(NetCDF / ASCII)", fillcolor="#E3F2FD", color="#1E88E5"]; reader [label="MeteoReaderBase\n(format-specific read)", fillcolor="#E3F2FD", color="#1E88E5"]; interp [label="MultiDomainInterpolator\n(bilinear to mesh nodes)", fillcolor="#BBDEFB", color="#1565C0"]; deep_copy [label="Kokkos::deep_copy\n(upload to device slot)", fillcolor="#FFE0B2", color="#E65100"]; ring [label="Ring Buffer\n(8 device-side slots)", fillcolor="#C8E6C9", color="#2E7D32"]; kernels [label="GWCE / Momentum\nKernels", fillcolor="#A5D6A7", color="#1B5E20"]; files -> reader; reader -> interp; interp -> deep_copy; deep_copy -> ring; ring -> kernels [label="prev + next\n+ alpha"]; } Pipeline Stages ^^^^^^^^^^^^^^^ 1. **File Read (host).** A format-specific reader (``CfNetcdfReader``, ``OwiAsciiReader``, or ``OwiNetcdfReader``) reads one snapshot from disk and returns a ``MeteoFieldSnapshot`` containing the pressure, wind_u, and wind_v fields on the meteorological grid. 2. **Spatial Interpolation (host).** The ``MultiDomainInterpolator`` interpolates from the meteorological grid(s) to the local mesh nodes using precomputed bilinear weights. For multi-domain formats, the innermost valid domain takes priority. The result is three host-side arrays (pressure, wind_u, wind_v) of length ``num_local_nodes``. 3. **Host-to-Device Transfer.** The interpolated node arrays are uploaded into a device-resident ``MetSnapshot`` slot via ``Kokkos::deep_copy``. Each slot holds three ``Scalar1DView`` Kokkos views. 4. **Temporal Interpolation (device).** At each simulation timestep, the GWCE and momentum kernels read two adjacent snapshots from the ring buffer and blend them using the interpolation weight :math:`\alpha` (see :ref:`meteo-temporal-interp`). Ring Buffer ^^^^^^^^^^^ The ``MeteoForcingProvider`` maintains a fixed-size circular buffer of 8 ``MetSnapshot`` slots on the device. Each slot holds three Kokkos views (pressure, wind_u, wind_v) with one value per local mesh node. The slots are allocated once at initialization and reused throughout the simulation. At any given simulation time, two adjacent slots form the **active bracket**: the snapshot immediately before and after the current time. The temporal interpolation weight :math:`\alpha` selects the blend between them. The remaining slots are pre-filled with upcoming snapshots so that advancing the bracket never requires a synchronous file read followed by a device transfer. When the simulation time passes the end of the current bracket: 1. The oldest slot is recycled (the ring buffer pops its front). 2. The next unread snapshot is read from disk, spatially interpolated, and uploaded into the newly available slot. 3. The bracket indices advance by one. This design provides two benefits: - **Amortized transfer cost.** Because upcoming snapshots are pre-loaded into free slots, the GPU is never stalled waiting for a single ``deep_copy`` at bracket boundaries. - **No redundant work.** Once a snapshot is spatially interpolated and uploaded, it remains device-resident until the buffer rotates past it. Meteorological Input Distribution ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Rank 0 reads the meteorological files and broadcasts the spatially interpolated node arrays to all compute ranks via ``MPI_Bcast``, pre-reading the next snapshot on a background task so file access overlaps computation. Each compute rank maintains its own ring buffer of device-side data for its local mesh partition.