Testing

Cocoa uses Catch2 for unit and integration testing, with Kokkos/Trilinos initialization handled in a custom main entry point.

Test Framework

Cocoa uses Catch2 (v3) as its testing framework. Tests are organized using TEST_CASE and SECTION macros:

#include <catch2/catch_test_macros.hpp>

TEST_CASE("Mesh", "[mesh]") {
  // Setup code runs for each section
  const auto mesh = create_test_mesh();

  SECTION("Valid mesh construction") {
    REQUIRE(mesh.num_nodes() == 9);
    REQUIRE(mesh.num_elements() == 8);
  }

  SECTION("Coordinate access") {
    const auto& nodes = mesh.nodes();
    REQUIRE_THAT(nodes[0].location().x(), WithinAbs(-88.0, 1e-10));
  }
}

The test executable uses a custom main() to initialize Tpetra/Kokkos before running tests:

#define CATCH_CONFIG_RUNNER
#include <catch2/catch_all.hpp>
#include "Tpetra_Core.hpp"

int main(int argc, char* argv[]) {
  Tpetra::ScopeGuard tpetra_scope_guard(&argc, &argv);
  return Catch::Session().run(argc, argv);
}

Building and Running Tests

# Build with tests (enabled by default)
mkdir build && cd build
cmake ..
make -j8

# Run all tests via CTest
ctest --output-on-failure

# Run the test executable directly
./test/cocoa_tests

# Run specific test cases by tag
./test/cocoa_tests "[mesh]"
./test/cocoa_tests "[gwce]"

# Run specific test case by name
./test/cocoa_tests "Mesh"

# List all available tests
./test/cocoa_tests --list-tests

Test Files

Test files are located in the test/ directory:

File

Description

test_main.cpp

Entry point with Tpetra/Kokkos initialization

test_mesh.cpp

Mesh construction and access

test_neighbor_table.cpp

Node connectivity and neighbor tables

test_point2d.cpp

2D point and geometry operations

test_projection.cpp

Coordinate projection (CPP)

test_datetime.cpp

DateTime and TimeDelta classes

test_timestepper.cpp

Time stepping iteration

test_temporal_field.cpp

Multi-level temporal field storage

test_gwce_consistent.cpp

GWCE consistent (implicit) solver and matrix assembly

test_gwce_lumped.cpp

GWCE lumped (explicit) solver

test_wetdry.cpp

Wetting and drying algorithm

test_tide_potential.cpp

Tidal potential forcing

test_tidal_boundary.cpp

Tidal boundary conditions

test_momentum.cpp

Momentum solver

test_bottom_friction.cpp

Bottom friction computation

test_coriolis.cpp

Coriolis force computation

test_advection.cpp

Advection terms

test_pressure_gradient.cpp

Pressure gradient computation

test_inline_lateral_stress.cpp

Inline lateral stress computation

test_smagorinsky.cpp

Smagorinsky turbulence model

test_basis_functions.cpp

Finite element basis functions

test_coordinate_rotation.cpp

Coordinate rotation for global meshes

test_ramp_function.cpp

Forcing ramp functions

test_configuration_reader.cpp

YAML configuration reader

test_netcdf_io.cpp

NetCDF I/O operations

test_logger.cpp

Logging system

test_tide_astronomic.cpp

Astronomical tide potential

test_forcing_manager.cpp

Forcing manager orchestration

test_boundary_type.cpp

Boundary type classification

test_flow_boundary.cpp

Flow boundary conditions (\(Q_{\text{force}}\) kernels, time series, segment integration)

test_schema_engine.cpp

JSON Schema validation engine (valijson glue: paths, typo suggestions, weak-type semantics)

test_schema_validation.cpp

Configuration schema validation against the embedded schema document

test_distributed.cpp

Distributed computing (serial mode)

test_distributed_mpi.cpp

MPI distributed computing (multi-rank)

Test Data

Test data and helper utilities are organized as follows:

test/
├── TestMeshData.hpp              # Test mesh generation utilities
├── TestPrecision.hpp             # Mixed-precision test tolerance helpers
├── test_*.cpp                    # Test source files
├── run_integration_test.cmake    # Integration test runner script
└── data/
    ├── test_config.yaml          # ConfigurationReader test fixture
    └── wnat/                     # WNAT integration test data
        ├── wnat.nc                             # Mesh file
        ├── cocoa_config_integration_test.yaml  # Tides-only config
        ├── compare_tides_reference.py          # Reference comparison script
        ├── plot_comparison.py                  # Comparison plot generator
        ├── wnat_reference_fp64.nc              # Serial reference
        └── wnat_reference_mpi_fp64.nc          # MPI reference

Runnable example cases live in the cocoa-examples repository; the fixtures above are separate copies maintained with the tests.

See Test Cases for detailed descriptions of each test case, including expected results and verification plots.

The TestMeshData.hpp helper provides functions to create small test meshes:

#include "TestMeshData.hpp"

using TestData::TestMeshData;

TEST_CASE("Example", "[example]") {
  const auto [x_coords, y_coords] = TestMeshData::create_coordinates();
  const auto elevation = TestMeshData::create_elevation();
  const auto triangles = TestMeshData::create_triangles();
  const auto boundaries = TestMeshData::empty_boundaries();

  // Construct HostMesh (held by unique_ptr), then wrap in Mesh
  auto host_mesh = std::make_unique<Geometry::HostMesh>(
      x_coords, y_coords, elevation, triangles, boundaries,
      TestMeshData::origin, TestMeshData::projection_type);
  const Cocoa::Mesh mesh(std::move(host_mesh));
}

Catch2 Assertions

Common assertions used in Cocoa tests:

Basic Assertions:

REQUIRE(expr);           // Fatal assertion
CHECK(expr);             // Non-fatal assertion
REQUIRE_FALSE(expr);     // Require expression is false
REQUIRE_NOTHROW(expr);   // Require no exception thrown
REQUIRE_THROWS_AS(expr, ExceptionType);  // Require specific exception

Floating-Point Comparisons:

#include <catch2/catch_approx.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>

using Catch::Approx;
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;

REQUIRE(value == Approx(expected));              // Default tolerance
REQUIRE(value == Approx(expected).epsilon(1e-6)); // Custom tolerance
REQUIRE_THAT(value, WithinAbs(expected, 1e-10)); // Absolute tolerance
REQUIRE_THAT(value, WithinRel(expected, 1e-6)); // Relative tolerance

Sections for Test Organization:

TEST_CASE("Component", "[tag]") {
  // Setup code runs before each section
  auto component = create_component();

  SECTION("First behavior") {
    // Test first behavior
    REQUIRE(component.first_method() == expected);
  }

  SECTION("Second behavior") {
    // Test second behavior (fresh component)
    REQUIRE(component.second_method() == expected);
  }
}

Writing New Tests

  1. Create or edit a test file in test/:

#include <catch2/catch_test_macros.hpp>
#include "your_header.hpp"

TEST_CASE("ComponentName", "[component][tag]") {
  SECTION("Descriptive behavior name") {
    // Arrange
    auto obj = create_object();

    // Act
    auto result = obj.method();

    // Assert
    REQUIRE(result == expected);
  }
}

  1. Add the file to test/CMakeLists.txt if creating a new file:

set(COCOA_TEST_SOURCES
    test_main.cpp
    # ... existing files ...
    test_your_new_file.cpp)

  1. Rebuild and run:

make cocoa_tests
./test/cocoa_tests "[your_tag]"

Compile-Time Tests

For constexpr functionality, use static_assert:

// Compile-time tests for constexpr functions
static_assert(TimeDelta::fromDays(1).totalHours() == 24,
              "Day to hours conversion failed");
static_assert(TimeDelta::fromHours(24) == TimeDelta::fromDays(1),
              "Equality comparison failed");

Integration Tests

In addition to unit tests, Cocoa includes end-to-end integration tests that run a complete simulation on the Western North Atlantic (WNAT) mesh and compare results against known-good reference solutions. These tests verify that the full simulation pipeline (mesh loading, tidal forcing with SAL, time stepping, output) produces bitwise-reproducible results.

Output is always delivered on a background writer thread, so every run exercises the asynchronous write path; there is no I/O mode to select.

Integration tests span several categories: WNAT tides (explicit and implicit GWCE, serial and MPI), the GAHM vortex-only and composed-with-gridded-met cases, the global tidal case (serial, split-output, and MPI), the internal weir case, and checkpoint/restart round-trips. Each test carries a RESOURCE_LOCK where it shares a working directory with siblings (see Shared Working Directories below). The authoritative, current list of integration tests – their names, rank counts, and configs – is test/CMakeLists.txt; it is not duplicated here because that duplication drifts as tests are added. Search it for add_test(NAME Integration_ to enumerate them.

Each WNAT test runs a 1-day tides-only simulation (M2 boundary + tide potential + SAL) with \(\Delta t\) = 20 s and compares the output against a serial reference solution. The MPI variants use a separate reference because mesh partitioning changes floating-point operation order.

Integration test tolerances are NOT uniform: they range from 1e-3 to 1e-10 depending on what the comparison must absorb. Serial runs against a same-arithmetic serial reference use the tightest tolerance (1e-10 for checkpoint round-trips with no intervening solver iteration; 1e-6 for most serial simulation comparisons). MPI runs are relaxed to 1e-3 because partitioning changes the order of floating-point reductions; checkpoint restart through the iterative consistent-GWCE solver is relaxed to 1e-5 because the CG solver’s own convergence tolerance is 1e-5. Each -DTOLERANCE= value in test/CMakeLists.txt carries a comment explaining why that value and not a tighter one – treat those comments, not this page, as the authority; do not tighten or loosen a tolerance without updating its comment to match.

The comparison script also verifies that wet/dry state (NaN locations) matches between test and reference at every timestep.

After comparison, a Cartopy-based plotting script generates a visual summary (wnat_comparison.png) with peak water level and velocity maps, difference maps, and time series at probe points. In CI, this plot is uploaded as a GitHub Actions artifact.

Prerequisites:

Integration tests require Python 3 with numpy, xarray, netCDF4, and cartopy. If these modules are not available, the integration tests are silently disabled at CMake configure time.

CMake options:

Integration tests are off by default and must be explicitly enabled:

# Enable integration tests
cmake .. -Dcocoa_ENABLE_INTEGRATION_TESTS=ON

In CI, integration tests are enabled for the serial build-and-test jobs and the coverage job. GPU compile-check jobs do not run tests.

Running integration tests selectively:

# Run only integration tests
ctest -R Integration

# Run only the serial integration test
ctest -R Integration_WNAT_tides$

# Run only the MPI integration test
ctest -R Integration_WNAT_tides_mpi

# Exclude integration tests (run unit tests only)
ctest -E Integration

Regenerating reference solutions:

If numerical methods change and produce different (but correct) results, the reference solutions must be regenerated with test/regenerate_references.sh – do not hand-run cocoa and mv the output; the script is the single source of truth for which config produces which reference file, and it enforces the conditions those references depend on:

# Regenerate everything (wnat, global, weir, gahm)
test/regenerate_references.sh --cocoa build/src/cocoa/cocoa

# Regenerate one case only
test/regenerate_references.sh --cocoa build/src/cocoa/cocoa wnat

Run it inside the build container (or an environment with the same compiler, Trilinos, and Kokkos as CI) so the regenerated references are reproducible: every run is single-threaded (OMP_NUM_THREADS=1), so the bytes it produces are bit-reproducible run to run on the same toolchain. The script clears any stale cocoa_output.nc before each run so a failed run cannot silently leave a misleading reference, and it discards the transient cocoa_output_peak.nc file rather than storing it. It regenerates the serial WNAT, MPI WNAT, implicit WNAT (serial and MPI), GAHM vortex-only, global, and internal-weir references; split-output and MPI variants of the global and weir cases compare against the serial reference rather than carrying their own, so those are not separately regenerated.

Commit the regenerated .nc files with provenance in the commit body: the commit and config that produced them, and why (what numerical change made the old references stale). Regenerate references deliberately, never as a drive-by fix for an unrelated failing test – a reference that moves without a documented reason erases the test’s ability to catch a real regression.

Shared working directories:

Integration tests run serially (label integration) because several of them share a working directory and would otherwise race on the same input and output files. Every test that runs against test/data/wnat carries set_tests_properties(... PROPERTIES RESOURCE_LOCK wnat_workdir) – currently nine tests, spanning the WNAT tides, implicit-tides, split-output, failstop, and GAHM (vortex-only, composed, composed-domains) cases; search test/CMakeLists.txt for RESOURCE_LOCK wnat_workdir for the exact, current set. A new test that reads or writes into test/data/wnat must take the same lock, or it can run concurrently with an existing WNAT test and corrupt its output. Tests rooted in their own directory (global_case, internal_weir, and the checkpoint tests under those directories) do not share a workdir with any other case and so carry no RESOURCE_LOCK – only add one when a new test genuinely reads or writes files another test also touches.

Debugging Failed Tests

# Run with verbose output
./test/cocoa_tests -s "[failing_tag]"

# Run under debugger
lldb ./test/cocoa_tests -- "[failing_tag]"

# Run specific named test
./test/cocoa_tests "Exact Test Name"

# Show test durations
./test/cocoa_tests -d yes