Testing

Cocoa uses Catch2 for the unit tests, with Kokkos/Trilinos initialization handled in a custom main entry point. The integration tests are ctest entries that run the cocoa binary and compare its output against a committed reference with a Python script.

Test Framework

Tests are written with Catch2 (v3) TEST_CASE and SECTION macros:

#include <catch2/catch_test_macros.hpp>

TEST_CASE("Mesh", "[mesh]") {
  // Setup code runs for each section
  const auto mesh = create_test_mesh();

  SECTION("Valid mesh construction") {
    REQUIRE(mesh.num_nodes() == 9);
    REQUIRE(mesh.num_elements() == 8);
  }

  SECTION("Coordinate access") {
    const auto& nodes = mesh.nodes();
    REQUIRE_THAT(nodes[0].location().x(), WithinAbs(-88.0, 1e-10));
  }
}

The test executable uses a custom main() to initialize Tpetra/Kokkos before running tests:

#define CATCH_CONFIG_RUNNER
#include <catch2/catch_all.hpp>
#include "Tpetra_Core.hpp"

int main(int argc, char* argv[]) {
  Tpetra::ScopeGuard tpetra_scope_guard(&argc, &argv);
  return Catch::Session().run(argc, argv);
}

Building and Running Tests

# Build with tests (enabled by default)
cmake --preset dev
cmake --build --preset dev

# Run all tests via CTest, from anywhere in the source tree
ctest --preset dev

# Run the test executable directly (paths below are relative to build/)
cd build
./test/cocoa_tests

# Run specific test cases by tag
./test/cocoa_tests "[mesh]"
./test/cocoa_tests "[gwce]"

# Run specific test case by name
./test/cocoa_tests "Mesh"

# List all available tests
./test/cocoa_tests --list-tests

Test Files

Test sources live in test/ as test_<component>.cpp, one file per component, with test_main.cpp as the entry point. The current set is COCOA_TEST_SOURCES in test/CMakeLists.txt; it is not listed here because a copy drifts as files are added. ./test/cocoa_tests --list-tests enumerates the test cases and --list-tags the tags they are filtered by.

Test Data

Test data and helper utilities are organized as follows:

test/
├── TestMeshData.hpp              # Test mesh generation utilities
├── TestPrecision.hpp             # Mixed-precision test tolerance helpers
├── test_*.cpp                    # Test source files
├── run_in_workdir.cmake          # Runs a driver in a private fixture copy
├── run_integration_test.cmake    # Default integration test driver
├── run_*_test.cmake              # Other drivers (checkpoint, serial-vs-MPI, ...)
├── regenerate_references.sh      # Reference regeneration (see below)
└── data/
    ├── test_config.yaml          # ConfigurationReader test fixture
    ├── baroclinic/               # Baroclinic forcing fixtures
    ├── channel/                  # Channel and beach cases (+ validation)
    ├── flow_channel/             # Flow-boundary case
    ├── global_case/              # Global tidal case
    ├── ice_drag/                 # Ice drag golden values
    ├── internal_tide_checkpoint/ # Internal tide checkpoint case
    ├── internal_weir/            # Internal weir case
    ├── meteo/                    # Per-format meteorological reader fixtures
    ├── partition/                # Partition cache fixtures
    ├── vortex/                   # ATCF tracks and GAHM control solutions
    └── wnat/                     # WNAT integration test data
        ├── wnat.nc                             # Mesh file
        ├── cocoa_config_*.yaml                 # One config per case
        ├── compare_tides_reference.py          # Reference comparison script
        ├── plot_comparison.py                  # Comparison plot generator
        └── *_reference{,_mpi}.nc               # Serial and MPI references

Runnable example cases live in the cocoa-examples repository; the fixtures above are separate copies maintained with the tests.

See Test Cases for detailed descriptions of each test case, including expected results and verification plots.

The TestMeshData.hpp helper provides functions to create small test meshes:

#include "TestMeshData.hpp"

using TestData::TestMeshData;

TEST_CASE("Example", "[example]") {
  const auto [x_coords, y_coords] = TestMeshData::create_coordinates();
  const auto elevation = TestMeshData::create_elevation();
  const auto triangles = TestMeshData::create_triangles();
  const auto boundaries = TestMeshData::empty_boundaries();

  // Construct HostMesh (held by unique_ptr), then wrap in Mesh
  auto host_mesh = std::make_unique<Geometry::HostMesh>(
      x_coords, y_coords, elevation, triangles, boundaries,
      TestMeshData::origin, TestMeshData::projection_type);
  const Cocoa::Mesh mesh(std::move(host_mesh));
}

Catch2 Assertions

Common assertions used in Cocoa tests:

Basic Assertions:

REQUIRE(expr);           // Fatal assertion
CHECK(expr);             // Non-fatal assertion
REQUIRE_FALSE(expr);     // Require expression is false
REQUIRE_NOTHROW(expr);   // Require no exception thrown
REQUIRE_THROWS_AS(expr, ExceptionType);  // Require specific exception

Floating-Point Comparisons:

#include <catch2/catch_approx.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>

using Catch::Approx;
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;

REQUIRE(value == Approx(expected));              // Default tolerance
REQUIRE(value == Approx(expected).epsilon(1e-6)); // Custom tolerance
REQUIRE_THAT(value, WithinAbs(expected, 1e-10)); // Absolute tolerance
REQUIRE_THAT(value, WithinRel(expected, 1e-6)); // Relative tolerance

Writing New Tests

Create or edit a test file in test/:

#include <catch2/catch_test_macros.hpp>
#include "cocoa_kernel/geometry/Mesh.hpp"

TEST_CASE("ComponentName", "[component][tag]") {
  SECTION("Descriptive behavior name") {
    // Arrange
    auto obj = create_object();

    // Act
    auto result = obj.method();

    // Assert
    REQUIRE(result == expected);
  }
}

Add the file to test/CMakeLists.txt if creating a new file:

set(COCOA_TEST_SOURCES
    test_main.cpp
    # ... existing files ...
    test_your_new_file.cpp)

Rebuild and run:

cmake --build --preset dev --target cocoa_tests
./build/test/cocoa_tests "[your_tag]"

Compile-Time Tests

For constexpr functionality, use static_assert:

// Compile-time tests for constexpr functions
static_assert(TimeDelta::fromDays(1).totalHours() == 24,
              "Day to hours conversion failed");
static_assert(TimeDelta::fromHours(24) == TimeDelta::fromDays(1),
              "Equality comparison failed");

Integration Tests

In addition to unit tests, Cocoa includes end-to-end integration tests that run a complete simulation on the Western North Atlantic (WNAT) mesh and compare results against known-good reference solutions. These tests verify that the full simulation pipeline (mesh loading, tidal forcing with SAL, time stepping, output) reproduces committed reference results within a per-test tolerance.

Output is always delivered on a background writer thread, so every run exercises the asynchronous write path; there is no I/O mode to select.

Integration tests span several categories: WNAT tides (explicit and implicit GWCE, serial and MPI), the GAHM vortex-only and composed-with-gridded-met cases, the global tidal case (serial, split-output, and MPI), the internal weir case, and checkpoint/restart round-trips. Each test runs in a private working directory (see Integration Working Directories below). The authoritative, current list of integration tests – their names, rank counts, and configs – is test/CMakeLists.txt; it is not duplicated here because that duplication drifts as tests are added. Search it for NAME Integration_ to enumerate them, or run ctest -N -L integration.

The WNAT tide tests run a 1-day simulation (M2 boundary + tide potential + SAL) and compare the output against a serial reference solution: the explicit case at \(\Delta t\) = 20 s, with CF meteorological forcing also read, and the implicit case at \(\Delta t\) = 60 s. The MPI variants use a separate reference because mesh partitioning changes floating-point operation order.

Integration test tolerances are NOT uniform: they range from 1e-3 to 1e-12 depending on what the comparison must absorb. Runs that must agree with no arithmetic difference use the tightest tolerances (1e-12 for two configurations that must produce the same run, 1e-10 for checkpoint round-trips with no intervening solver iteration); 1e-6 covers most serial simulation comparisons. MPI runs are relaxed to 1e-3 because partitioning changes the order of floating-point reductions; checkpoint restart through the iterative consistent-GWCE solver is relaxed to 1e-5 because the CG solver’s own convergence tolerance is 1e-5. Each -DTOLERANCE= value in test/CMakeLists.txt carries a comment explaining why that value and not a tighter one – treat those comments, not this page, as the authority; do not tighten or loosen a tolerance without updating its comment to match.

The comparison script also verifies that wet/dry state (NaN locations) matches between test and reference at every timestep.

After comparison, a Cartopy-based plotting script generates a visual summary (wnat_comparison_serial.png, wnat_comparison_mpi.png) with peak water level and velocity maps, difference maps, and time series at probe points. Plots are kept in build/test/integration_plots; in CI, they are uploaded as a GitHub Actions artifact.

Prerequisites:

Integration tests require Python 3 with numpy, xarray and netCDF4; cartopy is needed only by the comparison-plot script. If an import fails, CMake reports Integration tests disabled at configure time and skips them.

CMake options:

Integration tests are off by default and must be explicitly enabled:

# Enable integration tests
cmake --preset dev -Dcocoa_ENABLE_INTEGRATION_TESTS=ON

The dev-maintainer, ci and ci-coverage presets already enable them.

In CI, integration tests are enabled for the serial sanitizer job (build-and-test) and the coverage job. The GPU compile-check job (gpu-build) does not run tests.

Running integration tests selectively:

# Run only integration tests
ctest -R Integration

# Run only the serial integration test
ctest -R Integration_WNAT_tides$

# Run only the MPI integration test
ctest -R Integration_WNAT_tides_mpi

# Exclude integration tests (run unit tests only)
ctest -E Integration

Regenerating reference solutions:

If numerical methods change and produce different (but correct) results, the reference solutions must be regenerated with test/regenerate_references.sh – do not hand-run cocoa and mv the output; the script is the single source of truth for which config produces which reference file, and it enforces the conditions those references depend on:

# Regenerate everything (wnat, global, weir, gahm, baroclinic, channel)
test/regenerate_references.sh --cocoa build/src/cocoa/cocoa

# Regenerate one case only
test/regenerate_references.sh --cocoa build/src/cocoa/cocoa wnat

Run it inside the build container (or an environment with the same compiler, Trilinos, and Kokkos as CI) so the regenerated references are reproducible: every run is single-threaded (OMP_NUM_THREADS=1), so the bytes it produces are bit-reproducible run to run on the same toolchain. The script clears any stale cocoa_output.nc before each run so a failed run cannot silently leave a misleading reference, and it discards the transient cocoa_output_peak.nc file rather than storing it. It regenerates every reference listed in its header comment (the WNAT, GAHM, moving-ice, baroclinic, global, internal-weir and channel references, with 2-rank MPI references where a case has one); the split-output variants and the weir MPI variant compare against the serial reference rather than carrying their own, so those are not separately regenerated.

Commit the regenerated .nc files with provenance in the commit body: the commit and config that produced them, and why (what numerical change made the old references stale). Regenerate references deliberately, never as a drive-by fix for an unrelated failing test – a reference that moves without a documented reason erases the test’s ability to catch a real regression.

Working directories:

Each integration test runs in its own copy of test/data, build/test/integration/<test name>, with its case at <test name>/<case>, so tests never share output files and ctest -j runs them side by side. The copy leaves out the files a run writes (cocoa_output*, cocoa_checkpoint*, plots); a fixture’s ../channel/... paths resolve inside it, and nothing a run writes reaches the source tree. A passing test deletes its directory and keeps its plots in build/test/integration_plots; a failing one leaves the directory for inspection. Register new end-to-end tests with cocoa_add_integration_test (cmake/CocoaIntegrationTests.cmake); a file a fixture test generates reaches the consumer through its EXTRA_INPUTS.

Python Tools Tests

The cocoa-tools package (utils/cocoa_tools) has its own pytest suite, which is not part of ctest. Install the package and run the suite from its directory:

pip install ./utils/cocoa_tools pytest
cd utils/cocoa_tools
python -m pytest -q

CI runs the same suite in the cocoa_tools job. The mesh-tools tests read fixtures from test/data.

Debugging Failed Tests

# Run with verbose output
./test/cocoa_tests -s "[failing_tag]"

# Run under debugger
lldb ./test/cocoa_tests -- "[failing_tag]"

# Run specific named test
./test/cocoa_tests "Exact Test Name"

# Show test durations
./test/cocoa_tests -d yes