======= Testing ======= Cocoa uses Catch2 for unit and integration testing, with Kokkos/Trilinos initialization handled in a custom main entry point. Test Framework -------------- Cocoa uses `Catch2 `_ (v3) as its testing framework. Tests are organized using ``TEST_CASE`` and ``SECTION`` macros: .. code-block:: cpp #include TEST_CASE("Mesh", "[mesh]") { // Setup code runs for each section const auto mesh = create_test_mesh(); SECTION("Valid mesh construction") { REQUIRE(mesh.num_nodes() == 9); REQUIRE(mesh.num_elements() == 8); } SECTION("Coordinate access") { const auto& nodes = mesh.nodes(); REQUIRE_THAT(nodes[0].location().x(), WithinAbs(-88.0, 1e-10)); } } The test executable uses a custom ``main()`` to initialize Tpetra/Kokkos before running tests: .. code-block:: cpp #define CATCH_CONFIG_RUNNER #include #include "Tpetra_Core.hpp" int main(int argc, char* argv[]) { Tpetra::ScopeGuard tpetra_scope_guard(&argc, &argv); return Catch::Session().run(argc, argv); } Building and Running Tests -------------------------- .. code-block:: bash # Build with tests (enabled by default) mkdir build && cd build cmake .. make -j8 # Run all tests via CTest ctest --output-on-failure # Run the test executable directly ./test/cocoa_tests # Run specific test cases by tag ./test/cocoa_tests "[mesh]" ./test/cocoa_tests "[gwce]" # Run specific test case by name ./test/cocoa_tests "Mesh" # List all available tests ./test/cocoa_tests --list-tests Test Files ---------- Test files are located in the ``test/`` directory: .. list-table:: :header-rows: 1 :widths: 35 65 * - File - Description * - ``test_main.cpp`` - Entry point with Tpetra/Kokkos initialization * - ``test_mesh.cpp`` - Mesh construction and access * - ``test_neighbor_table.cpp`` - Node connectivity and neighbor tables * - ``test_point2d.cpp`` - 2D point and geometry operations * - ``test_projection.cpp`` - Coordinate projection (CPP) * - ``test_datetime.cpp`` - DateTime and TimeDelta classes * - ``test_timestepper.cpp`` - Time stepping iteration * - ``test_temporal_field.cpp`` - Multi-level temporal field storage * - ``test_gwce_consistent.cpp`` - GWCE consistent (implicit) solver and matrix assembly * - ``test_gwce_lumped.cpp`` - GWCE lumped (explicit) solver * - ``test_wetdry.cpp`` - Wetting and drying algorithm * - ``test_tide_potential.cpp`` - Tidal potential forcing * - ``test_tidal_boundary.cpp`` - Tidal boundary conditions * - ``test_momentum.cpp`` - Momentum solver * - ``test_bottom_friction.cpp`` - Bottom friction computation * - ``test_coriolis.cpp`` - Coriolis force computation * - ``test_advection.cpp`` - Advection terms * - ``test_pressure_gradient.cpp`` - Pressure gradient computation * - ``test_inline_lateral_stress.cpp`` - Inline lateral stress computation * - ``test_smagorinsky.cpp`` - Smagorinsky turbulence model * - ``test_basis_functions.cpp`` - Finite element basis functions * - ``test_coordinate_rotation.cpp`` - Coordinate rotation for global meshes * - ``test_ramp_function.cpp`` - Forcing ramp functions * - ``test_configuration_reader.cpp`` - YAML configuration reader * - ``test_netcdf_io.cpp`` - NetCDF I/O operations * - ``test_logger.cpp`` - Logging system * - ``test_tide_astronomic.cpp`` - Astronomical tide potential * - ``test_forcing_manager.cpp`` - Forcing manager orchestration * - ``test_boundary_type.cpp`` - Boundary type classification * - ``test_flow_boundary.cpp`` - Flow boundary conditions (:math:`Q_{\text{force}}` kernels, time series, segment integration) * - ``test_schema_engine.cpp`` - JSON Schema validation engine (valijson glue: paths, typo suggestions, weak-type semantics) * - ``test_schema_validation.cpp`` - Configuration schema validation against the embedded schema document * - ``test_distributed.cpp`` - Distributed computing (serial mode) * - ``test_distributed_mpi.cpp`` - MPI distributed computing (multi-rank) Test Data --------- Test data and helper utilities are organized as follows: .. code-block:: text test/ ├── TestMeshData.hpp # Test mesh generation utilities ├── TestPrecision.hpp # Mixed-precision test tolerance helpers ├── test_*.cpp # Test source files ├── run_integration_test.cmake # Integration test runner script └── data/ ├── test_config.yaml # ConfigurationReader test fixture └── wnat/ # WNAT integration test data ├── wnat.nc # Mesh file ├── cocoa_config_integration_test.yaml # Tides-only config ├── compare_tides_reference.py # Reference comparison script ├── plot_comparison.py # Comparison plot generator ├── wnat_reference_fp64.nc # Serial reference └── wnat_reference_mpi_fp64.nc # MPI reference Runnable example cases live in the `cocoa-examples `_ repository; the fixtures above are separate copies maintained with the tests. See :doc:`../user_guide/test_cases` for detailed descriptions of each test case, including expected results and verification plots. The ``TestMeshData.hpp`` helper provides functions to create small test meshes: .. code-block:: cpp #include "TestMeshData.hpp" using TestData::TestMeshData; TEST_CASE("Example", "[example]") { const auto [x_coords, y_coords] = TestMeshData::create_coordinates(); const auto elevation = TestMeshData::create_elevation(); const auto triangles = TestMeshData::create_triangles(); const auto boundaries = TestMeshData::empty_boundaries(); // Construct HostMesh (held by unique_ptr), then wrap in Mesh auto host_mesh = std::make_unique( x_coords, y_coords, elevation, triangles, boundaries, TestMeshData::origin, TestMeshData::projection_type); const Cocoa::Mesh mesh(std::move(host_mesh)); } Catch2 Assertions ----------------- Common assertions used in Cocoa tests: **Basic Assertions:** .. code-block:: cpp REQUIRE(expr); // Fatal assertion CHECK(expr); // Non-fatal assertion REQUIRE_FALSE(expr); // Require expression is false REQUIRE_NOTHROW(expr); // Require no exception thrown REQUIRE_THROWS_AS(expr, ExceptionType); // Require specific exception **Floating-Point Comparisons:** .. code-block:: cpp #include #include using Catch::Approx; using Catch::Matchers::WithinAbs; using Catch::Matchers::WithinRel; REQUIRE(value == Approx(expected)); // Default tolerance REQUIRE(value == Approx(expected).epsilon(1e-6)); // Custom tolerance REQUIRE_THAT(value, WithinAbs(expected, 1e-10)); // Absolute tolerance REQUIRE_THAT(value, WithinRel(expected, 1e-6)); // Relative tolerance **Sections for Test Organization:** .. code-block:: cpp TEST_CASE("Component", "[tag]") { // Setup code runs before each section auto component = create_component(); SECTION("First behavior") { // Test first behavior REQUIRE(component.first_method() == expected); } SECTION("Second behavior") { // Test second behavior (fresh component) REQUIRE(component.second_method() == expected); } } Writing New Tests ----------------- 1. Create or edit a test file in ``test/``: .. code-block:: cpp #include #include "your_header.hpp" TEST_CASE("ComponentName", "[component][tag]") { SECTION("Descriptive behavior name") { // Arrange auto obj = create_object(); // Act auto result = obj.method(); // Assert REQUIRE(result == expected); } } 2. Add the file to ``test/CMakeLists.txt`` if creating a new file: .. code-block:: cmake set(COCOA_TEST_SOURCES test_main.cpp # ... existing files ... test_your_new_file.cpp) 3. Rebuild and run: .. code-block:: bash make cocoa_tests ./test/cocoa_tests "[your_tag]" Compile-Time Tests ------------------ For ``constexpr`` functionality, use ``static_assert``: .. code-block:: cpp // Compile-time tests for constexpr functions static_assert(TimeDelta::fromDays(1).totalHours() == 24, "Day to hours conversion failed"); static_assert(TimeDelta::fromHours(24) == TimeDelta::fromDays(1), "Equality comparison failed"); Integration Tests ----------------- In addition to unit tests, Cocoa includes end-to-end integration tests that run a complete simulation on the Western North Atlantic (WNAT) mesh and compare results against known-good reference solutions. These tests verify that the full simulation pipeline (mesh loading, tidal forcing with SAL, time stepping, output) produces bitwise-reproducible results. Output is always delivered on a background writer thread, so every run exercises the asynchronous write path; there is no I/O mode to select. Integration tests span several categories: WNAT tides (explicit and implicit GWCE, serial and MPI), the GAHM vortex-only and composed-with-gridded-met cases, the global tidal case (serial, split-output, and MPI), the internal weir case, and checkpoint/restart round-trips. Each test carries a ``RESOURCE_LOCK`` where it shares a working directory with siblings (see `Shared Working Directories`_ below). The authoritative, current list of integration tests -- their names, rank counts, and configs -- is ``test/CMakeLists.txt``; it is not duplicated here because that duplication drifts as tests are added. Search it for ``add_test(NAME Integration_`` to enumerate them. Each WNAT test runs a 1-day tides-only simulation (M2 boundary + tide potential + SAL) with :math:`\Delta t` = 20 s and compares the output against a serial reference solution. The MPI variants use a separate reference because mesh partitioning changes floating-point operation order. Integration test tolerances are NOT uniform: they range from ``1e-3`` to ``1e-10`` depending on what the comparison must absorb. Serial runs against a same-arithmetic serial reference use the tightest tolerance (``1e-10`` for checkpoint round-trips with no intervening solver iteration; ``1e-6`` for most serial simulation comparisons). MPI runs are relaxed to ``1e-3`` because partitioning changes the order of floating-point reductions; checkpoint restart through the iterative consistent-GWCE solver is relaxed to ``1e-5`` because the CG solver's own convergence tolerance is ``1e-5``. Each ``-DTOLERANCE=`` value in ``test/CMakeLists.txt`` carries a comment explaining why that value and not a tighter one -- treat those comments, not this page, as the authority; do not tighten or loosen a tolerance without updating its comment to match. The comparison script also verifies that wet/dry state (NaN locations) matches between test and reference at every timestep. After comparison, a Cartopy-based plotting script generates a visual summary (``wnat_comparison.png``) with peak water level and velocity maps, difference maps, and time series at probe points. In CI, this plot is uploaded as a GitHub Actions artifact. **Prerequisites:** Integration tests require Python 3 with ``numpy``, ``xarray``, ``netCDF4``, and ``cartopy``. If these modules are not available, the integration tests are silently disabled at CMake configure time. **CMake options:** Integration tests are **off by default** and must be explicitly enabled: .. code-block:: bash # Enable integration tests cmake .. -Dcocoa_ENABLE_INTEGRATION_TESTS=ON In CI, integration tests are enabled for the serial build-and-test jobs and the coverage job. GPU compile-check jobs do not run tests. **Running integration tests selectively:** .. code-block:: bash # Run only integration tests ctest -R Integration # Run only the serial integration test ctest -R Integration_WNAT_tides$ # Run only the MPI integration test ctest -R Integration_WNAT_tides_mpi # Exclude integration tests (run unit tests only) ctest -E Integration **Regenerating reference solutions:** If numerical methods change and produce different (but correct) results, the reference solutions must be regenerated with ``test/regenerate_references.sh`` -- do not hand-run ``cocoa`` and ``mv`` the output; the script is the single source of truth for which config produces which reference file, and it enforces the conditions those references depend on: .. code-block:: bash # Regenerate everything (wnat, global, weir, gahm) test/regenerate_references.sh --cocoa build/src/cocoa/cocoa # Regenerate one case only test/regenerate_references.sh --cocoa build/src/cocoa/cocoa wnat Run it inside the build container (or an environment with the same compiler, Trilinos, and Kokkos as CI) so the regenerated references are reproducible: every run is single-threaded (``OMP_NUM_THREADS=1``), so the bytes it produces are bit-reproducible run to run on the same toolchain. The script clears any stale ``cocoa_output.nc`` before each run so a failed run cannot silently leave a misleading reference, and it discards the transient ``cocoa_output_peak.nc`` file rather than storing it. It regenerates the serial WNAT, MPI WNAT, implicit WNAT (serial and MPI), GAHM vortex-only, global, and internal-weir references; split-output and MPI variants of the global and weir cases compare against the serial reference rather than carrying their own, so those are not separately regenerated. Commit the regenerated ``.nc`` files with provenance in the commit body: the commit and config that produced them, and why (what numerical change made the old references stale). Regenerate references deliberately, never as a drive-by fix for an unrelated failing test -- a reference that moves without a documented reason erases the test's ability to catch a real regression. .. _Shared Working Directories: **Shared working directories:** Integration tests run serially (label ``integration``) because several of them share a working directory and would otherwise race on the same input and output files. Every test that runs against ``test/data/wnat`` carries ``set_tests_properties(... PROPERTIES RESOURCE_LOCK wnat_workdir)`` -- currently nine tests, spanning the WNAT tides, implicit-tides, split-output, failstop, and GAHM (vortex-only, composed, composed-domains) cases; search ``test/CMakeLists.txt`` for ``RESOURCE_LOCK wnat_workdir`` for the exact, current set. A new test that reads or writes into ``test/data/wnat`` must take the same lock, or it can run concurrently with an existing WNAT test and corrupt its output. Tests rooted in their own directory (``global_case``, ``internal_weir``, and the checkpoint tests under those directories) do not share a workdir with any other case and so carry no ``RESOURCE_LOCK`` -- only add one when a new test genuinely reads or writes files another test also touches. Debugging Failed Tests ---------------------- .. code-block:: bash # Run with verbose output ./test/cocoa_tests -s "[failing_tag]" # Run under debugger lldb ./test/cocoa_tests -- "[failing_tag]" # Run specific named test ./test/cocoa_tests "Exact Test Name" # Show test durations ./test/cocoa_tests -d yes