======= Testing ======= Cocoa uses Catch2 for the unit tests, with Kokkos/Trilinos initialization handled in a custom main entry point. The integration tests are ctest entries that run the ``cocoa`` binary and compare its output against a committed reference with a Python script. Test Framework -------------- Tests are written with `Catch2 `_ (v3) ``TEST_CASE`` and ``SECTION`` macros: .. code-block:: cpp #include TEST_CASE("Mesh", "[mesh]") { // Setup code runs for each section const auto mesh = create_test_mesh(); SECTION("Valid mesh construction") { REQUIRE(mesh.num_nodes() == 9); REQUIRE(mesh.num_elements() == 8); } SECTION("Coordinate access") { const auto& nodes = mesh.nodes(); REQUIRE_THAT(nodes[0].location().x(), WithinAbs(-88.0, 1e-10)); } } The test executable uses a custom ``main()`` to initialize Tpetra/Kokkos before running tests: .. code-block:: cpp #define CATCH_CONFIG_RUNNER #include #include "Tpetra_Core.hpp" int main(int argc, char* argv[]) { Tpetra::ScopeGuard tpetra_scope_guard(&argc, &argv); return Catch::Session().run(argc, argv); } Building and Running Tests -------------------------- .. code-block:: bash # Build with tests (enabled by default) cmake --preset dev cmake --build --preset dev # Run all tests via CTest, from anywhere in the source tree ctest --preset dev # Run the test executable directly (paths below are relative to build/) cd build ./test/cocoa_tests # Run specific test cases by tag ./test/cocoa_tests "[mesh]" ./test/cocoa_tests "[gwce]" # Run specific test case by name ./test/cocoa_tests "Mesh" # List all available tests ./test/cocoa_tests --list-tests Test Files ---------- Test sources live in ``test/`` as ``test_.cpp``, one file per component, with ``test_main.cpp`` as the entry point. The current set is ``COCOA_TEST_SOURCES`` in ``test/CMakeLists.txt``. ``./test/cocoa_tests --list-tests`` enumerates the test cases and ``--list-tags`` the tags they are filtered by. Test Data --------- Test data and helper utilities are organized as follows: .. code-block:: text test/ ├── TestMeshData.hpp # Test mesh generation utilities ├── TestPrecision.hpp # Mixed-precision test tolerance helpers ├── test_*.cpp # Test source files ├── run_in_workdir.cmake # Runs a driver in a private fixture copy ├── run_integration_test.cmake # Default integration test driver ├── run_*_test.cmake # Other drivers (checkpoint, serial-vs-MPI, ...) ├── regenerate_references.sh # Reference regeneration (see below) └── data/ ├── test_config.yaml # ConfigurationReader test fixture ├── baroclinic/ # Baroclinic forcing fixtures ├── channel/ # Channel and beach cases (+ validation) ├── flow_channel/ # Flow-boundary case ├── global_case/ # Global tidal case ├── ice_drag/ # Ice drag golden values ├── internal_tide_checkpoint/ # Internal tide checkpoint case ├── internal_weir/ # Internal weir case ├── meteo/ # Per-format meteorological reader fixtures ├── partition/ # Partition cache fixtures ├── vortex/ # ATCF tracks and GAHM control solutions └── wnat/ # WNAT integration test data ├── wnat.nc # Mesh file ├── cocoa_config_*.yaml # One config per case ├── compare_tides_reference.py # Reference comparison script ├── plot_comparison.py # Comparison plot generator └── *_reference{,_mpi}.nc # Serial and MPI references Runnable example cases live in the `cocoa-examples `_ repository; the fixtures above are separate copies maintained with the tests. See :doc:`../user_guide/test_cases` for detailed descriptions of each test case, including expected results and verification plots. Two directories hold unit-test fixtures rather than integration ones: - ``ice_drag/lupkes_golden.csv``: ``generate_lupkes_golden.py`` computes the Lupkes drag blend in Python from the paper formulas, and ``test_ice_drag.cpp`` holds the C++ implementation to it at 1e-12. - ``baroclinic/``: a 25-node ``tiny_mesh.nc`` and a 3-snapshot ``nodal_forcing.nc`` with analytic values (``make_test_fixture.py``), malformed variants of that file for the reader's error paths, and ADCIRC-derived control tables for the internal-tide drag and dispersion-friction unit tests under ``reference/``. The ``TestMeshData.hpp`` helper provides functions to create small test meshes: .. code-block:: cpp #include "TestMeshData.hpp" using TestData::TestMeshData; TEST_CASE("Example", "[example]") { const auto [x_coords, y_coords] = TestMeshData::create_coordinates(); const auto elevation = TestMeshData::create_elevation(); const auto triangles = TestMeshData::create_triangles(); const auto boundaries = TestMeshData::empty_boundaries(); // Construct HostMesh (held by unique_ptr), then wrap in Mesh auto host_mesh = std::make_unique( x_coords, y_coords, elevation, triangles, boundaries, TestMeshData::origin, TestMeshData::projection_type); const Cocoa::Mesh mesh(std::move(host_mesh)); } Catch2 Assertions ----------------- Common assertions used in Cocoa tests: **Basic Assertions:** .. code-block:: cpp REQUIRE(expr); // Fatal assertion CHECK(expr); // Non-fatal assertion REQUIRE_FALSE(expr); // Require expression is false REQUIRE_NOTHROW(expr); // Require no exception thrown REQUIRE_THROWS_AS(expr, ExceptionType); // Require specific exception **Floating-Point Comparisons:** .. code-block:: cpp #include #include using Catch::Approx; using Catch::Matchers::WithinAbs; using Catch::Matchers::WithinRel; REQUIRE(value == Approx(expected)); // Default tolerance REQUIRE(value == Approx(expected).epsilon(1e-6)); // Custom tolerance REQUIRE_THAT(value, WithinAbs(expected, 1e-10)); // Absolute tolerance REQUIRE_THAT(value, WithinRel(expected, 1e-6)); // Relative tolerance Writing New Tests ----------------- 1. Create or edit a test file in ``test/``: .. code-block:: cpp #include #include "cocoa_kernel/geometry/Mesh.hpp" TEST_CASE("ComponentName", "[component][tag]") { SECTION("Descriptive behavior name") { // Arrange auto obj = create_object(); // Act auto result = obj.method(); // Assert REQUIRE(result == expected); } } 2. Add the file to ``test/CMakeLists.txt`` if creating a new file: .. code-block:: cmake set(COCOA_TEST_SOURCES test_main.cpp # ... existing files ... test_your_new_file.cpp) 3. Rebuild and run: .. code-block:: bash cmake --build --preset dev --target cocoa_tests ./build/test/cocoa_tests "[your_tag]" Compile-Time Tests ------------------ For ``constexpr`` functionality, use ``static_assert``: .. code-block:: cpp // Compile-time tests for constexpr functions static_assert(TimeDelta::fromDays(1).totalHours() == 24, "Day to hours conversion failed"); static_assert(TimeDelta::fromHours(24) == TimeDelta::fromDays(1), "Equality comparison failed"); Integration Tests ----------------- The integration tests run complete simulations, most on the Western North Atlantic (WNAT) mesh, and check that the full pipeline (mesh loading, tidal forcing with SAL, time stepping, output) reproduces committed reference results within a per-test tolerance. Output always goes through the background writer thread, so every run exercises the asynchronous write path. Integration tests span several categories: WNAT tides (explicit and implicit GWCE, serial and MPI), the GAHM vortex-only and composed-with-gridded-met cases, the global tidal case (serial, split-output, and MPI), the internal weir case, and checkpoint/restart round-trips. Each test runs in a private working directory (see `Integration Working Directories`_ below). The list of integration tests, with their names, rank counts, and configs, is ``test/CMakeLists.txt``. Search it for ``NAME Integration_`` to enumerate them, or run ``ctest -N -L integration``. The WNAT tide tests run a 1-day simulation (M2 boundary + tide potential + SAL) and compare the output against a serial reference solution: the explicit case at :math:`\Delta t` = 20 s, with CF meteorological forcing also read, and the implicit case at :math:`\Delta t` = 60 s. The MPI variants use a separate reference because mesh partitioning changes floating-point operation order. Integration test tolerances are NOT uniform: they range from ``1e-3`` to ``1e-12`` depending on what the comparison must absorb. Runs that must agree with no arithmetic difference use the tightest tolerances (``1e-12`` for two configurations that must produce the same run, ``1e-10`` for checkpoint round-trips with no intervening solver iteration); ``1e-6`` covers most serial simulation comparisons. MPI runs are relaxed to ``1e-3`` because partitioning changes the order of floating-point reductions; checkpoint restart through the iterative consistent-GWCE solver is relaxed to ``1e-5`` because the CG solver's own convergence tolerance is ``1e-5``. Each ``-DTOLERANCE=`` value in ``test/CMakeLists.txt`` carries a comment explaining why a tighter value would fail. Those comments take precedence over this page; update a tolerance's comment whenever you change the tolerance. The comparison script also verifies that wet/dry state (NaN locations) matches between test and reference at every timestep. The comparison scripts (``compare_bitwise.py``, ``compare_tides_reference.py``) live in ``test/data/wnat/`` and are shared across test areas. The fixtures under ``test/data/`` pin ``physics.wetting_velocity: inertial``, the form Cocoa used for every run before 0.6.0, so their committed references carry that wetting rule; the default is ``linear``. After comparison, a Cartopy-based plotting script generates a visual summary (``wnat_comparison_serial.png``, ``wnat_comparison_mpi.png``) with peak water level and velocity maps, difference maps, and time series at probe points. Plots are kept in ``build/test/integration_plots``; in CI, they are uploaded as a GitHub Actions artifact. **Prerequisites:** Integration tests require Python 3 with ``numpy``, ``xarray`` and ``netCDF4``; ``cartopy`` is needed only by the comparison-plot script. If an import fails, CMake reports ``Integration tests disabled`` at configure time and skips them. **CMake options:** Integration tests are off by default: .. code-block:: bash # Enable integration tests cmake --preset dev -Dcocoa_ENABLE_INTEGRATION_TESTS=ON The ``dev-maintainer``, ``ci`` and ``ci-coverage`` presets already enable them. In CI, integration tests are enabled for the serial sanitizer job (``build-and-test``) and the coverage job. The GPU compile-check job (``gpu-build``) does not run tests. **Running integration tests selectively:** .. code-block:: bash # Run only integration tests ctest -R Integration # Run only the serial integration test ctest -R Integration_WNAT_tides$ # Run only the MPI integration test ctest -R Integration_WNAT_tides_mpi # Exclude integration tests (run unit tests only) ctest -E Integration **Regenerating reference solutions:** If numerical methods change and produce different (but correct) results, the reference solutions must be regenerated with ``test/regenerate_references.sh``. Do not hand-run ``cocoa`` and ``mv`` the output: the script defines which config produces which reference file, and it enforces the conditions those references depend on: .. code-block:: bash # Regenerate everything (wnat, global, weir, gahm, baroclinic, channel) test/regenerate_references.sh --cocoa build/src/cocoa/cocoa # Regenerate one case only test/regenerate_references.sh --cocoa build/src/cocoa/cocoa wnat Run it inside the build container (or an environment with the same compiler, Trilinos, and Kokkos as CI) so the regenerated references are reproducible: every run is single-threaded (``OMP_NUM_THREADS=1``), so the bytes it produces are bit-reproducible run to run on the same toolchain. The script clears any stale ``cocoa_output.nc`` before each run so a failed run cannot leave a misleading reference, and it discards the transient ``cocoa_output_peak.nc`` file rather than storing it. It regenerates every reference listed in its header comment (the WNAT, GAHM, moving-ice, baroclinic, global, internal-weir and channel references, with 2-rank MPI references where a case has one); the split-output variants and the weir MPI variant compare against the serial reference rather than carrying their own, so those are not separately regenerated. Commit the regenerated ``.nc`` files with provenance in the commit body: the commit and config that produced them, and why (what numerical change made the old references stale). Never regenerate references to fix an unrelated failing test: a reference that moves without a documented reason can no longer catch a real regression. .. _Integration Working Directories: **Working directories:** Each integration test runs in its own copy of ``test/data``, ``build/test/integration/``, with its case at ``/``, so tests never share output files and ``ctest -j`` runs them side by side. The copy leaves out the files a run writes (``cocoa_output*``, ``cocoa_checkpoint*``, plots); a fixture's ``../channel/...`` paths resolve inside it, and nothing a run writes reaches the source tree. A passing test deletes its directory and keeps its plots in ``build/test/integration_plots``; a failing one leaves the directory for inspection. Register new end-to-end tests with ``cocoa_add_integration_test`` (``cmake/CocoaIntegrationTests.cmake``); a file a fixture test generates reaches the consumer through its ``EXTRA_INPUTS``. .. _integration-test-fixtures: Integration Test Fixtures ------------------------- The tests below run on the in-repository cases described in :doc:`../user_guide/test_cases`. Entries with an ``_mpi`` suffix are registered only when MPI tests are enabled. Ideal channel ~~~~~~~~~~~~~ Fixtures in ``test/data/channel/`` (:ref:`channel-quad-validation-test`): - ``Integration_channel_straight_tide``, ``Integration_channel_meander_flow`` and their ``_implicit`` variants: the coarse hybrid meshes against committed references at 1e-6 on ``zeta``, ``ucx`` and ``ucy``. Each has an ``_mpi`` variant that compares a 2-rank run against a 2-rank reference at 1e-3. - ``Integration_channel_straight_flow``: the uniform-flow smoke case (completes, finite, bounded, hybrid encoding checked), lumped only. - ``Integration_hybrid_tide_lumped`` and ``Integration_hybrid_tide_consistent``: an M2 smoke run per mass matrix on ``channel_hybrid_tide.nc``; ``Integration_hybrid_tide_mpi_lumped`` and ``_consistent`` hold a 2-rank run to the serial one. - ``Integration_beach_cut_wetdry_mpi``: the coarse triangle beach on 2 ranks through a committed partition cache (``channel_beach_coarse_triangle_tide.nc.partition_2.nc``, written by ``make_beach_cut_partition.py``) whose cut the tidal front crosses. Every output's wet/dry set must equal the serial run's. - ``Integration_broad_inlet_flow_mpi2``/``3``/``4`` and ``Integration_two_inlets_flow_mpi2``/``3``/``4``: wide specified-flow inlets the partitioner has to cut, held to the serial run on the fields and on the cross-section discharge. - ``Integration_surplus_tide_values_rejected`` and its ``_mpi`` variant: a tidal ``boundary_values`` list one entry longer than the open boundary must be rejected the same way at every rank count. The hybrid tests check the output ``element`` encoding (``nvertex`` of 4, -1 in each triangle row's fourth column) with ``check_hybrid_element_encoding.py``. Two ``Fixture_channel_*`` entries check the fixtures themselves: a ``cocoa-plot`` render of the hybrid strip's mesh, and an exact-match check of the committed meshes against the generator. To regenerate the meshes and run that check by hand: .. code-block:: bash cd test/data/channel python make_channel_meshes.py . python make_channel_meshes.py --check-committed . Internal weir ~~~~~~~~~~~~~ Fixtures in ``test/data/internal_weir/`` (:ref:`internal-weir-test`): - ``Integration_internal_weir``: compares ``zeta`` at every output time against ``weir_reference.nc`` at 1e-6; the wet/dry pattern must match. ``Integration_internal_weir_mpi`` holds a 2-rank run to the same reference at 1e-3. - ``Integration_weir_checkpoint``: runs the full day, then a cold start to 12:00 that writes a checkpoint, then a restart from it to the end. The restart's ``zeta`` must match the full run at shared output times to 1e-10. ``Integration_weir_checkpoint_mpi`` runs it on 2 ranks at 1e-3. - ``Integration_internal_tide_checkpoint_mid_window`` (in ``test/data/internal_tide_checkpoint/``): the same checkpoint test with internal-tide wave drag active (:doc:`../user_guide/baroclinic_coupling`). The mesh ``internal_weir_it.nc`` is the weir mesh plus a 3-column (Lyard) ``internal_tide_friction`` attribute. The forcing file ``internal_tide_baroclinic_forcing.nc`` is a single frozen snapshot with ``n_bottom`` = 0.02, ``n_mean`` = 0.01 and zero BPG. The cold start ends at 11:55:00 (42900 s), which is not a multiple of the 720 s de-tiding sampling window, so the restart must resume the de-tiding ring mid-window. Its configs add ``physics.internal_tide_friction: mesh`` and a ``forcing.baroclinic`` block with ``internal_tide.enabled: true``. Tolerance 1e-10; the ``_mpi`` variant runs on 2 ranks at 1e-3. GAHM vortex ~~~~~~~~~~~ Fixtures in ``test/data/wnat/`` and ``test/data/vortex/`` (:ref:`gahm-vortex-test`). Each test compares ``zeta`` against a committed reference at 1e-6 with ``compare_tides_reference.py``. - ``Integration_gahm_vortex_only``: the vortex alone, with no blend ring, against ``gahm_vortex_only_reference.nc``. - ``Integration_gahm_composed``: the vortex plus the out-of-window gridded reader, against ``gahm_composed_reference.nc``. It guards that the vortex does not vanish outside the reader's time window. - ``Integration_gahm_composed_domains``: the composed config with the gridded reader moved into a one-entry ``domains:`` list. It runs against the same reference as the flat form, so a ``domains:`` singleton must be indistinguishable from the flat keys. Sea ice ~~~~~~~ Fixtures in ``test/data/wnat/`` (:ref:`sea-ice-test`): - ``Integration_wnat_zero_ice``: runs the no-ice config and the zero-ice config with the same binary. The primary check requires the two outputs to be identical (tolerance 0.0). A secondary check compares the zero-ice run against ``wnat_reference.nc`` at 1e-6. The zero-ice config turns off ``ice_concentration`` output so the two variable sets match. The ``Fixture_wnat_zero_ice_source`` entry generates its met file. - ``Integration_wnat_moving_ice``: ``compare_ice_moving.py`` checks that ``zeta`` differs from a no-ice run by more than 1e-4, that ``ice_concentration`` matches an independent bilinear interpolation of ``wnat_moving_ice.nc`` at every output time to 1e-6, and that the run matches ``wnat_moving_ice_reference.nc`` at 1e-6. Baroclinic forcing ~~~~~~~~~~~~~~~~~~ Fixtures in ``test/data/wnat/`` (:ref:`baroclinic-test`). Both tests run only on host backends: they require run-to-run determinism, and atomic scatter on GPU backends is not deterministic. - ``Integration_WNAT_baroclinic_zero_forcing``: runs the implicit config with baroclinic off, then with baroclinic on against the all-zero file. ``compare_bitwise.py`` requires the outputs to be identical (no tolerance) and the extra ``bpg_effective_x``/``bpg_effective_y`` outputs to be exactly zero. - ``Integration_WNAT_baroclinic_composed``: runs ``cocoa_config_baroclinic_composed_off.yaml`` and ``_on.yaml``. The ON run must differ from the OFF run by more than 0.05 m in ``zeta`` at the final output time, or in its wet/dry set. It must also match ``baroclinic_composed_reference.nc`` at 1e-6. Python Tools Tests ------------------ The ``cocoa-tools`` package (``utils/cocoa_tools``) has its own pytest suite, which is not part of ``ctest``. Install the package and run the suite from its directory: .. code-block:: bash pip install ./utils/cocoa_tools pytest cd utils/cocoa_tools python -m pytest -q CI runs the same suite in the ``cocoa_tools`` job. The mesh-tools tests read fixtures from ``test/data``. Debugging Failed Tests ---------------------- .. code-block:: bash # Run with verbose output ./test/cocoa_tests -s "[failing_tag]" # Run under debugger lldb ./test/cocoa_tests -- "[failing_tag]" # Run specific named test ./test/cocoa_tests "Exact Test Name" # Show test durations ./test/cocoa_tests -d yes