=======
Testing
=======

Cocoa uses Catch2 for the unit tests, with Kokkos/Trilinos initialization
handled in a custom main entry point. The integration tests are ctest entries
that run the ``cocoa`` binary and compare its output against a committed
reference with a Python script.

Test Framework
--------------

Tests are written with `Catch2 <https://github.com/catchorg/Catch2>`_ (v3)
``TEST_CASE`` and ``SECTION`` macros:

.. code-block:: cpp

   #include <catch2/catch_test_macros.hpp>

   TEST_CASE("Mesh", "[mesh]") {
     // Setup code runs for each section
     const auto mesh = create_test_mesh();

     SECTION("Valid mesh construction") {
       REQUIRE(mesh.num_nodes() == 9);
       REQUIRE(mesh.num_elements() == 8);
     }

     SECTION("Coordinate access") {
       const auto& nodes = mesh.nodes();
       REQUIRE_THAT(nodes[0].location().x(), WithinAbs(-88.0, 1e-10));
     }
   }

The test executable uses a custom ``main()`` to initialize Tpetra/Kokkos before
running tests:

.. code-block:: cpp

   #define CATCH_CONFIG_RUNNER
   #include <catch2/catch_all.hpp>
   #include "Tpetra_Core.hpp"

   int main(int argc, char* argv[]) {
     Tpetra::ScopeGuard tpetra_scope_guard(&argc, &argv);
     return Catch::Session().run(argc, argv);
   }

Building and Running Tests
--------------------------

.. code-block:: bash

   # Build with tests (enabled by default)
   cmake --preset dev
   cmake --build --preset dev

   # Run all tests via CTest, from anywhere in the source tree
   ctest --preset dev

   # Run the test executable directly (paths below are relative to build/)
   cd build
   ./test/cocoa_tests

   # Run specific test cases by tag
   ./test/cocoa_tests "[mesh]"
   ./test/cocoa_tests "[gwce]"

   # Run specific test case by name
   ./test/cocoa_tests "Mesh"

   # List all available tests
   ./test/cocoa_tests --list-tests

Test Files
----------

Test sources live in ``test/`` as ``test_<component>.cpp``, one file per
component, with ``test_main.cpp`` as the entry point. The current set is
``COCOA_TEST_SOURCES`` in ``test/CMakeLists.txt``; it is not listed here
because a copy drifts as files are added. ``./test/cocoa_tests --list-tests``
enumerates the test cases and ``--list-tags`` the tags they are filtered by.

Test Data
---------

Test data and helper utilities are organized as follows:

.. code-block:: text

   test/
   ├── TestMeshData.hpp              # Test mesh generation utilities
   ├── TestPrecision.hpp             # Mixed-precision test tolerance helpers
   ├── test_*.cpp                    # Test source files
   ├── run_integration_test.cmake    # Integration test runner script
   └── data/
       ├── test_config.yaml          # ConfigurationReader test fixture
       └── wnat/                     # WNAT integration test data
           ├── wnat.nc                             # Mesh file
           ├── cocoa_config_*.yaml                 # One config per case
           ├── compare_tides_reference.py          # Reference comparison script
           ├── plot_comparison.py                  # Comparison plot generator
           └── *_reference{,_mpi}.nc               # Serial and MPI references


Runnable example cases live in the
`cocoa-examples <https://github.com/cocoaorg/cocoa-examples>`_ repository; the
fixtures above are separate copies maintained with the tests.

See :doc:`../user_guide/test_cases` for detailed descriptions of each test case,
including expected results and verification plots.

The ``TestMeshData.hpp`` helper provides functions to create small test meshes:

.. code-block:: cpp

   #include "TestMeshData.hpp"

   using TestData::TestMeshData;

   TEST_CASE("Example", "[example]") {
     const auto [x_coords, y_coords] = TestMeshData::create_coordinates();
     const auto elevation = TestMeshData::create_elevation();
     const auto triangles = TestMeshData::create_triangles();
     const auto boundaries = TestMeshData::empty_boundaries();

     // Construct HostMesh (held by unique_ptr), then wrap in Mesh
     auto host_mesh = std::make_unique<Geometry::HostMesh>(
         x_coords, y_coords, elevation, triangles, boundaries,
         TestMeshData::origin, TestMeshData::projection_type);
     const Cocoa::Mesh mesh(std::move(host_mesh));
   }

Catch2 Assertions
-----------------

Common assertions used in Cocoa tests:

**Basic Assertions:**

.. code-block:: cpp

   REQUIRE(expr);           // Fatal assertion
   CHECK(expr);             // Non-fatal assertion
   REQUIRE_FALSE(expr);     // Require expression is false
   REQUIRE_NOTHROW(expr);   // Require no exception thrown
   REQUIRE_THROWS_AS(expr, ExceptionType);  // Require specific exception

**Floating-Point Comparisons:**

.. code-block:: cpp

   #include <catch2/catch_approx.hpp>
   #include <catch2/matchers/catch_matchers_floating_point.hpp>

   using Catch::Approx;
   using Catch::Matchers::WithinAbs;
   using Catch::Matchers::WithinRel;

   REQUIRE(value == Approx(expected));              // Default tolerance
   REQUIRE(value == Approx(expected).epsilon(1e-6)); // Custom tolerance
   REQUIRE_THAT(value, WithinAbs(expected, 1e-10)); // Absolute tolerance
   REQUIRE_THAT(value, WithinRel(expected, 1e-6)); // Relative tolerance

Writing New Tests
-----------------

1. Create or edit a test file in ``test/``:

.. code-block:: cpp

   #include <catch2/catch_test_macros.hpp>
   #include "cocoa_kernel/geometry/Mesh.hpp"

   TEST_CASE("ComponentName", "[component][tag]") {
     SECTION("Descriptive behavior name") {
       // Arrange
       auto obj = create_object();

       // Act
       auto result = obj.method();

       // Assert
       REQUIRE(result == expected);
     }
   }

2. Add the file to ``test/CMakeLists.txt`` if creating a new file:

.. code-block:: cmake

   set(COCOA_TEST_SOURCES
       test_main.cpp
       # ... existing files ...
       test_your_new_file.cpp)

3. Rebuild and run:

.. code-block:: bash

   make cocoa_tests
   ./test/cocoa_tests "[your_tag]"

Compile-Time Tests
------------------

For ``constexpr`` functionality, use ``static_assert``:

.. code-block:: cpp

   // Compile-time tests for constexpr functions
   static_assert(TimeDelta::fromDays(1).totalHours() == 24,
                 "Day to hours conversion failed");
   static_assert(TimeDelta::fromHours(24) == TimeDelta::fromDays(1),
                 "Equality comparison failed");

Integration Tests
-----------------

In addition to unit tests, Cocoa includes end-to-end integration tests that run
a complete simulation on the Western North Atlantic (WNAT) mesh and compare
results against known-good reference solutions. These tests verify that the full
simulation pipeline (mesh loading, tidal forcing with SAL, time stepping, output)
produces bitwise-reproducible results.

Output is always delivered on a background writer thread, so every run
exercises the asynchronous write path; there is no I/O mode to select.

Integration tests span several categories: WNAT tides (explicit and implicit
GWCE, serial and MPI), the GAHM vortex-only and composed-with-gridded-met
cases, the global tidal case (serial, split-output, and MPI), the internal
weir case, and checkpoint/restart round-trips. Each test carries a
``RESOURCE_LOCK`` where it shares a working directory with siblings (see
`Shared Working Directories`_ below). The authoritative, current list of
integration tests -- their names, rank counts, and configs -- is
``test/CMakeLists.txt``; it is not duplicated here because that duplication
drifts as tests are added. Search it for ``add_test(NAME Integration_`` to
enumerate them.

Each WNAT test runs a 1-day tides-only simulation (M2 boundary + tide
potential + SAL) with :math:`\Delta t` = 20 s and compares the output against
a serial reference solution. The MPI variants use a separate reference
because mesh partitioning changes floating-point operation order.

Integration test tolerances are NOT uniform: they range from ``1e-3`` to
``1e-10`` depending on what the comparison must absorb. Serial runs against a
same-arithmetic serial reference use the tightest tolerance (``1e-10`` for
checkpoint round-trips with no intervening solver iteration; ``1e-6`` for most
serial simulation comparisons). MPI runs are relaxed to ``1e-3`` because
partitioning changes the order of floating-point reductions; checkpoint
restart through the iterative consistent-GWCE solver is relaxed to ``1e-5``
because the CG solver's own convergence tolerance is ``1e-5``. Each
``-DTOLERANCE=`` value in ``test/CMakeLists.txt`` carries a comment
explaining why that value and not a tighter one -- treat those comments, not
this page, as the authority; do not tighten or loosen a tolerance without
updating its comment to match.

The comparison script also verifies that wet/dry state (NaN locations)
matches between test and reference at every timestep.

After comparison, a Cartopy-based plotting script generates a visual summary
(``wnat_comparison.png``) with peak water level and velocity maps, difference
maps, and time series at probe points. In CI, this plot is uploaded as a GitHub
Actions artifact.

**Prerequisites:**

Integration tests require Python 3 with ``numpy``, ``xarray`` and ``netCDF4``;
``cartopy`` is needed only by the comparison-plot script. If an import fails,
CMake reports ``Integration tests disabled`` at configure time and skips them.

**CMake options:**

Integration tests are **off by default** and must be explicitly enabled:

.. code-block:: bash

   # Enable integration tests
   cmake --preset dev -Dcocoa_ENABLE_INTEGRATION_TESTS=ON

The ``dev-maintainer``, ``ci`` and ``ci-coverage`` presets already enable them.

In CI, integration tests are enabled for the serial build-and-test jobs and the
coverage job. GPU compile-check jobs do not run tests.

**Running integration tests selectively:**

.. code-block:: bash

   # Run only integration tests
   ctest -R Integration

   # Run only the serial integration test
   ctest -R Integration_WNAT_tides$

   # Run only the MPI integration test
   ctest -R Integration_WNAT_tides_mpi

   # Exclude integration tests (run unit tests only)
   ctest -E Integration

**Regenerating reference solutions:**

If numerical methods change and produce different (but correct) results, the
reference solutions must be regenerated with ``test/regenerate_references.sh``
-- do not hand-run ``cocoa`` and ``mv`` the output; the script is the single
source of truth for which config produces which reference file, and it
enforces the conditions those references depend on:

.. code-block:: bash

   # Regenerate everything (wnat, global, weir, gahm)
   test/regenerate_references.sh --cocoa build/src/cocoa/cocoa

   # Regenerate one case only
   test/regenerate_references.sh --cocoa build/src/cocoa/cocoa wnat

Run it inside the build container (or an environment with the same compiler,
Trilinos, and Kokkos as CI) so the regenerated references are reproducible:
every run is single-threaded (``OMP_NUM_THREADS=1``), so the bytes it produces
are bit-reproducible run to run on the same toolchain. The script clears any
stale ``cocoa_output.nc`` before each run so a failed run cannot silently
leave a misleading reference, and it discards the transient
``cocoa_output_peak.nc`` file rather than storing it. It regenerates the
serial WNAT, MPI WNAT, implicit WNAT (serial and MPI), GAHM vortex-only,
global, and internal-weir references; split-output and MPI variants of the
global and weir cases compare against the serial reference rather than
carrying their own, so those are not separately regenerated.

Commit the regenerated ``.nc`` files with provenance in the commit body: the
commit and config that produced them, and why (what numerical change made the
old references stale). Regenerate references deliberately, never as a
drive-by fix for an unrelated failing test -- a reference that moves without
a documented reason erases the test's ability to catch a real regression.

.. _Shared Working Directories:

**Shared working directories:**

Integration tests run serially (label ``integration``) because several of
them share a working directory and would otherwise race on the same input
and output files. Every test that runs against ``test/data/wnat`` carries
``set_tests_properties(... PROPERTIES RESOURCE_LOCK wnat_workdir)``. Search
``test/CMakeLists.txt`` for ``RESOURCE_LOCK wnat_workdir`` for the current
set; it grows with every case added against that directory. A new test that
reads or writes into ``test/data/wnat`` must take the same lock, or it can
run concurrently with an existing WNAT test and corrupt its output. Tests rooted in their own directory (``global_case``,
``internal_weir``, and the checkpoint tests under those directories) do not
share a workdir with any other case and so carry no ``RESOURCE_LOCK`` --
only add one when a new test genuinely reads or writes files another test
also touches.

Debugging Failed Tests
----------------------

.. code-block:: bash

   # Run with verbose output
   ./test/cocoa_tests -s "[failing_tag]"

   # Run under debugger
   lldb ./test/cocoa_tests -- "[failing_tag]"

   # Run specific named test
   ./test/cocoa_tests "Exact Test Name"

   # Show test durations
   ./test/cocoa_tests -d yes