==========
Containers
==========

Cocoa is distributed as Docker containers pre-built for a range of GPU and CPU
architectures. This is the easiest way to get started without compiling from
source. The same images also run under Singularity/Apptainer on HPC systems
(see `Running with Singularity / Apptainer`_ below).

.. note::

   Cocoa computes in double precision; a few bandwidth-sensitive fields are
   stored in single precision internally (see :doc:`../theory/numerical_methods`).
   This is built in and requires no configuration, so there is a single build
   per architecture rather than separate precision variants.

Image Families and Naming
-------------------------

Cocoa images are split into three families by accelerator toolchain, identified
by the tag suffix. ``<tag>`` is the Cocoa release (for example ``latest`` or a
version such as ``0.6.0``):

.. list-table::
   :header-rows: 1
   :widths: 30 25 45

   * - Image
     - Hardware
     - Base
   * - ``cocoaorg/cocoa:<tag>-cpu``
     - CPU only (x86-64 and ARM64)
     - Ubuntu
   * - ``cocoaorg/cocoa:<tag>-cuda``
     - NVIDIA GPUs (CUDA)
     - ``nvidia/cuda`` (Ubuntu)
   * - ``cocoaorg/cocoa:<tag>-rocm``
     - AMD GPUs (ROCm/HIP)
     - ``rocm/dev-ubuntu``

The families are kept separate because the CUDA and ROCm toolkits are mutually
exclusive at the hardware level and each is large; a host is either NVIDIA or
AMD, so a combined image would only add bloat. Splitting the CPU build out of
the GPU images also keeps CPU and CI users from pulling a multi-gigabyte GPU
toolkit they will never use.

The ``-cpu`` image is a single **multi-architecture** tag: it carries both
``linux/amd64`` and ``linux/arm64`` variants behind one name, and ``docker
pull`` automatically selects the one matching your host. The ARM64 variant runs
natively on Apple Silicon Macs through Docker Desktop, as well as on AWS
Graviton, Ampere Altra, and NVIDIA Grace.

The ``-cuda`` and ``-rocm`` images are **GPU-only**: they do not include a CPU
fallback build. Use the ``-cpu`` image to run on a node without a supported GPU.

The Trilinos base images follow the same convention:
``cocoaorg/trilinos_base:<tag>-cpu``, ``-cuda``, and ``-rocm``.

Two moving tags exist alongside the pinned version tags:

.. list-table::
   :header-rows: 1
   :widths: 25 75

   * - Tag
     - Contents
   * - ``latest-<family>``
     - The most recent tagged release. Use this unless you need unreleased work.
   * - ``nightly-<family>``
     - The development build from ``main``, rebuilt automatically on any night
       that ``main`` received commits. Untagged, unreleased code.

Supported Architectures
-----------------------

**CPU** (``-cpu`` image, x86-64 and ARM64):

.. list-table::
   :header-rows: 1
   :widths: 20 30 35

   * - Architecture
     - Backend
     - ISA tuning
   * - ``serial``
     - Single-threaded
     - Haswell (x86-64) / ARMv8.1 (ARM64)
   * - ``openmp``
     - Multi-threaded
     - Haswell (x86-64) / ARMv8.1 (ARM64)

**NVIDIA GPUs** (``-cuda`` image, x86-64 hosts):

Each build is compiled for one compute capability. A GPU runs best on the
build for its own compute capability; the table lists the common GPUs under
each one. Cocoa computes in double precision, so the FP64 column matters:
data-center GPUs run FP64 at half their FP32 rate, while workstation and
inference GPUs run it at 1/32 or 1/64 and are far slower for Cocoa than their
FP32 figures suggest.

.. list-table::
   :header-rows: 1
   :widths: 18 10 52 20

   * - Build
     - Compute capability
     - GPUs
     - FP64 rate
   * - ``volta70``
     - 7.0
     - Tesla V100, Quadro GV100, Titan V
     - Full (1/2)
   * - ``turing75``
     - 7.5
     - Tesla T4, Quadro RTX 4000--8000, GeForce RTX 20 series
     - Low (1/32)
   * - ``ampere80``
     - 8.0
     - A100, A30
     - Full (1/2)
   * - ``ampere86``
     - 8.6
     - A10, A16, A40, RTX A2000--A6000, GeForce RTX 30 series
     - Low (1/64)
   * - ``ada89``
     - 8.9
     - L4, L40, L40S, RTX 6000 Ada, GeForce RTX 40 series
     - Low (1/64)
   * - ``hopper90``
     - 9.0
     - H100, H200, H800
     - Full (1/2)
   * - ``blackwell100``
     - 10.0
     - B200
     - Full (1/2)
   * - ``blackwell103``
     - 10.3
     - B300 (Blackwell Ultra)
     - Reduced
   * - ``blackwell120``
     - 12.0
     - RTX PRO 6000 Blackwell (Workstation and Server Edition), RTX PRO
       4000/5000 Blackwell, GeForce RTX 50 series
     - Low (1/64)

B300 trades FP64 throughput for low-precision AI throughput, so for Cocoa a
B200 is the faster Blackwell data-center GPU.

To find a GPU's compute capability:

.. code-block:: bash

   nvidia-smi --query-gpu=name,compute_cap --format=csv

The CUDA image needs a host driver that supports CUDA 12.9 and the GPU in
question. A GPU with no build of its own, such as one newer than the image,
runs the newest build of its major compute capability, whose code it runs as
is. Failing that, it runs the newest older build, which the driver compiles
from PTX at startup and which may run slower; Cocoa prints a warning naming
the build when it does this.
There is no build for compute capability 11.0 (Jetson Thor), and Kokkos 5.2 has
no target for it.

.. note::

   The GPU images are built for x86-64 hosts only. Grace-based systems --
   GH200, GB200, GB300 and DGX Spark -- have ARM64 hosts and cannot run the
   ``-cuda`` image; build Cocoa from source there (see
   :doc:`installation`).

**AMD GPUs** (``-rocm`` image, ROCm 7.2, x86-64 hosts):

.. list-table::
   :header-rows: 1
   :widths: 18 12 70

   * - Build
     - GFX target
     - GPUs
   * - ``mi350``
     - gfx950
     - Instinct MI350X, MI355X
   * - ``mi300``
     - gfx942
     - Instinct MI300X, MI300A, MI325X
   * - ``mi200``
     - gfx90a
     - Instinct MI210, MI250, MI250X

All three run FP64 at full rate. AMD GPU code does not carry over between GFX
targets: each GPU must run its own build. ``rocminfo | grep gfx`` shows a
GPU's GFX target. The host needs an ``amdgpu`` kernel
driver that supports ROCm 7.2; see AMD's ROCm compatibility matrix. Instinct
MI100 (gfx908) and Radeon GPUs have no build.

Within an image, the variant name is the build name, for example ``hopper90``,
``mi300`` or ``openmp``.

Running the Container
---------------------

**CPU (x86-64 or ARM64, auto-selected):**

.. code-block:: bash

   docker run -it -v $(pwd):/workspace cocoaorg/cocoa:latest-cpu

**NVIDIA GPU (requires the NVIDIA Container Toolkit):**

.. code-block:: bash

   docker run -it --gpus all -v $(pwd):/workspace cocoaorg/cocoa:latest-cuda

**AMD GPU (requires the ROCm kernel driver on the host):**

.. code-block:: bash

   docker run -it --device=/dev/kfd --device=/dev/dri \
       --group-add video --security-opt seccomp=unconfined \
       -v $(pwd):/workspace cocoaorg/cocoa:latest-rocm

The ``-v $(pwd):/workspace`` flag mounts your current directory into the
container's working directory so Cocoa can access your mesh and configuration
files.

Selecting a Variant
-------------------

Selection is automatic. Each time ``cocoa`` runs, it picks a build in this
order:

1. ``COCOA_ARCH``, if it is set.
2. Otherwise, in a GPU image, the build for the visible GPU: the compute
   capability that ``nvidia-smi`` reports (``docker --gpus``, ``singularity
   --nv``), or the GFX target of the AMD GPU (``docker --device=/dev/kfd``,
   ``singularity --rocm``).
3. Otherwise the image's default: ``serial`` for ``-cpu``, ``ampere80`` for
   ``-cuda``, ``mi300`` for ``-rocm``.

The choice is made by the ``cocoa`` command itself, so it holds however the
container is started: ``docker run`` or ``docker exec``, ``singularity run``,
``exec`` or ``shell``, and on every rank of an MPI job. A ``COCOA_ARCH`` that
names no build in the image is an error that lists the builds the image has,
as is a node whose GPUs are of more than one kind.

Set ``COCOA_ARCH`` to choose a build yourself, for example to compare two of
them:

.. code-block:: bash

   # Docker
   docker run --gpus all -e COCOA_ARCH=hopper90 \
       -v $(pwd):/workspace cocoaorg/cocoa:latest-cuda cocoa -i config.yaml

   # Singularity / Apptainer
   singularity exec --nv --env COCOA_ARCH=hopper90 cocoa-cuda.sif cocoa -i config.yaml

   # Singularity also passes a host export through, unless --cleanenv is given
   export COCOA_ARCH=hopper90
   singularity exec --nv cocoa-cuda.sif cocoa -i config.yaml

``SINGULARITYENV_COCOA_ARCH`` or ``APPTAINERENV_COCOA_ARCH`` on the host work
as well, including with ``--cleanenv``.

Inside an interactive shell, ``select_cocoa`` pins a build for that shell:

.. code-block:: bash

   source select_cocoa             # list the builds and show the one that will run
   source select_cocoa hopper90    # pin one
   source select_cocoa --auto      # back to the visible GPU

``which cocoa`` always shows the launcher, ``/usr/local/bin/cocoa``; to see
which build it will run, use ``source select_cocoa`` with no argument. Before
0.6.0 the images put the selected build itself on ``PATH``, and
``COCOA_ARCH`` took effect only under ``docker run`` and ``singularity run``.

Running a Simulation
--------------------

Inside the container:

.. code-block:: bash

   cocoa -i your_config.yaml

Four small cases from the
`cocoa-examples <https://github.com/cocoaorg/cocoa-examples>`_ repository
(``flow_channel``, ``flow_channel_radiation``, ``global_case`` and
``internal_weir``) are included in the container at ``/opt/cocoa/examples``:

.. code-block:: bash

   cp -r /opt/cocoa/examples/flow_channel .
   cd flow_channel
   cocoa -i cocoa_config.yaml

See :doc:`quick_start` for details on configuration files and expected output.

Converting ADCIRC Meshes
------------------------

The container has the ``cocoa-tools`` Python package installed (see
:doc:`../user_guide/cocoa_tools`), so ``cocoa-mesh-tools``, ``cocoa-ocean3d``
and ``cocoa-plot`` are on ``PATH``. ``cocoa-mesh-tools`` converts ADCIRC model
files to Cocoa's NetCDF mesh format.

**Basic mesh conversion (fort.14 only):**

.. code-block:: bash

   cocoa-mesh-tools from_adcirc \
       --mesh fort.14 \
       --output mesh.nc

**With nodal attributes (fort.13):**

.. code-block:: bash

   cocoa-mesh-tools from_adcirc \
       --mesh fort.14 \
       --attributes fort.13 \
       --output mesh.nc

**With self-attraction and loading (fort.24):**

.. code-block:: bash

   cocoa-mesh-tools from_adcirc \
       --mesh fort.14 \
       --attributes fort.13 \
       --sal fort.24 \
       --output mesh.nc

.. list-table:: Conversion Script Options
   :header-rows: 1
   :widths: 25 15 60

   * - Flag
     - Required
     - Description
   * - ``--mesh``
     - Yes
     - Path to ADCIRC fort.14 mesh file
   * - ``--output``
     - Yes
     - Path for output NetCDF file
   * - ``--attributes``
     - No
     - Path to ADCIRC fort.13 nodal attributes file
   * - ``--sal``
     - No
     - Path to self-attraction/loading file (fort.24 ASCII or NetCDF)
   * - ``--reorder [elements-only|all]``
     - No
     - Reorder the mesh for GPU performance; ``all`` also renumbers the nodes
       (see :ref:`element-ordering`)

A mesh converted without ``--reorder`` can be reordered afterwards with
``cocoa-mesh-tools reorder --mesh mesh.nc --output mesh_sorted.nc`` (add
``--renumber-nodes`` to also renumber the nodes).

See :doc:`../user_guide/mesh_preparation` for details on the NetCDF mesh format
and supported nodal attributes.

Plotting Results
----------------

``cocoa-plot`` renders any Cocoa output file on its unstructured mesh. It
reads the mesh and variable metadata from the file, so it needs no per-file
configuration; ``--list`` reports what a file contains (see
:ref:`cocoa-plot` for more).

.. code-block:: bash

   cocoa-plot cocoa_output.nc --list

   # a single snapshot, or a peak field from the *_peak.nc file
   cocoa-plot cocoa_output.nc -v zeta -t -1 -o zeta.png
   cocoa-plot cocoa_output_peak.nc -v zeta_max -o peak.png

   # velocity magnitude with arrows, zoomed to a region
   cocoa-plot cocoa_output.nc -v velocity --vectors \
       --extent -98 -80 18 31 -o velocity.png

   # every 10th snapshot as numbered stills, or as a movie
   cocoa-plot cocoa_output.nc -v zeta --every 10 -o zeta.png
   cocoa-plot cocoa_output.nc -v zeta --every 10 --fps 12 -o zeta.mp4

Global meshes are detected from the domain extent and drawn in a Robinson
projection with the antimeridian handled; ``--projection`` selects another.
Movie output is chosen by the extension (``.mp4``, ``.mov``, ``.webm``,
``.mkv``, ``.gif``); ffmpeg is installed in the image for the video formats.
``--help`` lists the full set of options.

The image carries a pre-fetched Natural Earth coastline cache, so plotting
works on compute nodes with no outbound network. If you run ``cocoa-plot``
outside the container in an offline environment, either set ``CARTOPY_DATA_DIR`` to a
cache you populated elsewhere or pass ``--no-coastline``.

Mounting Data Volumes
---------------------

Mount your simulation directory into the container so input files are
accessible and output files persist after the container exits:

.. code-block:: bash

   # Mount a single directory
   docker run -it --gpus all \
       -v /path/to/simulation:/workspace \
       cocoaorg/cocoa:latest-cuda

   # Mount input and output separately
   docker run -it --gpus all \
       -v /path/to/meshes:/data/meshes:ro \
       -v /path/to/output:/workspace \
       cocoaorg/cocoa:latest-cuda

.. tip::

   Use ``:ro`` (read-only) for input data mounts to prevent accidental
   modification of source files.

Non-Interactive Execution
-------------------------

Run a simulation without entering the container interactively:

.. code-block:: bash

   docker run --gpus all \
       -v $(pwd):/workspace \
       cocoaorg/cocoa:latest-cuda \
       cocoa -i config.yaml

Running with Singularity / Apptainer
------------------------------------

Many HPC clusters use Singularity (or its successor Apptainer) instead of
Docker, since it runs unprivileged and integrates cleanly with schedulers such
as SLURM. Singularity can pull and convert the same images directly from Docker
Hub -- no separate build or image format is required. The commands below use
``singularity``; substitute ``apptainer`` if that is what your site provides
(the two are command-line compatible).

**1. Pull and convert the image from Docker Hub.** Singularity fetches the
Docker image and converts it into a single ``.sif`` file:

.. code-block:: bash

   singularity pull cocoa-cuda.sif docker://cocoaorg/cocoa:latest-cuda

.. note::

   The GPU images are several gigabytes, and Singularity unpacks them into a
   temporary directory before assembling the ``.sif``. If ``/tmp`` is small (a
   common default on login nodes), point the cache and temporary directories at
   a filesystem with sufficient free space *before* pulling:

   .. code-block:: bash

      export SINGULARITY_CACHEDIR=/path/to/scratch/singularity_cache
      export SINGULARITY_TMPDIR=/path/to/scratch/singularity_tmp
      mkdir -p "$SINGULARITY_CACHEDIR" "$SINGULARITY_TMPDIR"

**2. Confirm the GPU is visible.** The ``--nv`` flag passes the host NVIDIA
driver into the container (use ``--rocm`` for the ``-rocm`` image). The quickest
check is to run ``nvidia-smi`` inside the container:

.. code-block:: bash

   singularity exec --nv cocoa-cuda.sif nvidia-smi

If this prints your GPU's statistics, the card was passed through successfully.

**3. Confirm Cocoa runs.** Cocoa picks the build for the visible GPU (see
`Selecting a Variant`_):

.. code-block:: bash

   singularity exec --nv cocoa-cuda.sif cocoa --version

To run a particular build instead, set ``COCOA_ARCH``, either with ``--env``
or by exporting it on the host before launching:

.. code-block:: bash

   singularity exec --nv --env COCOA_ARCH=hopper90 cocoa-cuda.sif cocoa --version

   export COCOA_ARCH=hopper90
   singularity exec --nv cocoa-cuda.sif cocoa --version

In a ``singularity shell`` session, ``source select_cocoa hopper90`` pins the
build for that shell, and ``source select_cocoa`` with no argument shows which
build will run.

You should see the version banner, for example:

.. code-block:: text

   cocoa <version>
   Copyright (C) 2026 Zach Cobell

**Running under MPI.** Launch one container per rank with the host's MPI;
each rank picks its build the same way:

.. code-block:: bash

   srun singularity exec --nv cocoa-cuda.sif cocoa -i config.yaml --partition partition.nc

The host MPI and the container's Open MPI must be compatible for this hybrid
launch; see your site's container documentation.

**4. Launch a clean session with your data mounted.** HPC login shells often
inject their own environment -- module systems, an activated Conda ``base``, a
custom ``LD_LIBRARY_PATH`` -- and because Singularity inherits the host
environment and bind-mounts your home directory by default, these can leak into
the container and shadow its Python and libraries. Start the container isolated
from the host environment and mount your working directory to ``/workspace``:

.. code-block:: bash

   singularity run -c --cleanenv --nv \
       --bind "$PWD:/workspace" --pwd /workspace \
       cocoa-cuda.sif

* ``-c`` / ``--cleanenv`` isolate the container from the host environment so it
  uses its own Python, libraries, and ``PATH``. A ``COCOA_ARCH`` exported on
  the host is dropped too, so pin a build with ``--env COCOA_ARCH=<build>``
  (or ``SINGULARITYENV_COCOA_ARCH``) instead.
* ``--bind "$PWD:/workspace"`` mounts your current directory (any path works)
  so inputs are visible and outputs persist after the container exits.
* ``--pwd /workspace`` starts you in that directory.

From inside this session, mesh conversion and model runs work exactly as with
the Docker image: see `Converting ADCIRC Meshes`_ above and :doc:`quick_start`
for configuring the YAML input file.

Building the Container
----------------------

Each family is built in two stages: first the Trilinos base image, then the
Cocoa image on top of it. The three families share one build context per image
and select the family with a per-family Dockerfile (``Dockerfile.cpu``,
``Dockerfile.cuda``, ``Dockerfile.rocm``), so the build scripts and entrypoints
are not duplicated. Substitute the family suffix throughout.

**1. Build the Trilinos base image** (example: CUDA):

.. code-block:: bash

   cd containers/base_trilinos_container
   docker build -f Dockerfile.cuda -t cocoaorg/trilinos_base:latest-cuda .

**2. Build the Cocoa image:**

.. code-block:: bash

   cd containers/cocoa_container
   DOCKER_BUILDKIT=1 docker build --ssh default \
       -f Dockerfile.cuda -t cocoaorg/cocoa:latest-cuda .

The ``--ssh default`` flag forwards your SSH agent for private repository
access during the build. Ensure your SSH agent is running with the appropriate
key loaded (``ssh-add``).

The ``-cpu`` family is multi-architecture. Build and push both platform
variants under one tag with ``buildx``:

.. code-block:: bash

   cd containers/cocoa_container
   docker buildx build --platform linux/amd64,linux/arm64 \
       -f Dockerfile.cpu -t cocoaorg/cocoa:latest-cpu --push .

On a cluster the build is driven by the SLURM batch scripts in
``containers/slurm/``. ``submit_all.sh`` queues every family, making each Cocoa
image depend on its Trilinos base; see those scripts for the exact ``buildx``
invocation and push steps.

The ``nightly-<family>`` tags are published by the ``Nightly Containers``
GitHub Actions workflow, which runs ``build_cocoa.sbatch`` on the self-hosted
runner for any night that ``main`` received commits. It rebuilds only the Cocoa
images, on top of the current ``trilinos_base:latest-<family>``.

.. note::

   Building a GPU family compiles Trilinos and Cocoa once per architecture in
   that family, which is resource-intensive and may take several hours. The
   AMD (``-rocm``) images are currently validated by compilation; runtime
   validation on AMD hardware is ongoing. The ARM64 half of the ``-cpu`` build
   runs natively on an ARM64 builder or, more slowly, under ``qemu`` emulation.