Containers

Cocoa is distributed as Docker containers pre-built for a range of GPU and CPU architectures, so it can run without compiling from source. The same images also run under Singularity/Apptainer on HPC systems (see Running with Singularity / Apptainer below).

Note

Cocoa computes in double precision; a few bandwidth-sensitive fields are stored in single precision internally (see Numerical Methods). This needs no configuration, and there is one build per architecture with no separate precision variants.

Image Families and Naming

Cocoa images are split into three families by accelerator toolchain, identified by the tag suffix. <tag> is the Cocoa release (for example latest or a version such as 0.6.0):

Image	Hardware	Base
`cocoaorg/cocoa:<tag>-cpu`	CPU only (x86-64 and ARM64)	Ubuntu
`cocoaorg/cocoa:<tag>-cuda`	NVIDIA GPUs (CUDA)	`nvidia/cuda` (Ubuntu)
`cocoaorg/cocoa:<tag>-rocm`	AMD GPUs (ROCm/HIP)	`rocm/dev-ubuntu`

The families are kept separate because the CUDA and ROCm toolkits are mutually exclusive at the hardware level and each is large; a host is either NVIDIA or AMD, so a combined image would only be bigger. With the CPU build in its own image, CPU and CI users do not pull a multi-gigabyte GPU toolkit.

The -cpu image is a single multi-architecture tag: it carries both linux/amd64 and linux/arm64 variants behind one name, and docker pull selects the one matching your host. The ARM64 variant runs natively on Apple Silicon Macs through Docker Desktop, as well as on AWS Graviton, Ampere Altra, and NVIDIA Grace.

The -cuda and -rocm images are GPU-only: they do not include a CPU fallback build. Use the -cpu image to run on a node without a supported GPU.

The Trilinos base images follow the same convention: cocoaorg/trilinos_base:<tag>-cpu, -cuda, and -rocm.

Two moving tags exist alongside the pinned version tags:

Tag	Contents
`latest-<family>`	The most recent tagged release. Use this unless you need unreleased work.
`nightly-<family>`	The development build from `main`, rebuilt automatically on any night that `main` received commits. Untagged, unreleased code.

Supported Architectures

CPU (-cpu image, x86-64 and ARM64):

Architecture	Backend	ISA tuning
`serial`	Single-threaded	Haswell (x86-64) / ARMv8.1 (ARM64)
`openmp`	Multi-threaded	Haswell (x86-64) / ARMv8.1 (ARM64)

NVIDIA GPUs (-cuda image, x86-64 hosts):

Each build is compiled for one compute capability. A GPU runs best on the build for its own compute capability; the table lists the common GPUs under each one. Cocoa computes in double precision, so the FP64 column matters: data-center GPUs run FP64 at half their FP32 rate, while workstation and inference GPUs run it at 1/32 or 1/64 and are far slower for Cocoa than their FP32 figures suggest.

Build	Compute capability	GPUs	FP64 rate
`volta70`	7.0	Tesla V100, Quadro GV100, Titan V	Full (1/2)
`turing75`	7.5	Tesla T4, Quadro RTX 4000–8000, GeForce RTX 20 series	Low (1/32)
`ampere80`	8.0	A100, A30	Full (1/2)
`ampere86`	8.6	A10, A16, A40, RTX A2000–A6000, GeForce RTX 30 series	Low (1/64)
`ada89`	8.9	L4, L40, L40S, RTX 6000 Ada, GeForce RTX 40 series	Low (1/64)
`hopper90`	9.0	H100, H200, H800	Full (1/2)
`blackwell100`	10.0	B200	Full (1/2)
`blackwell103`	10.3	B300 (Blackwell Ultra)	Reduced
`blackwell120`	12.0	RTX PRO 6000 Blackwell (Workstation and Server Edition), RTX PRO 4000/5000 Blackwell, GeForce RTX 50 series	Low (1/64)

B300 trades FP64 throughput for low-precision AI throughput, so for Cocoa a B200 is the faster Blackwell data-center GPU.

To find a GPU’s compute capability:

nvidia-smi --query-gpu=name,compute_cap --format=csv

The CUDA image needs a host driver that supports CUDA 12.9 and the GPU in question. A GPU with no build of its own, such as one newer than the image, runs the newest build of its major compute capability, whose code it runs as is. Failing that, it runs the newest older build, which the driver compiles from PTX at startup and which may run slower; Cocoa prints a warning naming the build when it does this. There is no build for compute capability 11.0 (Jetson Thor), and Kokkos 5.2 has no target for it.

Note

The GPU images are built for x86-64 hosts only. Grace-based systems (GH200, GB200, GB300 and DGX Spark) have ARM64 hosts and cannot run the -cuda image; build Cocoa from source there (see Installation).

AMD GPUs (-rocm image, ROCm 7.2, x86-64 hosts):

Build	GFX target	GPUs
`mi350`	gfx950	Instinct MI350X, MI355X
`mi300`	gfx942	Instinct MI300X, MI300A, MI325X
`mi200`	gfx90a	Instinct MI210, MI250, MI250X

All three run FP64 at full rate. AMD GPU code does not carry over between GFX targets: each GPU must run its own build. rocminfo | grep gfx shows a GPU’s GFX target. The host needs an amdgpu kernel driver that supports ROCm 7.2; see AMD’s ROCm compatibility matrix. Instinct MI100 (gfx908) and Radeon GPUs have no build.

Within an image, the variant name is the build name, for example hopper90, mi300 or openmp.

Running the Container

CPU (x86-64 or ARM64, auto-selected):

docker run -it -v $(pwd):/workspace cocoaorg/cocoa:latest-cpu

NVIDIA GPU (requires the NVIDIA Container Toolkit):

docker run -it --gpus all -v $(pwd):/workspace cocoaorg/cocoa:latest-cuda

AMD GPU (requires the ROCm kernel driver on the host):

docker run -it --device=/dev/kfd --device=/dev/dri \
    --group-add video --security-opt seccomp=unconfined \
    -v $(pwd):/workspace cocoaorg/cocoa:latest-rocm

The -v $(pwd):/workspace flag mounts your current directory into the container’s working directory so Cocoa can access your mesh and configuration files.

Selecting a Variant

Selection is automatic. Each time cocoa runs, it picks a build in this order:

COCOA_ARCH, if it is set.
Otherwise, in a GPU image, the build for the visible GPU: the compute capability that nvidia-smi reports (docker --gpus, singularity --nv), or the GFX target of the AMD GPU (docker --device=/dev/kfd, singularity --rocm).
Otherwise the image’s default: serial for -cpu, ampere80 for -cuda, mi300 for -rocm.

The choice is made by the cocoa command itself, so it holds however the container is started: docker run or docker exec, singularity run, exec or shell, and on every rank of an MPI job. A COCOA_ARCH that names no build in the image is an error that lists the builds the image has, as is a node whose GPUs are of more than one kind.

Set COCOA_ARCH to choose a build yourself, for example to compare two of them:

# Docker
docker run --gpus all -e COCOA_ARCH=hopper90 \
    -v $(pwd):/workspace cocoaorg/cocoa:latest-cuda cocoa -i config.yaml

# Singularity / Apptainer
singularity exec --nv --env COCOA_ARCH=hopper90 cocoa-cuda.sif cocoa -i config.yaml

# Singularity also passes a host export through, unless --cleanenv is given
export COCOA_ARCH=hopper90
singularity exec --nv cocoa-cuda.sif cocoa -i config.yaml

SINGULARITYENV_COCOA_ARCH or APPTAINERENV_COCOA_ARCH on the host work as well, including with --cleanenv.

Inside an interactive shell, select_cocoa pins a build for that shell:

source select_cocoa             # list the builds and show the one that will run
source select_cocoa hopper90    # pin one
source select_cocoa --auto      # back to the visible GPU

which cocoa always shows the launcher, /usr/local/bin/cocoa; to see which build it will run, use source select_cocoa with no argument. Before 0.6.0 the images put the selected build itself on PATH, and COCOA_ARCH took effect only under docker run and singularity run.

Running a Simulation

Inside the container:

cocoa -i your_config.yaml

Four small cases from the cocoa-examples repository (flow_channel, flow_channel_radiation, global_case and internal_weir) are included in the container at /opt/cocoa/examples:

cp -r /opt/cocoa/examples/flow_channel .
cd flow_channel
cocoa -i cocoa_config.yaml

See Quick Start for details on configuration files and expected output.

Converting ADCIRC Meshes

The container has the cocoa-tools Python package installed (see Cocoa Tools (Python)), so cocoa-mesh-tools, cocoa-ocean3d and cocoa-plot are on PATH. cocoa-mesh-tools converts ADCIRC model files to Cocoa’s NetCDF mesh format.

Basic mesh conversion (fort.14 only):

cocoa-mesh-tools from_adcirc \
    --mesh fort.14 \
    --output mesh.nc

With nodal attributes (fort.13):

cocoa-mesh-tools from_adcirc \
    --mesh fort.14 \
    --attributes fort.13 \
    --output mesh.nc

With self-attraction and loading (fort.24):

cocoa-mesh-tools from_adcirc \
    --mesh fort.14 \
    --attributes fort.13 \
    --sal fort.24 \
    --output mesh.nc

Table 2 Conversion Script Options
Flag	Required	Description
`--mesh`	Yes	Path to ADCIRC fort.14 mesh file
`--output`	Yes	Path for output NetCDF file
`--attributes`	No	Path to ADCIRC fort.13 nodal attributes file
`--sal`	No	Path to self-attraction/loading file (fort.24 ASCII or NetCDF)
`--reorder [elements-only\|all]`	No	Reorder the mesh for GPU performance; `all` also renumbers the nodes (see Element Ordering)

A mesh converted without --reorder can be reordered afterwards with cocoa-mesh-tools reorder --mesh mesh.nc --output mesh_sorted.nc (add --renumber-nodes to also renumber the nodes).

See Mesh Preparation for details on the NetCDF mesh format and supported nodal attributes.

Plotting Results

cocoa-plot renders any Cocoa output file on its unstructured mesh. It reads the mesh and variable metadata from the file, so it needs no per-file configuration; --list reports what a file contains (see cocoa-plot for more).

cocoa-plot cocoa_output.nc --list

# a single snapshot, or a peak field from the *_peak.nc file
cocoa-plot cocoa_output.nc -v zeta -t -1 -o zeta.png
cocoa-plot cocoa_output_peak.nc -v zeta_max -o peak.png

# velocity magnitude with arrows, zoomed to a region
cocoa-plot cocoa_output.nc -v velocity --vectors \
    --extent -98 -80 18 31 -o velocity.png

# every 10th snapshot as numbered stills, or as a movie
cocoa-plot cocoa_output.nc -v zeta --every 10 -o zeta.png
cocoa-plot cocoa_output.nc -v zeta --every 10 --fps 12 -o zeta.mp4

Global meshes are detected from the domain extent and drawn in a Robinson projection with the antimeridian handled; --projection selects another. Movie output is chosen by the extension (.mp4, .mov, .webm, .mkv, .gif); ffmpeg is installed in the image for the video formats. --help lists the full set of options.

The image carries a pre-fetched Natural Earth coastline cache, so plotting works on compute nodes with no outbound network. If you run cocoa-plot outside the container in an offline environment, either set CARTOPY_DATA_DIR to a cache you populated elsewhere or pass --no-coastline.

Mounting Data Volumes

Mount your simulation directory into the container so input files are accessible and output files persist after the container exits:

# Mount a single directory
docker run -it --gpus all \
    -v /path/to/simulation:/workspace \
    cocoaorg/cocoa:latest-cuda

# Mount input and output separately
docker run -it --gpus all \
    -v /path/to/meshes:/data/meshes:ro \
    -v /path/to/output:/workspace \
    cocoaorg/cocoa:latest-cuda

Tip

Use :ro (read-only) for input data mounts to prevent accidental modification of source files.

Non-Interactive Execution

Run a simulation without entering the container interactively:

docker run --gpus all \
    -v $(pwd):/workspace \
    cocoaorg/cocoa:latest-cuda \
    cocoa -i config.yaml

Running with Singularity / Apptainer

Many HPC clusters use Singularity (or its successor Apptainer) instead of Docker, since it runs unprivileged and works under schedulers such as SLURM. Singularity pulls and converts the same images from Docker Hub, so no separate build or image format is needed. The commands below use singularity; substitute apptainer if that is what your site provides (the two are command-line compatible).

1. Pull and convert the image from Docker Hub. Singularity fetches the Docker image and converts it into a single .sif file:

singularity pull cocoa-cuda.sif docker://cocoaorg/cocoa:latest-cuda

Note

The GPU images are several gigabytes, and Singularity unpacks them into a temporary directory before assembling the .sif. If /tmp is small (a common default on login nodes), point the cache and temporary directories at a filesystem with sufficient free space before pulling:

export SINGULARITY_CACHEDIR=/path/to/scratch/singularity_cache
export SINGULARITY_TMPDIR=/path/to/scratch/singularity_tmp
mkdir -p "$SINGULARITY_CACHEDIR" "$SINGULARITY_TMPDIR"

2. Confirm the GPU is visible. The --nv flag passes the host NVIDIA driver into the container (use --rocm for the -rocm image). Run nvidia-smi inside the container to check:

singularity exec --nv cocoa-cuda.sif nvidia-smi

If this prints your GPU’s statistics, the container can see the GPU.

3. Confirm Cocoa runs. Cocoa picks the build for the visible GPU (see Selecting a Variant):

singularity exec --nv cocoa-cuda.sif cocoa --device-info

To run a particular build instead, set COCOA_ARCH, either with --env or by exporting it on the host before launching:

singularity exec --nv --env COCOA_ARCH=hopper90 cocoa-cuda.sif cocoa --device-info

export COCOA_ARCH=hopper90
singularity exec --nv cocoa-cuda.sif cocoa --device-info

In a singularity shell session, source select_cocoa hopper90 pins the build for that shell, and source select_cocoa with no argument shows which build will run.

Cocoa reports the GPU it found and the target the selected build was compiled for, then exits. The two should match, for example on a V100:

[info    ] Cocoa version: <version>
[info    ] ==== Cocoa Execution Space Configuration ====
[info    ] Execution Space: CUDA (NVIDIA GPU)
[info    ]   Device: Tesla V100-PCIE-16GB
[info    ]   Compute Capability: 7.0
[info    ]   Compiled For: sm_70
...

Running under MPI. Launch one container per rank with the host’s MPI; each rank picks its build the same way:

srun singularity exec --nv cocoa-cuda.sif cocoa -i config.yaml --partition partition.nc

The host MPI and the container’s Open MPI must be compatible for this hybrid launch; see your site’s container documentation.

4. Launch a clean session with your data mounted. HPC login shells often inject their own environment (module systems, an activated Conda base, a custom LD_LIBRARY_PATH), and because Singularity inherits the host environment and bind-mounts your home directory by default, these can leak into the container and shadow its Python and libraries. Start the container isolated from the host environment and mount your working directory to /workspace:

singularity run -c --cleanenv --nv \
    --bind "$PWD:/workspace" --pwd /workspace \
    cocoa-cuda.sif

-c / --cleanenv isolate the container from the host environment so it uses its own Python, libraries, and PATH. A COCOA_ARCH exported on the host is dropped too, so pin a build with --env COCOA_ARCH=<build> (or SINGULARITYENV_COCOA_ARCH) instead.
--bind "$PWD:/workspace" mounts your current directory (any path works) so inputs are visible and outputs persist after the container exits.
--pwd /workspace starts you in that directory.

From inside this session, mesh conversion and model runs work as with the Docker image: see Converting ADCIRC Meshes above and Quick Start for configuring the YAML input file.

Building the Container

Each family is built in two stages: first the Trilinos base image, then the Cocoa image on top of it. The three families share one build context per image and select the family with a per-family Dockerfile (Dockerfile.cpu, Dockerfile.cuda, Dockerfile.rocm), so the build scripts and entrypoints are not duplicated. Substitute the family suffix throughout.

1. Build the Trilinos base image (example: CUDA):

cd containers/base_trilinos_container
docker build -f Dockerfile.cuda -t cocoaorg/trilinos_base:latest-cuda .

2. Build the Cocoa image:

cd containers/cocoa_container
DOCKER_BUILDKIT=1 docker build --ssh default \
    -f Dockerfile.cuda -t cocoaorg/cocoa:latest-cuda .

The --ssh default flag forwards your SSH agent for private repository access during the build; the agent must be running with the key loaded (ssh-add).

The -cpu family is multi-architecture. Build and push both platform variants under one tag with buildx:

cd containers/cocoa_container
docker buildx build --platform linux/amd64,linux/arm64 \
    -f Dockerfile.cpu -t cocoaorg/cocoa:latest-cpu --push .

On a cluster the build is driven by the SLURM batch scripts in containers/slurm/. submit_all.sh queues every family, making each Cocoa image depend on its Trilinos base; see those scripts for the exact buildx invocation and push steps.

The nightly-<family> tags are published by the Nightly Containers GitHub Actions workflow, which runs build_cocoa.sbatch on the self-hosted runner for any night that main received commits. It rebuilds only the Cocoa images, on top of the current trilinos_base:latest-<family>.

Note

Building a GPU family compiles Trilinos and Cocoa once per architecture in that family and can take several hours. The AMD (-rocm) images are validated by compilation only; runtime validation on AMD hardware is in progress. The ARM64 half of the -cpu build runs natively on an ARM64 builder or, more slowly, under qemu emulation.