Checkpoint / Restart

Cocoa supports checkpoint/restart for long-running simulations. A simulation can write checkpoint files, and a subsequent simulation can resume (restart) from a checkpoint without restarting from the beginning.

Overview

The checkpoint/restart system works as follows:

1. Checkpoint writing: When checkpointing is enabled, Cocoa writes a checkpoint file containing the full hydrodynamic state. By default a single checkpoint is written at the end of the run; it can optionally write at a regular interval instead.

2. Restart: A new simulation reads a checkpoint file and resumes from the saved state, continuing to the desired end time.

Each write produces a separate, timestamped file named {prefix}.{simulation_time}.nc (for example cocoa_checkpoint.20250101T120000.nc). The timestamp is the simulation time of the checkpoint in YYYYMMDDTHHMMSS form, so files are self-describing and sort chronologically by name. Because every write is a new file, a crash mid-write can only damage the file being written — previously completed checkpoints are never overwritten.

Configuration

The checkpoint section in the YAML configuration file controls checkpoint behavior.

Writing Checkpoints

The simplest configuration enables checkpointing and writes a single checkpoint at the end of the run:

checkpoint:
  enabled: true
  file_prefix: "cocoa_checkpoint"  # Output: cocoa_checkpoint.{simulation_time}.nc

To write checkpoints periodically as well, set write_interval:

checkpoint:
  enabled: true
  write_interval: 12h       # Write every 12 hours (plus one at the end)
  file_prefix: "cocoa_checkpoint"

Parameters:

Parameter	Type	Default	Description
enabled	bool	false	Enable checkpoint writing
write_interval	int or duration	0	How often to write checkpoints. 0 (the default) writes a single checkpoint at the end of the run. A positive value writes every N time steps and one at the final step; it accepts a plain integer (number of time steps) or a duration string (e.g., 6h, 1d). See Duration Syntax.
file_prefix	string	cocoa_checkpoint	Filename prefix. Each write produces {prefix}.{simulation_time}.nc (e.g. cocoa_checkpoint.20250101T120000.nc).

Note

Each checkpoint is a full snapshot of the model state and is not small. When write_interval is set such that more than ten checkpoint files would be produced, Cocoa logs a warning at startup suggesting a larger interval or the end-of-run default (write_interval: 0).

Restarting from a Checkpoint

To restart a simulation from a checkpoint file:

checkpoint:
  enabled: true
  restart_file: "cocoa_checkpoint.20250108T000000.nc"

Parameters:

Parameter	Type	Default	Description
restart_file	string	(none)	Explicit path to the checkpoint file to restart from. When set, the simulation resumes from this checkpoint instead of starting from initial conditions. Because filenames encode the simulation time, the file to use is deterministic — for an end-of-run checkpoint it is {prefix}.{end_time}.nc.

When restart_file is specified, Cocoa:

• Recovers the original simulation start time from the checkpoint file

• Restores all hydrodynamic state (elevation, velocity, flux at all time levels)

• Restores wet/dry status and element active flags

• Restores slope limiters and boundary forcing state

• Resumes time stepping from the checkpoint step

• Continues output numbering from the checkpoint offset

Workflow

A typical checkpoint/restart workflow uses three configuration files:

1. Full continuous run (reference or production):

simulation:
  start_time: 2025-01-01
  end_time: 2025-01-15
  time_step: 10s

output:
  filename: "cocoa_output.nc"
  step_interval: 1h

2. Cold start with checkpoint writing (first segment):

simulation:
  start_time: 2025-01-01
  end_time: 2025-01-08        # Run first half
  time_step: 10s

checkpoint:
  enabled: true
  write_interval: 12h         # Checkpoint every 12 hours
  file_prefix: "cocoa_checkpoint"

output:
  filename: "cocoa_output_coldstart.nc"
  step_interval: 1h

3. Restart from checkpoint (second segment):

simulation:
  end_time: 2025-01-15        # Run to final end time
  time_step: 10s             # Must match original time step

checkpoint:
  enabled: true
  # The cold-start segment ends at 2025-01-08, so its final checkpoint is
  # cocoa_checkpoint.20250108T000000.nc
  restart_file: "cocoa_checkpoint.20250108T000000.nc"
  write_interval: 12h         # Optionally continue writing checkpoints

output:
  filename: "cocoa_output_restart.nc"
  step_interval: 1h

Important

The restart configuration must use the same time_step as the original simulation. start_time may be omitted – the checkpoint records the original cold-start time and the run recovers it before anything reads it. If start_time IS specified, it must equal the recorded one: a restart resumes on the original cold-start clock, and the forcing ramps anchor to start_time, so a moved value would silently re-anchor every ramp (e.g. a fully spun-up run re-ramping from zero) – the reader rejects the mismatch instead. The end_time can be different (typically extended to the desired final time). The checkpoint to restart from is named after its simulation time, so the cold-start segment’s final checkpoint is {prefix}.{cold_start_end_time}.nc.

Checkpoint Contents

Each checkpoint file (NetCDF format) stores:

• Hydrodynamic state: Water surface elevation (zeta), velocity components (u, v), and volume flux (qx, qy) at all three time levels (n+1, n, n-1)

• Derived fields: Rate of change of elevation (del_zeta)

• Wet/dry state: Node wet/dry status, element active status, slope limiters

• Boundary state: Normal flux (qn) and, for radiation boundaries, boundary elevation (en) at all three time levels (n+1, n, n-1)

• Meteorological state (if enabled): Wind stress components (current and previous) and atmospheric pressure (current and previous)

• Tidal potential state (if enabled): Potential values (current and previous)

• Mesh geometry: Total element area per node

• Metadata: Steps completed, time step, simulation start time, output time index, write sequence number

Considerations

Choosing a write interval:

For most workflows the end-of-run default (write_interval: 0) is sufficient: it leaves one checkpoint you can resume from. Set a positive write_interval only when you need intermediate restart points. Each checkpoint is a full state snapshot, so writing involves file I/O and (in MPI mode) gathering data to rank 0; choose an interval that balances restart granularity against I/O overhead and disk usage. Common choices:

• write_interval: 6h to 12h for storm surge runs

• write_interval: 1d to 2d for tidal spinup

If a chosen interval would produce more than ten checkpoint files over the run, Cocoa warns at startup — prefer a larger interval or the end-of-run default in that case.

Peak values are not preserved:

Peak elevation (zeta_max) and other tracked extrema reset on restart because they are not stored in the checkpoint file. If you need continuous peak tracking, use a single continuous run.

Implicit solver restart precision:

When using the implicit (consistent) solver, restarted simulations may show small differences (within the iterative solver’s convergence tolerance) compared to a continuous run. This is because the iterative solver’s initial guess differs on the first restart step (zero vs. previous solution). These differences are inherent to iterative solvers and do not indicate a problem.

Changing inputs across a restart:

A checkpoint carries the fingerprint of the mesh it was written for (mesh_id), and the restart compares it section by section. Node positions, element connectivity and boundaries must match: the saved state lives on that discretization, so a restart onto a different one is refused with a message naming the sections that differ and both digests. Bathymetry, nodal attributes and self-attraction/loading data may differ — the restart proceeds and logs a warning naming each changed section, because the restored state then evolves under the new inputs. A fingerprint that does not parse (a future scheme) is compared whole.

MPI compatibility:

Checkpoint files are written in global (non-partitioned) format. A restart simulation can use a different number of MPI ranks than the original run. Boundary state is stored in the mesh’s global flux boundary node ordering, so each rank recovers exactly its own nodes regardless of how the mesh is split.

Checkpoints written by Cocoa 0.3.1 and earlier under MPI stored only one rank’s share of the flux boundary state. Such a file is rejected on restart with a message naming both node counts; regenerate it by rerunning the cold-start segment.