Checkpoint / Restart

A simulation can write checkpoint files holding its full hydrodynamic state, and a later simulation can restart from one instead of starting from the initial conditions. By default Cocoa writes a single checkpoint at the end of the run; it can write at a regular interval instead.

Checkpoint Files

Each write produces a separate, timestamped file named {prefix}.{simulation_time}.nc (for example cocoa_checkpoint.20250101T120000.nc). The timestamp is the simulation time of the checkpoint in YYYYMMDDTHHMMSS form, so files are self-describing and sort chronologically by name. Because every write is a new file, a crash mid-write can only damage the file being written; completed checkpoints are never overwritten.

Configuration

The checkpoint section in the YAML configuration file controls checkpoint behavior.

Writing Checkpoints

This configuration writes a single checkpoint at the end of the run:

checkpoint:
  enabled: true
  file_prefix: "cocoa_checkpoint"  # Output: cocoa_checkpoint.{simulation_time}.nc

To write checkpoints periodically as well, set write_interval:

checkpoint:
  enabled: true
  write_interval: 12h       # Write every 12 hours (plus one at the end)
  file_prefix: "cocoa_checkpoint"

Parameters:

Parameter	Type	Default	Description
`enabled`	bool	false	Enable checkpoint writing
`write_interval`	int or duration	0	How often to write checkpoints. `0` (the default) writes a single checkpoint at the end of the run. A positive value writes every `N` time steps and one at the final step; it accepts a plain integer (number of time steps) or a duration string (e.g., `6h`, `1d`). See Duration Syntax.
`file_prefix`	string	`cocoa_checkpoint`	Filename prefix. Each write produces `{prefix}.{simulation_time}.nc` (e.g. `cocoa_checkpoint.20250101T120000.nc`).

Restarting from a Checkpoint

To restart a simulation from a checkpoint file:

checkpoint:
  enabled: true
  restart_file: "cocoa_checkpoint.20250108T000000.nc"

Parameters:

Parameter	Type	Default	Description
`restart_file`	string	(none)	Explicit path to the checkpoint file to restart from. When set, the simulation resumes from this checkpoint instead of starting from initial conditions. Because filenames encode the simulation time, the file name is known in advance: for an end-of-run checkpoint it is `{prefix}.{end_time}.nc`.

When restart_file is specified, Cocoa:

Recovers the original simulation start time from the checkpoint file
Restores all hydrodynamic state (elevation, velocity, flux at all time levels)
Restores wet/dry status and element active flags
Restores slope limiters and boundary forcing state
Resumes time stepping from the checkpoint step, counting steps from the original cold start
Keeps the output cadence of the original run: output is written when that step count is a multiple of the output interval, so the restart’s output times fall on the continuous run’s

The restart writes a new output file. Give it a different filename from the earlier segment’s output: a file of the same name is overwritten, not appended to.

Workflow

A typical checkpoint/restart workflow uses three configuration files:

1. Full continuous run (reference or production):

simulation:
  start_time: 2025-01-01
  end_time: 2025-01-15
  time_step: 10s

output:
  global:
    filename: "cocoa_output.nc"
    step_interval: 1h

2. Cold start with checkpoint writing (first segment):

simulation:
  start_time: 2025-01-01
  end_time: 2025-01-08        # Run first half
  time_step: 10s

checkpoint:
  enabled: true
  write_interval: 12h         # Checkpoint every 12 hours
  file_prefix: "cocoa_checkpoint"

output:
  global:
    filename: "cocoa_output_coldstart.nc"
    step_interval: 1h

3. Restart from checkpoint (second segment):

simulation:
  end_time: 2025-01-15        # Run to final end time
  time_step: 10s             # Must match original time step

checkpoint:
  enabled: true
  # The cold-start segment ends at 2025-01-08, so its final checkpoint is
  # cocoa_checkpoint.20250108T000000.nc
  restart_file: "cocoa_checkpoint.20250108T000000.nc"
  write_interval: 12h         # Optionally continue writing checkpoints

output:
  global:
    filename: "cocoa_output_restart.nc"
    step_interval: 1h

Important

The restart configuration must use the same time_step as the original simulation. start_time may be omitted: the checkpoint records the original cold-start time and the run recovers it before anything reads it. If start_time IS specified, it must equal the recorded one: a restart resumes on the original cold-start clock, and the forcing ramps anchor to start_time, so a moved value would silently re-anchor every ramp (e.g. a fully spun-up run re-ramping from zero), so the reader rejects the mismatch. The end_time can be different (typically extended to the desired final time), but it must be later than the checkpoint time: a restart that would resume at or after end_time is refused at startup.

Checkpoint Contents

Each checkpoint file (NetCDF format) stores:

Group	Contents
Hydrodynamic state	Water surface elevation (zeta), velocity components (u, v), and volume flux (qx, qy) at all three time levels (n+1, n, n-1)
Derived fields	Rate of change of elevation (del_zeta)
Wet/dry state	Node wet/dry status, element active status, slope limiters
Boundary state	Normal flux (qn) and, for radiation boundaries, boundary elevation (en) at all three time levels (n+1, n, n-1)
Meteorological state (if enabled)	Wind stress components and atmospheric pressure, current and previous
Tidal potential state (if enabled)	Potential values, current and previous
Baroclinic state (if enabled)	Effective baroclinic pressure gradient components, current and previous
De-tiding state (if `forcing.baroclinic.internal_tide` is enabled)	The velocity sample ring buffers, their running sums, and the sampling position
Mesh	Node coordinates, depth and element connectivity (UGRID), and the mesh id
Metadata	Steps completed, time step, simulation start time, write sequence number, mesh node and element counts, and the checkpoint format version

Checkpoints written before Cocoa 0.6.0 also carry total_element_area, which is no longer written or read; such a checkpoint still restores. A checkpoint in any other format version than the current one is rejected; regenerate it by rerunning.

Considerations

Choosing a write interval

For most workflows the end-of-run default (write_interval: 0) is enough: it leaves one checkpoint you can resume from. Set a positive write_interval only when you need intermediate restart points. Each checkpoint is a full state snapshot, so writing involves file I/O and (in MPI mode) gathering data to rank 0; choose an interval that balances restart granularity against I/O overhead and disk usage. Common choices:

write_interval: 6h to 12h for storm surge runs
write_interval: 1d to 2d for tidal spinup

If a chosen interval would produce more than ten checkpoint files over the run, Cocoa warns at startup; use a larger interval or the end-of-run default in that case.

Peak values are not preserved

Peak elevation (zeta_max) and the other tracked extrema are not stored in the checkpoint, so they reset on restart and each peak field covers exactly the run that reports it. If you need continuous peak tracking, use a single continuous run.

A maximum cannot be retracted. If peaks carried across restarts, a single bad step would stay in the peak field of every later cycle in a hot-started chain, even after the cause was fixed, and only a cold start would clear it. Resetting on restart truncates the envelope instead, which is recoverable.

Implicit solver restart precision

With the implicit (consistent) solver, a restarted simulation may differ from a continuous run by up to the iterative solver’s convergence tolerance, because the solver’s initial guess on the first restart step is zero rather than the previous solution.

Changing inputs across a restart

A checkpoint carries the fingerprint of the mesh it was written for (mesh_id), and the restart compares it section by section. Node positions, element connectivity and boundaries must match: the saved state lives on that discretization, so a restart onto a different one is refused with a message naming the sections that differ and both digests. Bathymetry, nodal attributes and self-attraction/loading data may differ: the restart proceeds and logs a warning naming each changed section, because the restored state then evolves under the new inputs. A fingerprint that does not parse (a future scheme) is compared whole.

MPI compatibility

Checkpoint files are written in global (non-partitioned) format. A restart simulation can use a different number of MPI ranks than the original run. Boundary state is stored in the mesh’s global flux boundary node ordering, so each rank recovers exactly its own nodes regardless of how the mesh is split.

Checkpoints written by Cocoa 0.3.1 and earlier under MPI stored only one rank’s share of the flux boundary state. Such a file is rejected on restart with a message naming both node counts; regenerate it by rerunning the cold-start segment.