Step 9: Send your data for archiving

Step 09: Send your data for archiving

Who’s involved

Instrument scientist, NCAS Data Manager

Introduction

This ensures that the instrument scientist only releases data when they are ready for wider use.

The standard delivery pipeline means that the data will be delivered to the CEDA Archive in a way that the ingestion workflows can automatically handle and that they have passed through Checksit to ensure we're getting properly structure files through to the archive.

Workflow

Once your data passes the checker and you’re happy that other quality control measures have been put in place, it’s time to get your data delivered to the archive.

To do this there is a delivery tool on the NCAS Obs GWS. All you need to do is move your data over to the designated delivery area, run the delivery script and the pipeline will do the rest.

Upload to CEDA

1. Navigate to the processing folder ready to be archived on CEDA
2. Run the transfer_to_upload script to do a final checksit run and move the data into the upload folder from where they will be moved to the CEDA 'Arrivals' area automatically. This script can be used on one file, multiple files, or an entire directory:

cd /gws/pw/j07/ncas_obs_vol1/scripts/transfer_to_upload
./transfer-to-upload /gws/pw/j07/ncas_obs_vol1/iao/processing/ncas-aws-10/20010101_longterm

where /gws/pw/j07/ncas_obs_vol1/iao/processing/ncas-aws-10/20010101_longterm  is a directory that has netCDF files in structured within yyyy/mm  folders as described in the processing folder section, and could be replaced by a file or a space-separated list of files. The absolute path to the directory with the data files must be given, not a relative path from the current working directory.

Future Plan

None

Further Details

See the Upload to CEDA tool documentation at: https://ncas-obs-gws.readthedocs.io/en/latest/tools.html#upload-to-ceda