Step 4: Getting your raw back-ups sorted

Who’s involved

Instrument Scientist, NCAS IT, NCAS Data Manager

Present Action

Establishing the connection with the instrument to ensure that the raw data (at the appropriate level) are captured and fed through to the NCAS GWS area and through to back-up.

Reason

Without the raw data being secured, there’s little that can be done if reprocessing ever needs to happen. As such it’s fundamental that there is a good data management for the lowest level raw data that are meaningful to retain.

Workflow

1. Getting data to JASMIN

   The ncas_obs Group Workspace (GWS) on JASMIN provides an area for raw data from the instruments to be stored. Backing up your data to this space means you are not reliant on a local hard drive being the only copy of your data, and will provide additional back ups managed by the GWS managers through the JASMIN tape system, adding additional security to your data.

   1. Where on JASMIN?

      Raw instrument data should be stored in /gws/pw/j07/ncas_obs_vol{1,2}/raw_data/ . Within this folder are two subfolders - <incoming> should be used for data coming in “live” to JASMIN, e.g. during a field campaign. Once you are sure all of the data has arrived, it should be moved to the data folder, where it will be made read-only.

   2. Which instrument data should be backed up?

      The lowest-level meaningful data that is needed for processing or might be needed for future re-processing should be stored on the GWS.

   3. How do I get it there?

      There are many different tools available for transferring data to JASMIN, and which one is best to use depends on factors such as how much data you need to copy, what operating system is on the computer you are copying from, where (geographically) you are copying data from, and how reliable your internet connection is. See https://help.jasmin.ac.uk/docs/data-transfer/data-transfer-tools/ for a guide from JASMIN on some of the available data transfer tools, or contact NCAS IT for advice and support.

   4. How often/regularly?

      Data should be backed up to the GWS as soon as possible to minimise the chance of data being lost due to hard drive or computer failure. This could be a script that runs every day, or more regularly, that copies the data to JASMIN. In some cases it might not be possible to copy data that quickly, for example a remote field campaign with no internet connection. In those cases, data should be copied to the GWS as soon as is possible afterward to reduce the chances of data loss, with multiple local copies being stored before then.

2. Document the process

   Make sure to record how data is moving from the instrument to JASMIN somewhere that can be accessed by other people, and in a way others can understand and can execute themselves if needed. This ensures that you do not become a single point of failure (for example, if you are ill for a length of time, someone else can manage the data backup process for your instrument if needed).

Future Plan

Further Details