Quick Guide E: NCAS Observations Group Work Space on JASMIN

What is the NCAS Observations GWS on JASMIN?

The NCAS Observations Group Workspace (“ncas_obs”) is an area on JASMIN dedicated to the storage, processing, and formatting of NCAS observational data in preparation for archiving with CEDA. The workspace is split over two volumes, and is structured such that each instrument has its own space for raw data, processing software, files created through processing, plotting code, and for adding any additional information about the instrument (e.g. calibration certificates, documented processes e.t.c.). Folders for software, tools and additional data are linked to the NCAS GitHub account, with repositories available for each instrument for writing and backing up code, and creating tagged releases.

When to Use it

The ncas_obs space should be used for

• storing and backing up raw data from the instrument
• hosting and running code and software that processes the raw data and converts into netCDF format
• transferring data ready for the CEDA archive
• creating quick-look plots and the data

How to use it

1. Apply for access through JASMIN to the group workspace.
   • Visit https://accounts.jasmin.ac.uk/ and log in with your JASMIN details
   • Click on the green “My Services” button towards the bottom of the page
   • Click on “Group Workspaces” in the menu on the left and search for “ncas_obs”
   • Click on the blue “More information” button
   • Click the green “Apply” button next to “User” on the right hand side
   • Enter some information saying why you need access to the group workspace, and press “Apply”
   • You should receive an email to the address associated with your JASMIN account to say your application has been submitted, and another when the application has been approved
2. Accessing and using the group workspace
   • The group workspace can be found at /gws/pw/j07/ncas_obs_vol1 and /gws/pw/j07/ncas_obs_vol2 from any of the “sci” or “xfer” machines on JASMIN
   • Generally, instruments based at Chilbolton (“cao”) are in ncas_obs_vol2, all other instruments are in ncas_obs_vol1
   • You should be able to see relevant folders for your instrument(s) within the folder for the observatory where they are based (or under “amf” for mobile instruments)
   • Raw data for the instrument should be stored in /gws/pw/j07/ncas_obs_vol{1,2}/<observatory>/raw_data/<instrument name>/<deployment directory>
   • Code needed for processing and formatting data should be stored in /gws/pw/j07/ncas_obs_vol{1,2}/<observatory>/software/<instrument name>, which should be linked to https://github.com/ncasuk/ name>-software
   • Processed data should be stored in /gws/pw/j07/ncas_obs_vol{1,2}/<observatory>/processing/<instrument name>/. When this data is ready to be uploaded to CEDA, netCDF files should be in year/month/day subdirectories as needed (e.g. daily netCDF files should be in year/month subfolders).
   • If you have any problems finding or accessing the folders, or find that the folders that should be GitHub repositories are not working as such, please contact ncas-it@ncas.ac.uk

Tools

1. CEDA upload

   A script is provided for when data is ready to be uploaded from the group workspace to CEDA archive. The script can be run from any of the sci machines on JASMIN: /gws/pw/j07/ncas_obs_vol1/scripts/transfer_to_upload/transfer_to_upload /gws/pw/j07/ncas_obs_vol{1,2}/<observatory>/processing/<instrument name>/<deployment directory> . This script goes through the given directory, runs checksit on each file in the directory and all subdirectories, and if all files pass checksit then they are moved to the upload area, ready to be sent to CEDA. If there are more than 50 files within the given directory, the script will use LOTUS to check all the files.

2. Public HTML

   A public HTML space is provided for the purpose of sharing small files from the group workspace, such as log files, small data files that need to be shared quickly, or latest plots of data. The space is NOT for hosting websites or building tools and applications - please see this article from JASMIN for more details. This space can be found at /gws/pw/j07/ncas_obs_vol1/public, and is available online at https://gws-access.jasmin.ac.uk/public/ncas_obs/.

Further Details

More detail on the structure of the group workspace can be found at https://ncas-obs-gws.readthedocs.io/en/latest/