Step 6: Setting up your NRT feed

Who’s involved

Instrument scientist, NCAS Data Manager

Introduction

Sharing early access to data aids instrument scientists to identify issues with instruments and data feeds. It also provides wider early access to observations ahead of verified release of data to support campaigns etc.

To support this the NCAS IT team can provide a dashboard view to help monitor data flows and visualise various data products.

Additionally, there is a dedicated public access folder on the NCAS Group Work Space which can be used to share some initial files and plots.

Workflow

Grafana for data monitoring

To set up a Grafana dashboard to monitor time series data (e.g. this could be looking at data from a single instrument, looking at multiple instruments together (e.g. same project or platform), or monitoring status of various instrument parameters):

1. Make sure the previous steps of getting data onto the JASMIN group workspace have been completed to allow data to then be pulled into the required database.
2. Discuss with the NCAS IT team what data you want available for NRT feed, including where on JASMIN the data is and which bits of the data you want (e.g. if the raw data is in a CSV file, which columns from that file do you want).
3. NCAS IT will set up an account for you on Grafana, ingest the data you want into the appropriate database, initiate a dashboard (if necessary), and create an example panel with the data.
4. You can then add more panels and customise the dashboard to suit your needs, using the example panel as an initial template for the code required to query the database (NCAS IT can help explain this if unfamiliar):
   1. Visit https://science.ncas.ac.uk/grafana and log in.
   2. Click through to your dashboard.
   3. Create a new panel by clicking “Add” toward the top right of the dashboard, then “Visualisation”. Then create the query you need and adjust any of the options on the right side of the page that are relevant.
   4. Make sure to press the save button (just to the left of the “Add” button), and “Save” on the slide-in.

Other types of data, particularly multi-dimensional data (e.g. time and height, or spatial data) do not work as well on Grafana. If you require a Near Real Time display of these types of data, please contact NCAS IT.

Public Web Folder for early-access sharing

For sharing temporary files etc. (i.e. nothing for long-term re-use) see the NCAS Group Workspace Quick Start guide about use of the public web-folder on the NCAS Obs GWS.

Future Plan

None

Further Details

See linked articles below.