Step 6: Setting up your NRT feed

Who’s involved

Instrument scientist, NCAS Data Manager

Present Action

Near Real Time (NRT) feeds into Grafana.

Other feeds to other external services or campaign tools - e.g. FORCE as arranged by instrument scientist.

Reason

Sharing early access to data aids instrument scientists to identify issues with instruments and data feeds. It also provides wider early access to observations ahead of verified release of data to support campaigns etc.

Workflow

Grafana for time series data - this could be looking at data from a single instrument, looking at multiple instruments together (e.g. same project or platform), or monitoring status of various instrument parameters.

1. Make sure the previous steps of getting data onto the JASMIN group workspace have been completed to allow data to then be pulled into the required database.
2. Discuss with the NCAS IT team what data you want available for NRT feed, including where on JASMIN the data is and which bits of the data you want (e.g. if the raw data is in a CSV file, which columns from that file do you want).
3. NCAS IT will set up an account for you on Grafana, ingest the data you want into the appropriate database, initiate a dashboard (if necessary), and create an example panel with the data.
4. You can then add more panels and customise the dashboard to suit your needs, using the example panel as an initial template for the code required to query the database (NCAS IT can help explain this if unfamiliar):
   1. Visit https://science.ncas.ac.uk/grafana and log in.
   2. Click through to your dashboard.
   3. Create a new panel by clicking “Add” toward the top right of the dashboard, then “Visualisation”. Then create the query you need and adjust any of the options on the right side of the page that are relevant.
   4. Make sure to press the save button (just to the left of the “Add” button), and “Save” on the slide-in.

Other types of data, particularly multi-dimensional data (e.g. time and height, or spatial data) do not work as well on Grafana. If you require a Near Real Time display of these types of data, please contact NCAS IT.

Future Plan

Further Details