Skip to content

Data Access

Overview

Dashboard access

The following dashboard tools are provided by MONIT:

Programmatic access

The following options are avaiable (some tools are not maintained by MONIT):

  • Grafana Proxy: generic access to all data in MONIT (details below)
  • HDFS: command line tool to explore MONIT data in HDFS
  • Spark: processing platform for stream/batch processing of data
  • SWAN: notebook tool to access MONIT data

Grafana

The MONIT Grafana service is available at: https://monit-grafana.cern.ch. You should login with SSO (the user/password login does not work). The official Grafana documentation can be found here. A 10min beginners guide to building dashboards can be found here.

Organizations

Grafana is managed around the concept of organizations, grouping related data sources, users, and dashboards together.

MONIT is the default organization and provides few core dashboards:

Other organizations created on-demand for projects or services: ATLAS, EOS, etc.

User and Roles

All users that try to access Grafana (onced logged in via SSO) automatically become Grafana users. They are also automatically associated to all Public organizations.
For each organization, each user can have one of the following roles: Admin, Editor, Viewer. More details here.

Datasources

As part of the integration with MONIT you should also get a readonly account credentials so you can configure datasources on your side. Supported Datasources from MONIT are "Elasticsearch" (for opensearch access), InfluxDB (for InfluxDB access) and Prometheus (for accessing Mimir) since these are the backends we currently provide (this doesn't mean you can't use other datasources from Grafana).

Elasticsearch datasource

  • URL: https://\<cluster>.cern.ch:443/es
    • Valid clusters are: monit-timber (public logs), monit-timberprivate (private logs), monit-opensearch (shor-term metrics), monit-opensearch-lt (long-term metrics)
  • Basic auth: enabled
  • User: \<producer>_ro
  • Password: Check the tbag service
  • Index name: [monit_\_\<producer>*]
  • Pattern: Daily (most of the cases)
  • Time field name: metadata.timestamp (unless you need to plot by some other specific field)
  • ES version: 7.10+

InfluxDB datasource

  • URL: Depends on the data you want to access, so you will need to ask the MONIT team
  • Basic auth: enabled
  • User: \<producer>_ro
  • Password: Check the tbag service
  • Database: monit_production_\<producer>
  • Time filtering show tag queries: 3h

Prometheus datasource

  • URL: https://monit-prom-lts.cern.ch/prometheus/
  • Basic auth: enabled
  • User: \<producer>
  • Password: Check the tbag service
  • Manage alerts via Alerting UI: disabled

Grafana annotations

Grafana provides the possibility to use some datasources as annotations to add extra useful information in your plots. From the monitoring team, we aim to provide a set of central datasources that can be used for this, under this page we will be building a list of them and try to summarise which information is inside in order for service managers to use them.

Please note that in order to get the datasource added into your organisation, you need to request so opening a SNOW ticket against us first so we can add it.

Grafana Proxy

JSON data can be retrieved with a HTTP GET request to Grafana specifying the desired query for a particular database.

curl -H "Authorization: Bearer GRAFANA_TOKEN" "https://monit-grafana.cern.ch/api/datasources/proxy/DATA_SOURCE_ID/query?db=DATABASE_NAMEr&q=QUERY"
Important notes:

  • The query is organization specific so please avoid having the browser sessions for the same user running on different organizations while querying.
  • The data source id can be retreived from Grafana by inspecting the actual queries run by Grafana in the browser. You can also ask an organisation admin to provide it to you, it can be retrieved on the datasources page for the organisation.
  • In case the request is done via CLI (e.g. curl, wget), hence without a valid session cookie, a Grafana API Token should be used for authentication. The token has to be created by the organization admin and specified as HTTP header of the request.

Query example for InfluxDB 

curl -X POST -H "Authorization: Bearer TOKEN_HERE" "https://monit-grafana.cern.ch/api/datasources/proxy/7731/query?db=monit_production_cmsjm" --data-binary "@query_file"
The query file must follow this format: 
q=select * from condor where time > now() - 1h limit 1

Query example for Elasticsearch/Opendistro/OpenSearch 

curl -X POST -H 'Content-Type: application/json' -H "Authorization: Bearer TOKEN_HERE" "https://monit-grafana.cern.ch/api/datasources/proxy/7617/_msearch" --data-binary "@query_file"
The query file must follow this format: * the 1st describes the index to use * the 2nd contains the actual query

{"search_type": "query_then_fetch","ignore_unavailable": true,"index": ["monit_prod_wmagent_*", "monit_prod_wmagent_*", "monit_prod_wmagent_*"]}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}
Please refer to the Elasticsearch _msearch documentation for more information

Grafana Upgrades

Every once in a while the MONIT Grafana service is upgraded. As rule of thumb we stay away from fresh versions (e.g. 6.2.0) and upgrade when the first bugfix version is available (e.g. 6.2.1).

Nevertheless this is susceptible to the service's status. For instance a security event may trigger a forceful upgrade.

Procedure

This information only refers to major upgrades (e.g. 6.7 to 6.8), minor upgrades (e.g. 6.7.2 to 6.7.3) are not included since they are always backward compatible.

The following procedure was developed so users can still benefit from their unofficial plugins and remain able to fix any incompatibility against the new Grafana version.

  • Grafana upgrades are announced with at least 1 week notice;
  • On the scheduled day the Grafana service is upgraded;
  • A temporary Grafana (monit-grafana-old.cern.ch) is enabled running the previous release;
  • The Grafana old endpoint is maintained for the period of 1 month;
  • Users are expected to solve any conflicts with the main Grafana version during this period;
  • Once the period is over the Grafana old endpoint is discontinued;

Plugins

All oficial plugins are maintained by the MONIT team. With every upgrade the team tests and assures all oficial plugins are working.

The MONIT team is however not responsible for the maintenance of unofficial plugins. If an unofficial plugin breaks with an upgrade, it's up to the team that requested the plugin to fix it.

Opensearch

The home page gives direct access to all functions such as:

  • Discover provides direct access to the data. See the "Discovering Data" section.
  • Visualize provides access to the list of all visuazations
  • Dashboard provide access to the lastest open Dashboard
  • Management allows access and modification of all setting of OpenSearch and all objects stored (searches, visuaizations and dashboards)

If you have any doubt, please have a look to the Centralized Elasticsearch docs provided by the IT-ES team.

Tenancy

For the clusters handling private data (monit-timberprivate.cern.ch) access is managed using e-groups. In general for a given tenant, there will be two extra e-groups created:

  • es-timber-\<tenant>_kibana: Provides read only access to people assigned to it
  • es-timber-\<tenant>_kibana_rw: Provide read/write access to people assigned to it

These egroups are generally assigned to the person that requested the tenant creation as owner and to monit-support as an admin egroup, which means that the requester should be able to manage them as required to provide access to the resources.

Data discover

All documents available in OpenSearch are visible in the Discover Panel of the Opensearch portal in the section with the "monit_prod_*" patterns. Selecting one pattern you will choose the data displayed (e.g "monit_prod_fts_ern_complete_* to see the FTS transfers enriched data). You will then be able to see all fields in each document. These fields are those used for all the searches, visualizations and the filters in the Opensearch dashobards in this portal.

HDFS

The monitoring data in HDFS is stored under `/project/monitoring/archive'.

Take a look to the HDFS documentation for tips on how to connect to a client node. Once connected, you can start accessing the data as in the example below:

hadoop fs -ls /project/monitoring/archive/collectd/