Using HTTP

Before starting

The first step is to open a SNOW Request providing these details:

which service logs you wish to send and if possible an example
the expected daily data volume and data rate
how you plan to access your logs: hdfs files, kafka stream, opensearch dashboard, etc.

Important

Please respect the agreed data volume/rate. We have limited quota in all backends used by MONIT and the usage is monitored. In case a significant change is required please contact us in advance.

Your data must be represented as a valid JSON object with the following fields (as strings):

(mandatory) producer: used to name your data set, only one value allowed
(mandatory) type: used to classify your data set, you can define multiple values
(optional) type_prefix: used to categorise your metrics, possible values are raw|agg|enr
(optional) timestamp: used to indicate the event submission time
(optional) host: used to add extra information about the node submitting your data
(optional) _id: if you wish to set your own ID, we assign one random ID by default
(optional) raw: in case logs are written to Elasticsearch, this field gets indexed

To understand how to access your data, please refer to the Data Access section.

Sending data

Logs can be sent to the HTTP endpoint listening in http://monit-logs.cern.ch:10012 (Please note this endpoint is getting deprecated).

For better isolation/security we are providing endpoints per producer, so you will need to send your logs to https://monit-logs.cern.ch:10013/\<producer\> instead, you will be provided with a password to do so.

We are not opinionated on the best way to send your data, and we can't offer much support on how to configure the tool you select. For your information, many Service Managers are using Logstash and the Logstash Puppet module.

The JSON document that you send to us must contain all the mandatory values specified above. A valid example is the following:

{
  "producer": "autoinstall",
  "type": "dbmetric", 
  "valueA": 0,
  "raw": "myfulllogmessagethatiwanttosearchinelasticsearch", 
  "hostname": "hostname.cern.ch",
  "data": {
    "valueB": 1,
    "type": "iwantthistobedata"
  }
}

Please pay attention to the following:

all timestamps must be in UTC milliseconds or seconds (with or without subdecimal part)
use double quotes and not single quote (not valid in JSON)
send multiple documents in the same batch grouping them in a JSON array.
make sure your document fits in one line as we don't support multi-line JSON.
anything that is considered metadata for the infrastructure will be promoted to the metadata field in the produced JSON, and the rest will be put inside data
only UTF-8 charset is accepted and it must be explicitly specified into the Content-Type entity of the request if you are sending the logs using HTTP

Flume and Java > 8

Usually, people using Flume to send data will have morphlines configured with an import like:

importCommands : ["org.kitesdk.**"]

This won't work from Java9 onwards and you will need to adapt your morphlines with proper imports of the needed classes, although this depends a lot on what the morphline itself is doing, you can find an example in the punch hostgroup.

Clients

A number of projects started by different IT teams can help you to send your logs directly into the monitoring infrastructure.

Python This has been done by the IT-CM-LCS section. The code can be found here.

C# This has been done by the IT-CDA-IC section. The code can be found here.

Data access

Logs send to MONIT are stored in the monit-timber OpenSearch cluster and can be queried via Grafana or Opensearch. Private logs are instead present in monit-timberprivate Opensearch cluster and can be queried from Opensearch or Grafana respectively. Please note timberprivate uses tenants for isolation.

Opensearch dynamically sets the type of all fields is the logs. Please refer to this page for more details on this mapping. In case you are not happy with the default assigmnet please contact the MONIT team.