Elastic: Limit of total fields has been exceeded

When the fields in a document exceeds the limit of total fields of an index, this error happens.

{
  "index": "$INDEX",
  "type": "_doc",
  "id": "675b5f10-dcd2-11ea-80e8-0a580a8002ca",
  "cause": {
      "type": "illegal_argument_exception",
      "reason": "Limit of total fields [1000] in index [$INDEX] has been exceeded"
  },
  "status": 400
}

This could be fixed by updating index setting:

curl --request PUT \
  --url $ES_URL/$INDEX/_settings \
  --header 'Content-Type: application/json' \
  --data '{ "index.mapping.total_fields.limit": 8192 }'

Elastic: Timed out while waiting for a dynamic mapping update

{
  "index": "$INDEX",
  "type": "_doc",
  "id": "-iS-Lm8B-sHL5u3eAKZr",
  "cause": {
      "type": "mapper_exception",
      "reason": "timed out while waiting for a dynamic mapping update"
  },
  "status": 500
}

This error could be caused by multiple reasons. But if it is simply caused by low performance of the Elastic instance (which is my case), it could be avoided by extend time out length from the default 1 minute. For example, adding the parameter timeout=1h for reindex process:

curl --request POST \
  --url '$ES_URL/_reindex?timeout=1h&wait_for_completion=false' \
  --header 'Content-Type: application/json' \
  --data @payload.json

Grafana: failed to create query: field expansion matches too many fields

When using Elastic as a data source, graph drawing might fail:

failed to create query: field expansion matches too many fields, limit: 1024, got: 1165

This is caused by too low indices.query.bool.max_clause_count value which is default to 1024.

This setting limits the number of clauses a Lucene BooleanQuery can have. The default of 1024 is quite high and should normally be sufficient.

Unfortunately this is a static cluster setting and could only be configured without stopping a node. Here I need to add an environment variable in docker-compose.yml:

version: '2.2'
services:
  es-01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es01
    environment:
      ....
      - indices.query.bool.max_clause_count=8192
      ....
  ....

And then restart the nodes using docker-compose restart for the configuration to take effects. Now Grafana could show the graph.