Skip to content

Elasticsearch

"Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured." - https://www.elastic.co/what-is/elasticsearch

Examples

Dev console

Kibana ships with a dev console available which is useful for accessing the below examples. More documentation about APIs that can be used in the dev console can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html

Interact with elasticsearch over HTTP

The Compact Aligned Text interface is available at something like https://${elasticsearch_host}:9200/_cat/ and has a variety of endpoinds you can inspect over http in a human friendly output.

/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates

Accessing any of these will show columns of data. For example, to see all shards, you can do:

curl -s "https://${elasticsearch_host}:9200/_cat/shards?v=true"

Which will show something like:

index                                        shard  prirep  state    docs    store   ip            node
fluentd.quasaric-spacecraft-0412.2021.10.15  0      r       STARTED  53277   7.6mb   10.32.4.26    example-elasticsearch-data-3
fluentd.quasaric-spacecraft-0412.2021.10.15  0      p       STARTED  53277   7.6mb   10.32.63.204  example-elasticsearch-data-9
fluentd.true-ion-0733.2021.10.16             0      p       STARTED  47771   8.2mb   10.32.78.225  example-elasticsearch-data-11
fluentd.true-ion-0733.2021.10.16             0      r       STARTED  47771   8.2mb   10.32.70.57   example-elasticsearch-data-10
fluentd.desolate-terminator-1537.2021.10.19  0      p       STARTED  31216   5.7mb   10.32.70.57   example-elasticsearch-data-10
fluentd.desolate-terminator-1537.2021.10.19  0      r       STARTED  31216   5.7mb   10.32.63.205  example-elasticsearch-data-6
fluentd.false-perihelion-2673.2021.10.14     0      p       STARTED  144118  19.8mb  10.32.4.26    example-elasticsearch-data-3
fluentd.false-perihelion-2673.2021.10.14     0      r       STARTED  144118  19.8mb  10.32.35.26   example-elasticsearch-data-2

The ?v=true enables column headers. ?help is also available. More documentation is available at the following URLs:

Delete indexes by regex

Assuming the indexes you want to delete all have a common string, and assuming you have local http access to elasticserach (EG: you did sudo -E kubfwd svc -n es-namespace)

curl -fsSL 'http://redacted-elasticsearch:9200/_cat/shards' |
awk '$1 ~ /\.2021\.10\.14$/ {print $1}' |
sort -u |
while read -r index ; do
    curl -X DELETE "http://redacted-elasticsearch:9200/${index}"
done

You could also use this same logic to delete large shards by using bytes=b and filtering on the index size:

curl -fsSL 'http://redacted-elasticsearch:9200/_cat/shards?bytes=b' |
awk '$1 ~ /^fluentd/ && $6 > 7500000 {print}'

Move a large shard to a full node to one that has lots of free data

Assuming you have elasticsearch available on localhost, eg from kubectl -n "$namespace" port-forward svc/elasticsearch 9200:9200,

Find a large shard

curl -s http://localhost:9200/_cat/shards?bytes=b | sort -n -k6 | grep <name of node that is full>

If you have GNU sort installed you can append | gsort -k6 -h to sort by shard size.

Find a node with lots of free space

The following output shows "free_space, hostname"

curl -s http://localhost:9200/_nodes/stats |
    jq -rc '.nodes | to_entries | map([.value.fs.data[].free_in_bytes/1024/1024/1024, .value.name])[] | "\(.[0]) \(.[1])"' |
    column -t |
    sort -n

Move the large shard to the new node

curl -s --location --request POST 'http://localhost:9200/_cluster/reroute' \
--header 'Content-Type: application/json' \
--data-raw '{
    "commands" : [
        {
            "move" : {
                "index" : "<name of shard to move>",
                "shard" : <shard number, probably 0>,
                "from_node" : "<node large shard was on that is low on volume>",
                "to_node" : "<node that has low volume to move shard to>"
            }
        }
    ]
}' | jq .