jq¶

"jq is a lightweight and flexible command-line JSON processor." - https://jqlang.org

Examples¶

Sort a json file¶

jq -S . foo.json

Select key name with dots¶

The syntax .foo.bar is the same as .["foo"]["bar"], so to select keys that have dots, you would do .annotations["deployment.kubernetes.io/revision"]

Grab first element of an array, and print the value of 'timestamp' of that element.¶

echo '
[
  {
    "foo": "this is foo string",
    "timestamp": "this is the timestamp"
  },
  {
    "second element": "second element value"
  }
]' | jq '.[0].timestamp'

Construct Flickr URLs from an API call¶

curl -s "https://api.flickr.com/services/rest/?"\
"&api_key=0123456789abcdef"\
"&format=json"\
"&method=flickr.photos.getRecent"\
"&nojsoncallback=1" |
  jq -S '
    .photos.photo[] |
    "https://flickr.com/photos/" + .owner + "/" + .id
  '

Use mco to find packages of a certain version on a certain OS¶

This example could be used as an alternative to grep, where only the value of a key/value pair is matched.

mco rpc package status package=apt -j -F lsbdistcodename=trusty |
  jq -c '
    .[] |
    select(.data.ensure == "1.0.1ubuntu2") | {
      version: .data.ensure, hostname: .sender
    }
  '

Print only objects whose name matches a string¶

This example echoes some yaml, uses python to convert it to json, then filters matching data using jq. It could be used as an alternative to grep, where only the key of a key/value pair is matched.

echo "
data:
  - This is a string, not an object, and contains the substrings foo and bar
  - name: foo_name
    value: foo_value
  - name: bar_name
    value: bar_value" |
python -c "import yaml, sys, json; print(json.dumps(yaml.safe_load(sys.stdin)))" |
jq '
  .data[] |
  select(type=="object") |
  select (.name | . and contains("bar_n"))
'

Build a json entry from scratch¶

This uses bash paramber expansion and subshell syntax, and may not work in other shells.

create_json() {
  local user=${1:-${USER}}
  local host=${2:-${HOSTNAME}}
  local more_stuff=${3:-$(uname -a)}
  json=$(
    jq -c -n \
      --arg timestamp  "$(date "+%F %T%z")" \
      --arg host       "${host}" \
      --arg user       "${user}" \
      --arg more_stuff "${more_stuff}" \
      '{
        timestamp:  $timestamp,
        host:       $host,
        user:       $user,
        more_stuff: $more_stuff
      }'
  )
  echo "$json"
}

Render yaml with anchors as json data¶

This example shows how you can use python and jq to view the result of dereferenced yaml anchors, a construct that is not supported by json. This example is less about how to use jq syntaxes, and more about how it can be used to view data that is otherwise difficult to sort through.

echo "
job1: &template
  directory: /tmp
  extra_parameters: nosuid,noatime
  remote_host: 10.1.1.1
  user: nobody
job2:
  <<: *template
  remote_host: 10.2.2.2
job3:
  <<: *template
  remote_host: 10.3.3.3
" |
python -c "import yaml, sys, json; print(json.dumps(yaml.safe_load(sys.stdin)))" |
jq -S .

Select matches, and print a subset of values¶

jq '.[] | select(.data.ensure != "purged") | [.sender,.data.ensure]' $*

Output bare values for use as inputs¶

Use "The value of .key is: \(.key)" to interpolate the value of .key into the string. EG:

$ lsblk -J | jq -cr '.blockdevices[0].children[] | "Partition \(.name) has a size of \(.size)"'
Partition sda1 has a size of 464.8G
Partition sda2 has a size of 1K
Partition sda5 has a size of 975M

Show labels for each locally stored docker SHA¶

docker images --format '{{.ID}}' |
while read -r X ; do
  docker inspect $X |
  jq '.[] | [ .RepoTags, .Config.Labels ]'
done

Sort all JSON contents¶

Be aware that sometimes JSON should not be sorted, as arrays are sometimes expected to maintain their order.

jq -S '. | walk( if type == "array" then sort else . end )'

Or set up a shell alias

alias jqsort="jq -S '. | walk( if type == \"array\" then sort else . end )'"

Store a value as a variable¶

When creating long pipelines, it's useful to be able to store a deep value as a variable. In the following example we store .metadata.namespace as $namespace and .metadata.name as $podname before digging into .status where we would no longer have access to .metadata:

kubectl get pod -A -l k8s-app=kube-dns -o=json |
jq -r '
  .items[] |
  .metadata.namespace as $namespace |
  .metadata.name as $podname |
  .status.containerStatuses[] |
  "\($namespace) \($podname) \(.name) \(.restartCount)"
' |
column -t

The output of this command is something like:

kube-system  kube-dns-66f64447b8-7tzkn  dnsmasq           5
kube-system  kube-dns-66f64447b8-7tzkn  kubedns           0
kube-system  kube-dns-66f64447b8-7tzkn  prometheus-to-sd  0
kube-system  kube-dns-66f64447b8-7tzkn  sidecar           0
kube-system  kube-dns-66f64447b8-b2jsf  dnsmasq           3
kube-system  kube-dns-66f64447b8-b2jsf  kubedns           0
kube-system  kube-dns-66f64447b8-b2jsf  prometheus-to-sd  0
kube-system  kube-dns-66f64447b8-b2jsf  sidecar           0

Default value for missing keys¶

$ echo '{"foo": 1, "bar": 2}' | jq '[.foo // "missing", .bar // "missing", .baz // "missing"]'
[
  1,
  2,
  "missing"
]

Craft json data for use with curl¶

When using curl --data, in-line JSON can get out of hand real quick. To avoid confusing syntaxes, use jq to create a temp file, then reference that file in curl:

JSON_DATA=$(mktemp)
jq -n '{service_account: env.SERVICE_ACCOUNT_ID, secret_key: env.SERVICE_ACCOUNT_KEY}' > "$JSON_DATA"

The above commands create a temporary file with the correct and valid JSON:

$ cat "$JSON_DATA"
{
  "service_account": "whoever@serviceaccount.example.com",
  "secret_key": "abc123"
}

Then reference that in curl:

$ curl -s --request POST --header "content-type: application/json" --data "@${JSON_DATA}" https://httpbin.org/anything
{
  "args": {},
  "data": "{  \"service_account\": \"whoever@serviceaccount.example.com\",  \"secret_key\": \"abc123\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "84",
...

Get the most recently dated blob¶

curl -fssL https://storage.googleapis.com/updates.astronomer.io/astronomer-certified |
jq '.available_releases | sort_by(.release_date)[-1]'

jq¶