jq
"jq is a lightweight and flexible command-line JSON processor." - https://stedolan.github.io/jq/
It's worth noting at the outset that jq is having a maintenance crisis. The version 1.6 is currently the newest version and was released in late 2018. There are over 600 issues and 114 pull requests. Although jq is a great software, it may be worth looking at something else. There are projects that reimplement jq in other languages that are more maintained, and there are alternatives like dasel.
Examples
Sort a json file
jq -S . foo.json
Select key name with dots
The syntax .foo.bar is the same as .["foo"]["bar"], so to select keys that have dots, you would do .annotations["deployment.kubernetes.io/revision"]
Grab first element of an array, and print the value of 'timestamp' of that element.
echo '
[
{
"foo": "this is foo string",
"timestamp": "this is the timestamp"
},
{
"second element": "second element value"
}
]' | jq '.[0].timestamp'
Construct Flickr URLs from an API call
curl -s "https://api.flickr.com/services/rest/?"\
"&api_key=0123456789abcdef"\
"&format=json"\
"&method=flickr.photos.getRecent"\
"&nojsoncallback=1" |
jq -S '
.photos.photo[] |
"https://flickr.com/photos/" + .owner + "/" + .id
'
Use mco to find packages of a certain version on a certain OS
This example could be used as an alternative to grep, where only the value of a key/value pair is matched.
mco rpc package status package=apt -j -F lsbdistcodename=trusty |
jq -c '
.[] |
select(.data.ensure == "1.0.1ubuntu2") | {
version: .data.ensure, hostname: .sender
}
'
Print only objects whose name matches a string
This example echoes some yaml, uses python to convert it to json, then filters matching data using jq. It could be used as an alternative to grep, where only the key of a key/value pair is matched.
echo "
data:
- This is a string, not an object, and contains the substrings foo and bar
- name: foo_name
value: foo_value
- name: bar_name
value: bar_value" |
python -c "import yaml, sys, json; print(json.dumps(yaml.safe_load(sys.stdin)))" |
jq '
.data[] |
select(type=="object") |
select (.name | . and contains("bar_n"))
'
Build a json entry from scratch
This uses bash paramber expansion and subshell syntax, and may not work in other shells.
create_json() {
local user=${1:-${USER}}
local host=${2:-${HOSTNAME}}
local more_stuff=${3:-$(uname -a)}
json=$(
jq -c -n \
--arg timestamp "$(date "+%F %T%z")" \
--arg host "${host}" \
--arg user "${user}" \
--arg more_stuff "${more_stuff}" \
'{
timestamp: $timestamp,
host: $host,
user: $user,
more_stuff: $more_stuff
}'
)
echo "$json"
}
Render yaml with anchors as json data
This example shows how you can use python and jq to view the result of dereferenced yaml anchors, a construct that is not supported by json. This example is less about how to use jq syntaxes, and more about how it can be used to view data that is otherwise difficult to sort through.
echo "
job1: &template
directory: /tmp
extra_parameters: nosuid,noatime
remote_host: 10.1.1.1
user: nobody
job2:
<<: *template
remote_host: 10.2.2.2
job3:
<<: *template
remote_host: 10.3.3.3
" |
python -c "import yaml, sys, json; print(json.dumps(yaml.safe_load(sys.stdin)))" |
jq -S .
Select matches, and print a subset of values
jq '.[] | select(.data.ensure != "purged") | [.sender,.data.ensure]' $*
Output bare values for use as inputs
This is a contrived example, the better way to get this info would be awless list instances --format tsv --columns name,privateip,launched
$ awless list instances --format json | jq -r '.[] | "\(.Name) \(.PrivateIP) \(.Launched)"' | column -t
salt-master 172.18.9.48 2015-04-10T21:28:03Z
consul-server-01 172.18.9.116 2015-05-15T06:13:19Z
consul-server-02 172.18.9.117 2015-05-15T06:13:19Z
consul-server-03 172.18.9.118 2015-05-15T06:13:19Z
Show labels for each locally stored docker SHA
docker images --format '{{.ID}}' |
while read -r X ; do
docker inspect $X |
jq '.[] | [ .RepoTags, .Config.Labels ]'
done
Sort all JSON contents
Be aware that sometimes JSON should not be sorted, as arrays are sometimes expected to maintain their order.
jq -S '. | walk( if type == "array" then sort else . end )'
Or set up a shell alias
alias jqsort="jq -S '. | walk( if type == \"array\" then sort else . end )'"
Store a value as a variable
When creaeting long pipelines, it's useful to be able to store a deep value as a variable. In the following example we store .metadata.namespace as $namespace and .metadata.name as $podname before digging into .status where we would no longer have access to .metadata:
kubectl get pod -A -l k8s-app=kube-dns -o=json |
jq -r '
.items[] |
.metadata.namespace as $namespace |
.metadata.name as $podname |
.status.containerStatuses[] |
"\($namespace) \($podname) \(.name) \(.restartCount)"
' |
column -t
The output of this command is something like:
kube-system kube-dns-66f64447b8-7tzkn dnsmasq 5
kube-system kube-dns-66f64447b8-7tzkn kubedns 0
kube-system kube-dns-66f64447b8-7tzkn prometheus-to-sd 0
kube-system kube-dns-66f64447b8-7tzkn sidecar 0
kube-system kube-dns-66f64447b8-b2jsf dnsmasq 3
kube-system kube-dns-66f64447b8-b2jsf kubedns 0
kube-system kube-dns-66f64447b8-b2jsf prometheus-to-sd 0
kube-system kube-dns-66f64447b8-b2jsf sidecar 0
Default value for missing keys
$ echo '{"foo": 1, "bar": 2}' | jq '[.foo // "missing", .bar // "missing", .baz // "missing"]'
[
1,
2,
"missing"
]
Craft json data for use with curl
When using curl --data, in-line JSON can get out of hand real quick. To avoid confusing syntaxes, use jq to create a temp file, then reference that file in curl:
JSON_DATA=$(mktemp)
jq -n '{service_account: env.SERVICE_ACCOUNT_ID, secret_key: env.SERVICE_ACCOUNT_KEY}' > "$JSON_DATA"
The above commands create a temporary file with the correct and valid JSON:
$ cat "$JSON_DATA"
{
"service_account": "whoever@serviceaccount.example.com",
"secret_key": "abc123"
}
Then reference that in curl:
$ curl -s --request POST --header "content-type: application/json" --data "@${JSON_DATA}" https://httpbin.org/anything
{
"args": {},
"data": "{ \"service_account\": \"whoever@serviceaccount.example.com\", \"secret_key\": \"abc123\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "84",
...