how to merge two JSON objects using jq?

Question

I have two file json. I want to append two array of SomeFile2.json to SomeFile1.json as below.

SomeFile1.json

[ { "DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0886ed703de64028a" } ] }, { "DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0561634c4g3b4fa25" } ] } ]

SomeFile2.json

[ { "InstanceId": "i-0886ed703de64028a", "State": "InService" }, { "InstanceId": "i-0561634c4g3b4fa25", "State": "InService" } ]

I want the result as below:

[ { "DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0886ed703de64028a" "State": "InService" } ] }, { "DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0561634c4g3b4fa25" "State": "InService" } ] } ]

I'm processing in bash shell via jq. But, unsuccessful.

What have you tried so far and what error messages do you get? — Stig Perez
– Stig Perez, Commented Oct 10, 2018 at 16:48
I using "jq -s add SomeFile1.txt SomeFile2.txt". But, It not as my desire. — donald.sys
– donald.sys, Commented Oct 10, 2018 at 16:52

peak · Accepted Answer · 2018-10-11 03:01:28Z

Since the contents of the second file are evidently intended to define a mapping from InstanceId to State, let's start by hypothesizing the following invocation of jq:

jq --argfile dict SomeFile2.json -f program.jq SomeFile1.json

Next, let's create a suitable dictionary:

reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d

Now the rest is easy:

map(.Instances |= map(. + {State: $d[.InstanceId]}))

Putting the pieces together in program.jq:

(reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d | map(.Instances |= map(. + {State: $d[.InstanceId]}))

Alternatives

The dictionary as above can be constructed without using reduce, as follows:

($dict | map( {(.InstanceId): .State}) | add) as $d

Another alternative is to use INDEX/2:

(INDEX($dict[]; .InstanceId) | map_values(.State))) as $d

If your jq does not have INDEX/2 you can snarf its def from https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq

glenn jackman · Accepted Answer · 2018-10-10 18:25:47Z

Since I find jq pretty hard, I started in a procedural way: using ruby's json module:

ruby -rjson -e ' states = JSON.parse(File.read(ARGV.shift)).map {|o| [o["InstanceId"], o["State"]]}.to_h data = JSON.parse(File.read(ARGV.shift)) data.each do |obj| obj["Instances"].each do |instance| instance["State"] = states[instance["InstanceId"]] || "unknown" end end puts JSON.pretty_generate data ' SomeFile2.json SomeFile1.json

But we want jq, so after some trial and error, and finding this in the manual: https://stedolan.github.io/jq/manual/#Complexassignments -- (note, I changed the state for one of the instances so I could verify the output better)

$ cat SomeFile2.json [ { "InstanceId": "i-0886ed703de64028a", "State": "InService" }, { "InstanceId": "i-0561634c4g3b4fa25", "State": "NOTInService" } ]

First, extract the states into an object mapping the id to the state:

$ state_map=$( jq -c 'map({"key":.InstanceId, "value":.State}) | from_entries' SomeFile2.json ) $ echo "$state_map" {"i-0886ed703de64028a":"InService","i-0561634c4g3b4fa25":"NOTInService"}

Then, update the instances in the first file:

jq --argjson states "$state_map" '.[].Instances[] |= . + {"State": ($states[.InstanceId] // "unknown")}' SomeFile1.json

[ { "DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0886ed703de64028a", "State": "InService" } ] }, { "DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com", "Instances": [ { "InstanceId": "i-0561634c4g3b4fa25", "State": "NOTInService" } ] } ]

When I run the command # /usr/bin/jq --argjson states "$state_map" '.[].Instances[] |= . + {"State": ($states[.InstanceId] // "unknown")}' SomeFile1.json", the error is as below: /usr/bin/jq: invalid JSON text passed to --argjson Use /usr/bin/jq --help for help with command-line options, or see the jq manpage, or online docs at stedolan.github.io/jq
Did you execute the previous command that sets the states_map variable?
I found this answer very useful. I learned a little more about jq. Thank you.

Collectives™ on Stack Overflow

how to merge two JSON objects using jq?

2 Answers 2

Alternatives

Comments

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Alternatives

Comments

3 Comments

Linked

Related