Building JSON command with bash script

The mentioned line is the trickiest thing of all.

You need to know what that eval is actually for. Eval adds an additional evaluation round(so I get from content of $i which may be the number "3" to the content of the following parameter $4 ). An additional evaluation round means that quotes are removed at the first pass, so I had to add protected quotes, which become normal quotes in the second pass.

Read about eval function in the bash manpage.

---

The question "it does not work, tell me the error" drove me away from giving any further answer, because I suspected lack of even minimal initiative/competence to investigate the problem.

The second question was very specific and showed much more of that initiative/competence, so I'm motivated to answer again.

It's no problem with having low competence level - everyone starts from nothing. But I'm not willing to explain at that level.

Ok, I'll read about eval more. I just know one of its usages which I think it is to append "parameters" to build a command. When I say this, it may also mean to add/append whatever you want to add/append to whatever you want. Append a string to another string, etc.
But this might be a very inaccurate description. Or bad wording choice!

Anyways, I'll also try to build the versions of your functions to create non-ordered JSON lists (["item_1", "item_2", "item_n"].

Thanks
Psy

A short comment on debugging:

To check if a function does what it should do, the easiest method is to simply put the function in a small extra program, call it with some valid test data and just print the result to terminal and check if it matches the expectations, or generally speaking, check if the function succesfully executed the expected actions causing the expected results.

Yeah, I've been trying to do something like that sometimes. I also printf and echo many things to try to see where results go wrong.

I think I made it with the json list. I used your 2 functions and change them to try to build my json list in the following format: ["key_1", "key_2", ... , "key_n"]
Tell me if this can be improved.

mk_json_lst(){
   items=""
   for(( i = 1; i <= ${#addr_arr[@]}; i++))
   do
     [ -z "$items" ] || items+=","
     items+="$(eval echo \'\"\'${addr_arr[$i-1]}\'\"\')"
     echo "items is $items"
   done
   echo "[$items]"
 }

mk_json_lst_one_val(){
   local args=""
   for lst in "$@"
   do
     args+="$lst "
   done
   mk_json_lst $args
 }

--- Post updated 09-25-19 at 12:19 AM ---

Ok, building the json object and json list is ok but still bitcoin-cli is not accepting the $val value for some reason I cannot figure out!

I'm evaluating this variable in this line of code:

val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))

I did it this way because if I used a simpler way I would get a value in the format of:

and not in the format of

which is the normal value I see in bitcoin-cli docs. So I used the first approach to try to convert the format of that number to the format that looked like what is in bitcoin core docs. And probably I might have a problem there. No idea how to fix it or what else can I try.
When I try to run the command with the json obj and the json list built with the script like:

bitcoin-cli -testnet sendmany "" {$pairs} 6 Payments [$items] true 6 CONSERVATIVE

I get the following error from bitcoin core:

Any ideas would be appreciated as I'm out of knowledge and expertise!
Thanks
Psy

Hello Psy,

I suggest for now to only check if the JSON you want to generate is ok and only after having this 100% correct then go on to build the complete program call.

Regarding the json_object, this should be currently ok:

Example test script:

mk_json_object() {

        local pairs=""
        for((i=1;i<=$#;i+=2)) ; do
                [ -z "$pairs" ]  || pairs+=","
                pairs+="$(eval echo \'\"\'\$$i\'\"\': \'\"\'\$$(($i+1))\'\"\')"
        done
        echo "{ $pairs }"
}

mk_json_object_one_val() {

        local args=""
        local val="$1"
        shift
        for key in "$@"; do
                args+="$key $val "
        done
        mk_json_object $args # this variable should stay unquoted
}

mk_json_object_one_val myvalue mykey1 mykey2 mykey3

expected output:

{ "mykey1": "myvalue","mykey2": "myvalue","mykey3": "myvalue" }

You should have the same for the list, to check it's really working as expected.

About your command line:

bitcoin-cli -testnet sendmany "" {$pairs} 6 Payments [$items] true 6 CONSERVATIVE

If that is the final command line before calling the command, it is wrong, because {$pairs} is not a valid JSON object and [$items] is not a valid JSON Array. If this is not the final command, but the command that is to be executed(and evaluated) it is also most likely wrong, because {$pairs} and [$items] are not quoted and will result in wrong data after shell evaluation. Use "{$pairs}" and "[$items]" . Aside from that the so far introduced functions already provide brackets. You probably have double brackets, which is not valid JSON.

Please look at the command and show it like it is executed, like this:

set -x
bitcoin-cli...

So you get the output from bash on the console how the final command really looks like.

Example:

object="$(mk_json_object_one_val myvalue mykey1 mykey2 mykey3)"
set -x
bitcoin-cli "$object" "par2 "par3"
set +x

Output:

[stomp@tu02c7dl log]$ ./testscript

+ bitcoin-cli '{ "mykey1": "myvalue","mykey2": "myvalue","mykey3": "myvalue" }' par2 par3
+ set +x
 

You see in the output the single quotes where the shell set the parameter boundary. The whole object is one parameter, which is exactly as desired.

Ok, I'll do that later when I get home.

I'll evaluate the list and array as suggested using 'jq' and will post results here!
Then I'll proceed to the 'set -x' thing.

Thanks.
Psy

Had a go at simplifying Stomps two functions above. My goal was to get rid of eval, which I achieved for mk_json_object but couldn't achieve this for mk_json_object_one_val:

mk_json_object(){
   local items rest

   printf -v items '"%s": "%s"' "$1" "$2"
   shift 2
   [ $# -gt 1 ] && printf -v rest ',"%s": "%s"' "$@"
   echo "{ $items$rest }"
}

mk_json_object_one_val() {
   local val="$1"
   shift
   eval set -- "$(printf "\"$val\" \"%s\" " "$@")"
   mk_json_object "$@"
}

I'm home now and almost going bed. That's my life. Today free time is about 1h...

I have just tested the json object and array created by the functions using 'jq' tool. I think this is a pretty fair way of testing if json objects are correctly built!
So, I just pipped the 'echo' command at the end of stopms both functions through "jq '.'" like this:

echo "{$pairs}" | jq '.'

Result is:

{
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC": "0.079319635",
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ": "0.079319635"
}

Which I think it's perfect.

and

echo "[$items]" | jq '.'

result is:

[
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC",
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"
]

which I think its also perfect.

Now, according to the examples from bitcoin core:

> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}" 1 "" "[\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\",\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\"]"

So the functions outputs are not matching exactly the docs example. The thing is that the docs example wouldn't also pass in a json checker tool like jsonlint. This said,I think the backslashes escaping the inner double quotes of each address can be removed.
Example:

> bitcoin-cli sendmany "" "{"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02}" 1 "" "["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]"

Now, there is a question that might be asked at this point. Will these backslashes be handled by bitcoin core and removed later to pass the command to the RPC server? Are they really needed? I would take a chance in saying they are not needed.
The same for the double quotes outside the {} and the []. But I think I can try these to some extent. I can try to issue the command manually in terminal and see if bitcoin core complains or it it runs the command smoothly.

The other thing I think I need to change is the value of each key because as they are numbers (floating point) they probably don't need to be wrapped around with double quotes. At least the example from the docs have no double quotes around the key values. So the command will become:

> bitcoin-cli sendmany "" {"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02} 1 "" ["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]

I think this might be enough to bitcoin core run the command smoothly!

Summarizing, regarding the validity of json objects built by the functions, I think they are perfect! However, I'll test the command with the changes I said. No backslashes, no double quotes around key values and no outer double quotes wrapping {} and [] around.

In a couple of minutes I'll update this post!

About the improved stomp functions, I'll leave them for later after I have this actual version running and for after figuring out all tricks to make the command to work.

Update:
ok, so I tested 2 versions of the command directly in terminal and I typed in the command myself by hand (no scripts)

Attempt 1 - no backslashes and no outter double quotes wrapping {} and [] around and no double quotes around key values(the floating point numbers):

bitcoin-cli -testnet sendmany "" {"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC":0.079319635, "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ":0.079319635} 6 "Payments" ["n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC", "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"] true 6 CONSERVATIVE

Result:

error: Error parsing JSON:{n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC:0.079319635,

Attempt 2 - re-added back the outter double quotes wrapping {} and []. Still no backslashes and no double quotes wrapping around key values

bitcoin-cli -testnet sendmany "" "{"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC":0.079319635, "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ":0.079319635}" 6 "Payments" "["n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC", "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"]" true 6 CONSERVATIVE

Result:

error: Error parsing JSON:{n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC:0.079319635, 2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ:0.079319635}

Attempt 3 - readded backslashes because it's clear that they are needed.

bitcoin-cli -testnet sendmany "" "{\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\":"0.079319635", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\":"0.079319635"}" 6 "Payments" "[\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\"]" true 6 CONSERVATIVE

Result:

error code: -3
error message:
Invalid amount

Ok, but no problem here because I foun out the problem... I'm just dumb and stupid. That's all. Bitcoin lowest decimal value is 1x10^(-8). Meaning 8 decimal places and in my code I was using "%.9f". Damn stupid of me!

I have already tried the command from the script and yes, I get an error but it's not from the script, is an "insuficient funds error". But it's a bug in my code. It's a matter of math. I mean, let's say I have 0.08444443 BTC to send to 2 addresses. What I'm doing is to divide that amount by 2 which would be 0.042222215. But this value is 9 decimal places long. So I format it to 8 decimal places and those to last decimal places becomes one by rounding up the 15 to 2 (or 20 if we consider 9 decimal places). This in turn generates insufficient amount error because 0.042222 * 2 is more than the total I had originally to send. So I have here a rounding problem I need to fix.

With some more mucking around, I now have a version without eval that also escapes the reserved JSON characters:

#!/bin/bash
mk_json_object(){
   local items rest

   printf -v items '  "%s": "%s"' $(json_quote "$1" "$2")
   shift 2
   [ $# -gt 1 ] && printf -v rest ',\n  "%s": "%s"' $(json_quote "$@")
   printf "{\n%s\n}\n" "$items$rest"
}

mk_json_lst(){
   local list rest
   printf -v list '  "%s"' $(json_quote "$1")
   shift
   [ $# -gt 0 ] && printf -v rest ',\n  "%s"' $(json_quote "$@")
   printf "[\n%s\n]\n" "$list$rest"
}

json_quote() {
    while [ $# -gt 0 ]
    do
        val=${1//\\/\\\\}
        val=${val//\"/\\\"}
        val=${val//$'\n'/\\\n}
        val=${val//$'\t'/\\\t}
        val=${val//$'\r'/\\\r}
        val=${val//$'\f'/\\\f}
        val=${val//$'\b'/\\\b}
        printf "%s" "$val"
        shift
        [ $# -gt 0 ] && printf " "
    done
}

mk_json_object_one_val() {
        local key=$1
        while [ $# -gt 1 ]
        do
            shift
            args=( "${args[@]}" "$key" "$1" )
        done
        mk_json_object "${args[@]}"
}

mk_json_object_one_val '$(uname)' "key
1" '"test"' more$'\r'
mk_json_lst test$'\b' key-2

output:

{
  "$(uname)": "key\n1",
  "$(uname)": "\"test\"",
  "$(uname)": "more\r"
}
[
  "test\b",
  "key-2"
]

Note there are some extra space and newline characters added to the output in mk_json_object() and mk_json_lst() that you many want to remove if you a building variables to send as paramters to another program.

--- Post updated at 09:50 AM ---

These escaped quotes are simply to protect them from the shell the actual argument values that will react the bitcoin-cli process will be

1:sendmany
2:
3:{"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02}
4:1
5:
6:["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]

But I still need to understand something else. But it's something related with the way bitcoin core processes the command.

Stomp functions don't add any backslashes and the script don't return any script related errors. The error returned is from bitcoin core which complains about insufficient funds. However, when I tried to run the commands above manually without the backslashes, bitcoin core complained about json parsing error. So, why running the command by hand with no backslashes, produces errors and running the command without backslashes using the script, don't produce any errors (related with the script code)????

Tomorrow I need to look into this new issue!

ok, enough for today. Time to sleep.

Thanks to both of you.

The reason for this is that quote removal only occurs on the original input words, not the result of expansions. To demonstrate:

$ echo "Hello, world"
Hello, world
$ VAR='echo "Hello, world"'
$ $VAR
"Hello, world"

It's also worth noting that quote matching is done before any expansion, for example:

$ QT='"'
$ echo "$QT"
"
$ echo $QT
"

If you save this code a pparam(or make it a function in your script) you can use it to debug what the shell is doing to your parameters before they are passed on to scripts/functions:

for((i=1;i<=${#};i++))
do
   echo Param $i: ${!i}
done

$ VAR='"one" "two"'
$ ./pparam $VAR
Param 1: "one"
Param 2: "two"
$ ./pparam "$VAR"
Param 1: "one" "two"
$ ./pparam "one" "two"
Param 1: one
Param 2: two

bitcoin-cli -testnet sendmany "" "{\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\":"0.079319635", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\":"0.079319635"}" 6 "Payments" "[\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\"]" true 6 CONSERVATIVE

Result:

error code: -3
error message:
Invalid amount

I think that is also wrong quoting, because the amounts are not escaped(prepended with slashes). But this causes no errors, because numbers should not be quoted in json. Maybe it will cause an error if you fix the quoting.

In bitcoin core docs, says that the amount can be strings too.

Later after work, I'll look into this.
Thanks
Psy

Ok, I'm going to try your examples. This is not trivial for a starter. I've just been reading but not practicing much, so this is not clear at first sight!

--- Post updated at 10:02 PM ---

I don't fully understand what you mean by:

and

.

I need to read about expansions!

I finally reached my goal...

Some changes in the script in the meantime.

I share the link in github:
TugaRevoBTC/TugaRevoTestnet.sh at master . PsySc0rpi0n/TugaRevoBTC . GitHub

Edited;
But this is not over... Heheh
I'll be doing some changes and I'll be doing tutorials and asking questions here.

Fortunately this stuff is pretty well documented. I'd start with Shell Expansions, it's probably best while reading this to have a shell handy and try out the different expansions/substitutions mentioned with something like the pparm script posted above. Ensure you understand why they shell works the way it does.

For example word splitting occurs before pathname expansion, so when a file containing white space is expanded it wont be split.
Parameter and variable expansion occurs before word splitting so variables with white spaces are split:

$ ls
 jason_expand   pparm  'test with space'
$ ./pparm test*
Param 1: test with space
$ VAR="test with space"
$ ./pparm $VAR
Param 1: test
Param 2: with
Param 3: space
$ VAR=test
$ ./pparm ${VAR}*
<work thru the manual and decide what will happen here before trying it>

Hello peeps...

I've been changing my script in several ways until I'm good with it.

I'm now trying to run a command depending on the value of a variable but this is not working.
The code that is not working is:

[ "$used_net" == "testnet" ] && btc_dec=$(bitcoin-cli -testnet getbalance)\
                             || btc_dec=$(bitcoin-cli getbalance)

I want to run one of those 2 commands depending on $used_net variable value! The thing is that no matter the value of $used_net, the issued command is always the one after '||'. What am I doing wrong?

Thanks
Psy

Your code is correct and works here as you described it should. So I assume the variable $used_net never is set to the correct "testnet" value.

Check the variable value directly before the shown command.

Another variant to run it:

[ "$used_net" == "testnet" ] && btcopt="-testnet" || btcopt=
btc_dec=$(bitcoin-cli $btcopt getbalance)

Ok, I'll try that later today.

You want to explain how bash processes 2 variable atributions within the same line of code? Probably following some rule like left to right or right to left. But what is the return value of a successeful variable attribution (which I think will be the value assigned to 'btc_opt') in case of 'test' go false?

It's nearly the same code as yours: If $used_net is "testnet" set btcopts to "-testnet" otherwise set to empty value. A shorthand if-construct.

The return value of a variable assignment is always 0 - unless the command produces a syntax error.