Hello.
I'm new to bash script and I'm learning the basics by writing some scripts.
Recently a friend of mine asked me if I could try to write a script to him to automate a couple of processes that uses JSON RPCs.
I'll try to explain in few words the workflow just to contextualize the problem.
There is an app which runs as a deamon. This app is the one which parses the commands and sends them to the remote server. It also receives the response and then sends it to another more userfriendly app which is the app the user interacts with.
Something like this:
daemon starts
user-firendly app send RPC command to daemon
daemon parses it and sends to the remote server
daemon gets the response and parses it to user friendly format
user-friendly app prints the JSON object nicely formatted.
One of these commands takes up to 8 parameters and 2 of these 8 parameters are lists of items.
The format is as follows:
The goal here is to make the items inside those two lists, dynamic, I mean, I load those items from a file, line by line, then num1, num2 and num_n are evaluated in the code and used there. Other params are kind of static.
I already have a version that builds this command correctly with all brackets and double quotes and all special chars in place but the problem is that in the way I did it, JSON parser sees the whole command as a single long string and cannot be that way.
What I did in the code I have, was simply append the static parts of the command before and after of dynamically add the list items with for loops. But as I said, I did it in a way that it's just a huge single string.
JSON RPC expects each parameter as separate strings.
The other factor adding some complexity is what I mentioned of adding items to the lists dynamically. As those items are loaded from a file, it all depends on how many items there are in the file!
Does anyone knows a way of doing this with bash script?
Yes, you can do this in bash , but most people working with JSON are serializing data to be displayed on the web and so, they generally do not program web-based applications using bash .
What I find at unix.com is a lot of people, especially during the start of the school year in September, tend to have these "odd problems" where they are required to use one scripting language or another to do a homework assignment and then they attempt to hide the fact it is homework by coming up with some odd reason to do something that, in the real world, is simply not done this way.
For example, I write a lot of code to process JSON and I do this either in Javascript or in PHP; because 99% of all the code I write is related to web projects; because most people (like me) use JSON APIs to retrieve JSON objects across the network and use those objects in an application, generally written for the web (PHP, Javascript, Python) or maybe a modern gaming engine (which will use C++, Javascript, or C#, for example).
When you ask, "is this possible" my thoughts are why do you ask this? Because just about everything in a computing environment is possible.
The trick is to use the right tools for the job based on the application, the skill sets of the "shop" you are working in, etc.
However, during this time of year, more often than not, the driving requirement we see from new members is the strict and narrow task in a homework problem.
Now, I'm not staying for sure you are posting a vailed homework assignment, but I can tell you that this time of year, every year, we see a lot of homework posts, often written like yours, in a kind of obscure post about "a friend wants me to do this or that" or "I must do this in bash, because only bash is permitted" ; and it turns out that 95% of the time, these are all some forms of classwork where the member is new and posts something in some "disguised post" to get answers to homework assignments.
So, having said that....
What are you actually doing? For what actual project?
Please provide the project details of what you are really doing.
Well, in fact it looks alot like that. However I can assure it's not an homework. I have already graduated earlier this year in February. And I'm not someone that just finished school in the 20's. I'm a full time worker, parents a enthusiast in electronics and programming.
This was supposed to be a simple script to process payments in bitcoins but this last command is messing with my nerves. I'm trying to start a simple project if a friend of mine and I told him I could do this script to automate some tasks.
I would appreciate any help. I'll come around here later.
For simple JSON tasks jq is a versatile tool for querying, manipulating and creating JSON within a Shell script. You'll find a lot of usage examples in the web and of course in the excellent jq documentation.
For anything more complex: Use a decent scripting language of your choice, which supports JSON-processing.
Well, I'm already using it (since yesterday which was when I found it - jq I mean).
So, I'll be a little bit more accurate about the thing.
The app is to process bitcoin payments to several bitcoin addresses. These addresses are in a file. One address per line.
The code I have (I cannot share it right now because I'm at work and the code I have in Github is not updated with the one I have in my personal laptop) reads the file, line by line, counts how many lines it read, then checks how much BTC there is in a certain wallet and divides that value by the number of addresses loaded from the file. This way, the available BTC amount is equally sent to as many addresses that were loaded from the file.
The only thing missing in my script is building this long command.
I'll share the code I have at the moment, later when I get home.
But the way I built it was by preppending the leading special chars to a variable, then appending each address loaded from the file already wrapped in braces, backslash and double quotes. Something like:
load_addresses # this is a function
BTCsendVal=$(printf "%.8f" $(bitcoin-cli -getbalance / $numberAddr | bc -l)) # evaluate the amount of BTC to send to each loaded address
com_params="\"\" \"{"
count=0
for i in $addr_list
count++
do
if [[ $count -eq $numberAddr ]]; then
com_param+="\\\"$i\\\""
else
com_param+="\\\"$i\\\","
done
com_params+="$params_n"
com_params+="params_[n+1]"
#etc etc
........
I might have missed some details in this code. I can't remember all the details accuratelly of the code I have, but this way, the command parameters will be seen as a single very long string!
I have some questions before I can even start playing with 'jq'.
The first parameter/argument is a double quotes with nothing inside it. Docs says its for back compatibility. But is this first parameter also considered a JSON element/or whatever it can be called?
What about the other parameters that are not wrapped in {}s or []s ???? Are those also to be processed by 'jq'?
maybe you can spare the escaping of " if you mix ' (single quotes) and " (double quotes).
simple shell scripting will suffice for what you want to achieve, jq seems not necessary(but hey! Read the docs maybe it's way simpler! I'm not an jq expert, I just use it occasionally)
if you want to know if jq accepts the input, in other words to validate json, just pipe your data into jq . like echo "$data" | jq . . jq will complain if the data is not well formed.
But I cannot do it this way because this "com_params" variable that I set to store the command parameters is seen as a single long string, and if I do it this way, the app that parses the command to JSON complains.
How do you say this can be done with a simple bash script without using 'jq' ? Please give me a hint because I'm not getting there. I'm around this for like 2 or 3 weeks and I can't find a way to make the command to be accepted by the daemon that parses the command!
Note:
This is the code I actually have:
(I can't post links yet)
Search for my nick in Github anf for "TugaRevoBTC" repository if you want to see the code!
Check my github repository named TugaRevoBTC for the entire code.
send_many(){
load_addr_data
count=0
backcptb='""'
min_conf=6
comm1="Periodic payments"
replcbl=true
conf_targ=6
est_mde=CONSERVATIVE
val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))
printf "Value to send: %.8f BTC\n" "$val"
while true
do
echo "Confirm with YES or cancel with NO (caps matter)"
read -r -p '> ' opt
case $opt in
"YES")
obj_init="\""
json_obj='{'
for i in "${addr_arr[@]}"
do
((count++))
json_obj+="\"$i\""
json_obj+=":"
if [ "$count" -lt "$num_addr" ]; then
json_obj+="$val,"
else
json_obj+="$val}"
fi
done
obj_end+='"'
json_final=$(echo $json_obj | jq '.')
printf "bitcoin-cli -testnet sendmany %s %s%s%s %d %s\n" "$backcptb" "$obj_init" "$json_final" "$obj_end" $min_conf "$comm1"
return 1
;;
"NO") echo "Action cancelled!"
return 2
;;
esac
done
}
I just hope bitcoin-cli accepts the command the way I did it here! I haven't tested it yet. Bed time for now!
Any suggestions/corrections are greatly appreciated.
Thanks
Psy
Edited;
I just noticed something I can improve, at least visually. I can just add the leading ' " ' and the trailing ' " ' statically in that printf instead of passing each one as parameters.
@psysc0rpi0n Great! You did it! Your code is good.
One of my goal in programming is trying to create code which is readable well. So I encapsule code into functions, even at the cost (in bash) to make code less resource efficient. Here the function for the creation of a json object:
This function operates on the given parameters($1, $2, ...):
Since it is bash it's a bit of a quoting and eval mess here.
Since you have a special object where all values are equal, I create another function, so the main call stays simple:
mk_json_object_one_val() {
local args=""
local val="$1"
shift
for key in "$@"; do
args+="$key $val "
done
mk_json_object $args # this variable should stay unquoted
}
mydata will contain: { "key1": "value","key2": "value","key3": "value" }
The quoting is different from yours, but escaping the " with \ is not necessary, if you use the mydata variable in quotes this way "$mydata" , since the content of mydata will not be parsed by the shell again.
Note: The function can not cope with values containing whitespace characters.
I'm trying to apply the code to my specific case, and I think I have the same problem.
How can I add as many item pairs as lines that are loaded from the file?
The examples you posted are like static, right? You need to write each item pairs by hand as needed, right?
I tried my version of the code, but somehow I think there might be escaping issues because bitcoin-cli is still finding "json parsing errors" or "wrong amount" (wrong value in item pair), wrong number of arguments, etc.
send_many(){
load_addr_data
count=0
min_conf=6
comm1="Periodic payments"
replcbl=true
conf_targ=6
est_mde=CONSERVATIVE
val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))
printf "Value to send: %.8f BTC\n" "$val"
while true
do
echo "Confirm with YES or cancel with NO (caps matter)"
read -r -p '> ' opt
case $opt in
"YES")
json_obj='{'
for i in "${addr_arr[@]}"
do
((count++))
json_obj+="\"$i\""
json_obj+=":"
if [ "$count" -lt "$num_addr" ]; then
json_obj+="$val,"
else
json_obj+="$val}"
fi
done
count=0
json_lst="["
for i in "${addr_arr[@]}"
do
((count++))
if [ "$count" -lt "$num_addr" ]; then
json_lst+="\"$i\","
else
json_lst+="\"$i\"]"
fi
done
#json_obj_final=$(echo $json_obj | jq '.')
#json_lst_final=$(echo $json_lst | jq '.')
#printf "bitcoin-cli -testnet sendmany %s \"%s\" %d \"%s\" \"%s\" %s %d %s\n" "$backcptb" "$json_obj" $min_conf "$comm1" "$json_lst" "true" $min_conf "$est_mde"
bitcoin-cli -testnet sendmany "" "$json_obj_final" $min_conf "$comm1" "$json_lst_final" "$replcbl" $min_conf "$est_mde"
return 1
;;
"NO") echo "Action cancelled!"
return 2
;;
esac
done
}
I tried to change "", add escaping chars, replacing "" by ' ' but I just can't make it work. I need to make sure that each of the 8 parameters are seen as strings. Param1 one string, param 2 another string, param3 another string and so on! I would like to try to make my version to work and then I could try your version and adapt it to load and create as many item pairs as needed, dynamically!
Thanks
Psy
Edited;
I know you already told me I can avoi escaping double quotes. But as I'm still trying to memorize the special chars that must be escaped and the ones that need no escaping, the way to save command outputs, the way to run commands in a nested way, etc, it's still too many details to just keep in mind at the same time and in a short period of time, I'll be escaping them because as of now that's the way I'm more comfortable with it.
--- Post updated at 11:00 PM ---
I tried something else and things gets even more weird.
The help menu for the command "sendmany" is:
and knowing that the bitcoin-cli command syntax is (from bitcoin core docs):
I decided to try to set all 8 parameters by hand directly on terminal.
After playing a while with incluing/excluding '' and "" and \ I get to a point where the only way 'jq' validates the objects and non-ordered lists is as follows:
Looks like the command is broken on variable $comment.
But to be honest, and as I said I'm still grasping about the escapings, expansions and etc, so I either try all possible combinations of escaping and expansions to see when it works or you guys help me figuring this out!
You mean to add double quotes when I run the command? Because I have already added double quotes when I assigned the variable due to the existence of 2 separate words.
Won't "$comment" expand to " "Periodic payments" "? Or should I remove the double quotes at the assignment of the comment variable?
Please don't full quote forum posts. Please only quote the short passage of text you're relating to. Refrain from full quoting because it adds redundant information and makes the thread harder to read.
regarding quoting #1
text bitcoin-cli -testnet sendmany $backcomp $json_obj $min_conf $comment $json_lst $replc $min_conf $est_mde
It's always a good idea to quote parameters. Else the shell will split your parameters at whitespace. For sure it splits your json-object - which is not what you want. Do it that way:
This(single quotes) '$comment' does not replace $comment with the value contained in $comment. This(double quotes) "$comment" does.
regarding quoting #3
[quote]
You mean to add double quotes when I run the command? Because I have already added double quotes when I assigned the variable due to the existence of 2 separate words. Won't "$comment" expand to ""Periodic payments""? Or should I remove the double quotes at the assignment of the comment variable?
[/quote]
You misunderstood the concept of quoting a bit. If you assign a variable like that: variable="my value" , the content of $variable is not "my value" it is my value (without the quotes). The quotes just protect the content from word splitting. If you leave the quotes out, it would be a complete different command. If you run this command: variable=my value this means assign my to variable for the next program call and call the program named value.
You can only get quotes as data into some variable if you quote it. like this quotes_variable='"' or like this: quotes_variable="\"" .
I recommend to read a bit about quoting to fully understand the concept, as it is really important!
[list]
Please, just let me know if your code is ready to work and it's me that need more study about bash scripting or is it still something about the functions you wrote?
I found the primary problem. I forgot to call one of my functions after introducing your functions to my code!
I don't know if you still willing to give me casual explanations about details in your functions. I'll ask, if you want, you reply, if you don't, no problem.
I'm actually reading/following one basic tutorial about bash script and actually running their examples. I'm also reading the links you posted.
My question is why you added the single quotes twice in the value part of the item of the list in your "make_json_object" funciton here:
like this