A $(( expression )) bug?

This is for the big guns...
I have been modifying AudioScope.sh to bring it inline with more current practices.
I hit a bug which IS not present in the original code but was after modification.
Shell check first:-

#!/bin/sh
txt="1234567890"
echo "$(( $txt ))"
echo "$(( ${#txt} - 1 ))"
echo "$(( #txt - 1 ))"

ERROR WINDOW.

$ shellcheck myscript
 
Line 3:
echo "$(( $txt ))"
          ^-- SC2004: $/${} is unnecessary on arithmetic variables.

$ 

Here is a longhand input to the bash version on 'sh' and yes I did have this inside my code, and what a RPITA it was too. (This also fails inside the version of bash for OSX 10.7.5; I don't know about current versions as i don't have them.)

Last login: Wed Sep 28 09:36:29 on ttys000
AMIGA:barrywalker~> sh
AMIGA:barrywalker~> txt="1234567890"
AMIGA:barrywalker~> echo "$(( $txt ))"
1234567890
AMIGA:barrywalker~> echo "$(( ${#txt} - 1 ))"
9
AMIGA:barrywalker~> echo "$(( #txt - 1 ))"
sh: bad substitution: no closing `)' in "$(( #txt - 1 ))"
AMIGA:barrywalker~> _

Either "$/${}" is needed or it is not, which is it?
From now on I will be using $ and ${} as I don't want this aggro again...

The

(c.f. man bash ) is available in recent shells only, not in sh . As you can see, variable expansion is done implicitly, no $ nor ${} necessary. Your message above is informative, not an error.

It fails on ksh too!

Last login: Wed Sep 28 10:21:58 on ttys000
AMIGA:barrywalker~> ksh
AMIGA:uw> txt="1234567890"
AMIGA:uw> echo "$(( ${#txt} - 1 ))"
9
AMIGA:uw> echo "$(( #txt - 1 ))"
ksh:  #txt - 1 : arithmetic syntax error
AMIGA:uw> exit 
AMIGA:barrywalker~> _

---------- Post updated at 11:09 AM ---------- Previous update was at 10:30 AM ----------

And the WRONG answer with zsh!

AMIGA:barrywalker~> zsh
AMIGA:\u\w> zsh --version
zsh 4.3.11 (i386-apple-darwin11.0)
AMIGA:\u\w> txt="1234567890"
AMIGA:\u\w> echo "$(( ${#txt} - 1 ))"
9
AMIGA:\u\w> echo "$(( #txt - 1 ))" 
48
AMIGA:\u\w> _

EDIT:-
This should have been a new post but was added to this one, no idea why!

In an arithmetic evaluation, the value of a variable can be specified by expanding the variable (i.e. $var or ${var} ) or by just giving the variable name (i.e. var ). But the expression that is the numeric evaluation of the length of the string that is the value of a variable must use the expansion of the variable that produces that number (i.e. ${#var} ); you cannot skip the ${ and } when you are using shell variable expansion operators like ${#var} or ${var%%pattern} or ${var:offset:length} .

OK, I'll go with that BUT, and a big but, why does Shell Check echo "$(( #txt - 1 ))" NOT come up with a syntax error knowing from the various scenarios that it does?
Should there not be a warning saying so. Note that in the Shell Check example I was using the worst case sh . Tested on the others gave errors or incorrect results too.
On reading RudiC's post it looks as though it SHOULD work but obviously it doesn't, or I am misinterpreting it?
Anyhow thanks, but I will keep '$/${}' in all cases just to be on the safe side.

There is something in what RudiC quoted in post #2 that says that the string ${#variable_name} inside an arithmetic expansion will undergo parameter expansion resulting in a number that is the number of characters in the string to which $variable_name expands. (I.e., All tokens in the expression undergo parameter and variable expansion, command substitution, and quote removal.)

There is nothing in what RudiC quoted in post #2 that says that the string #variable_name inside an arithmetic expansion should have any meaning (and I don't know of any shells where it does).

There is something in what RudiC quoted in post #2 that says that the strings $variable_name and ${variable_name} inside an arithmetic expansion will expand to the string assigned to that variable. (I.e., All tokens in the expression undergo parameter and variable expansion, command substitution, and quote removal.)

There is nothing in what RudiC quoted in post #2 that says that the string variable_name inside an arithmetic expansion will expand to the string that has been assigned to that variable. But, in addition to what RudiC quoted, the latest version of the standard also states the following requirement for arithmetic expansions:

and I don't know of any shell that performs arithmetic expansions that does not also provide this shortcut.

I make absolutely no comment as to why shellcheck seems to think that $((#variable_name)) and $(( #variable_name - 1 )) are not errors. You would have to ask the developers or maintainers of that code why they do not flag these constructs as errors in arithmetic expressions in shell scripts.

Thanks Don...

I will attempt to get in touch with the Shell Check developers.
Consider this thread solved.

EDIT:-
As an addendum...
I have left a message on their bug report board...

Thanks for the heads up Don.

Arithmetic expansion is part of the POSIX specification. So it depends on what you mean by sh . If it means the classic (non-POSIX) Bourne shell, then yes. If it means a POSIX shell (which is its meaning on almost any POSIX compliant system, /usr/xpg4/bin/sh on Solaris), then no.

You are right, Scrutinizer, it is even available in my dash ...

Hi.

Here is my take on this. Given a modified script:

#!/bin/sh
txt="1234567890"
echo "$(( $txt ))"
echo "$(( txt ))"
echo "$(( ${#txt} ))"
echo "$(( ${#txt} - 1 ))"
echo "$(( #txt - 1 ))"

On my system:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie)

The shell sh is /bin/sh: symbolic link to dash
When we run shellcheck on the script, we get:

$ shellcheck s1

In s1 line 3:
echo "$(( $txt ))"
          ^-- SC2004: $ on variables in (( )) is unnecessary.

So far so good, the warning is correct.

If we then run the script, we get:

$ ./s1
1234567890
1234567890
10
9
./s1: 7: ./s1: arithmetic expression: expecting primary: " #txt - 1 "

So we have fed a syntactically incorrect line into the shell, which has diagnosed it.

If we complain that shellcheck is not diagnosing it, we are on a slippery slope. We would have to re-create the shell mechanism into shellcheck in order to diagnose syntax errors.

The solution, it seems to me, is to get rid of syntax errors, then ask shellcheck to look over the code.

I think we'll all be interested in what the developers say.

Best wishes ... cheers, drl

Well I got 2 emails back from [koalaman/shellcheck]:-
"""
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
$(( expression )) error? * Issue #739 * koalaman/shellcheck * GitHub
"""

First:-
"Closed #739."

So bug report #739 was looked at.

Now second:-
"""
You're right, but shellcheck also does not produce a warning for `echo "$(( ${#txt} - 1 ))"`.

It only produces a warning where the `$` is unnecessary, on line 3.

I've updated the wiki to mention this explicitly.
"""

/Me shrugs.

What is the point of just updating the wiki when it actually does do RED sentences when it is not able to parse a line; after all it is a syntax error...
So in essence it was a waste of time, a lesson learnt for us all...

Just had a first look at shellcheck.
It looks like it makes suggestions in addition to the real faults that are found by sh -nx

Hi all...
Been messing around and guess what...
ShellCheck:-

#!/bin/sh
txt="12345"
echo "$(( # == txt ))"
echo "$(( $# == txt ))"
echo "$(( $# == $txt ))"
echo "$(( # == $txt ))"
echo "$(( # = txt ))"
echo "$(( $# = txt ))"
echo "$(( $# = $txt ))"
echo "$(( # = $txt ))"

ShellCheck reaults:-

$ shellcheck myscript
 
Line 5:
echo "$(( $# == $txt ))"
                ^-- SC2004: $/${} is unnecessary on arithmetic variables.
 
Line 6:
echo "$(( # == $txt ))"
               ^-- SC2004: $/${} is unnecessary on arithmetic variables.
 
Line 9:
echo "$(( $# = $txt ))"
               ^-- SC2004: $/${} is unnecessary on arithmetic variables.
 
Line 10:
echo "$(( # = $txt ))"
              ^-- SC2004: $/${} is unnecessary on arithmetic variables.

$ 

Manually using DASH.

Last login: Sun Oct  2 10:30:26 on ttys000
AMIGA:barrywalker~> /usr/local/bin/dash
AMIGA:\u\w> txt="123456"
AMIGA:\u\w> echo "$(( # == txt ))"
/usr/local/bin/dash: 2: arithmetic expression: expecting primary: " # == txt "
AMIGA:\u\w> echo "$(( $# == txt ))"
0
AMIGA:\u\w> echo "$(( $# == $txt ))"
0
AMIGA:\u\w> echo "$(( # == $txt ))"
/usr/local/bin/dash: 5: arithmetic expression: expecting primary: " # == 123456 "
AMIGA:\u\w> echo "$(( # = txt ))"
/usr/local/bin/dash: 6: arithmetic expression: expecting primary: " # = txt "
AMIGA:\u\w> echo "$(( $# = txt ))"
/usr/local/bin/dash: 7: arithmetic expression: expecting EOF: " 0 = txt "
AMIGA:\u\w> echo "$(( $# = $txt ))"
/usr/local/bin/dash: 8: arithmetic expression: expecting EOF: " 0 = 123456 "
AMIGA:\u\w> echo "$(( # = $txt ))"
/usr/local/bin/dash: 9: arithmetic expression: expecting primary: " # = 123456 "
AMIGA:\u\w> exit
AMIGA:barrywalker~> _

I have posted a report and it is public here so I am off air with this now...
I am well aware of the errors now and this site has a thread on it...

Hi.

It turns out that zsh has a specific meaning for #txt :

       expression of the form `#foo' gives the value of the first character of
       the  contents  of the parameter foo.

excerpt from man zshall , section ARITHMETIC EVALUATION

For example:

#!/bin/zsh

txt="1234567890"
echo "$(( #txt - 1 ))"

# txt="1234567890"
txt="1"
echo "$(( #txt - 1 ))"
txt="2"
echo "$(( #txt - 1 ))"

producing:

$ ./z4
48
48
49

On a system:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
zsh 5.0.7

The zsh and man zsh both are incredibly complicated.

Best wishes ... cheers, drl

Well here is the code that shellcheck checked:-

Last login: Sun Oct  2 11:18:10 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> cat BUG1.sh
#!/bin/sh
txt="12345"
echo "$(( # == txt ))"
echo "$(( $# == txt ))"
echo "$(( $# == $txt ))"
echo "$(( # == $txt ))"
echo "$(( # = txt ))"
echo "$(( $# = txt ))"
echo "$(( $# = $txt ))"
echo "$(( # = $txt ))"
AMIGA:barrywalker~/Desktop/Code/Shell> sh -n BUG1.sh
AMIGA:barrywalker~/Desktop/Code/Shell> bash -n BUG1.sh
AMIGA:barrywalker~/Desktop/Code/Shell> _

And I decided to [ba]sh -n BUG1.sh and it says it is syntactically correct.
Again /Me shrugs...
Have I discovered a general SHELL bug, although esoteric is STILL a bug?
Are these arithmetic related ERRORs capable of being hacked from the outside?
(I have no idea as I am a mere amateur.)
Why in fact does bash -n NOT pick it up on at least my OSX 10.11.6 version of 'bash'?

Hi.

The definition for the syntax check is:

              -n      Read commands but do not execute them.  This may be used
                      to  check  a  shell  script  for syntax errors.

Here is how I see this.

If no code is executed, then variables are not defined. If there is no execution, then commands like [ are not executed, and there can be no check for the internal syntax of expressions after [ -- which is the same as command test .

I think the syntax check of the -n option can only check structural syntax. Below are some examples that I thought of, and there are probably more.

So to debug a code, I think a preliminary run with bash -n can be used to help eliminate gross errors in structure, missing elements like then or do , unmatched parens and quotes, etc. Then the script can be run to get rid of the internal syntax errors, like those mentioned above with test and friends, and finally shellcheck to look at other items.

So check, run, check would be how I would approach this if I were to write long sections of code (my longest shell script is just under 800 lines, but most are far smaller). There might be instances where check, check, run might be more useful. It depends on the code and coder.

Best wishes ... cheers, drl

Here are some samples that bash -n can catch as noted by script run-all :

#!/bin/bash

echo
echo " File contents:"
head -20 se-*

echo
echo " Results of syntax check, noexec:"
for script in se-*
do
  echo
  echo " Results for $script:"
  bash -n $script
done

exit 0

producing:

$ ./run-all 

 File contents:
==> se-1 <==
#!/bin/bash

if [ something ]
then
  :
else
  :
fi

if [ other ]
  :
else
  :
fi

==> se-2 <==
#!/bin/bash

echo "$(( 0 == 1 ))"

echo "$(( 0 == 1)"

==> se-3 <==
#!/bin/bash

for xxx
do
  :
done

for yyy
do
  :
else
done

==> se-4 <==
#!/bin/bash

echo 1 | cat

| echo 2

==> se-5 <==
#!/bin/bash

echo 1 | cat

& echo 2

==> se-6 <==
#!/bin/bash

[ 0 ] && echo hi

[ 1 ] || echo lo

[ 0 ] &&

==> se-7 <==
#!/bin/bash

echo {1..10}

a() { echo hi; }

b() { echo lo } # omitted semi-colon before }

 Results of syntax check, noexec:

 Results for se-1:
se-1: line 12: syntax error near unexpected token `else'
se-1: line 12: `else'

 Results for se-2:
se-2: line 5: unexpected EOF while looking for matching `"'
se-2: line 6: syntax error: unexpected end of file

 Results for se-3:
se-3: line 11: syntax error near unexpected token `else'
se-3: line 11: `else'

 Results for se-4:
se-4: line 5: syntax error near unexpected token `|'
se-4: line 5: `| echo 2'

 Results for se-5:
se-5: line 5: syntax error near unexpected token `&'
se-5: line 5: `& echo 2'

 Results for se-6:
se-6: line 8: syntax error: unexpected end of file

 Results for se-7:
se-7: line 8: syntax error: unexpected end of file

Hi drl...
How about this then:-
$[ ( expression ) ] is runtime identical to $(( expression )) , yes...
If the error reports are different then does that imply that they might not be runtime identical?
If not then explain why some of these errors are different?
These old arithmetic expressions were what brought about this thread in the first place.

Last login: Mon Oct  3 14:53:41 on console
AMIGA:barrywalker~> txt="12345"
AMIGA:barrywalker~> echo "$(( # = txt ))"
-bash: bad substitution: no closing `)' in "$(( # = txt ))"
AMIGA:barrywalker~> echo "$[ ( # = txt ) ]"
-bash: ( # = txt ) : syntax error: operand expected (error token is "# = txt ) ")
AMIGA:barrywalker~> echo "$(( # = $txt ))"
-bash: bad substitution: no closing `)' in "$(( # = $txt ))"
AMIGA:barrywalker~> echo "$[ ( # = $txt ) ]"
-bash: ( # = 12345 ) : syntax error: operand expected (error token is "# = 12345 ) ")
AMIGA:barrywalker~> echo "$(( $# = $txt ))"
-bash: 0 = 12345 : attempted assignment to non-variable (error token is "= 12345 ")
AMIGA:barrywalker~> echo "$[ ( $# = $txt ) ]"
-bash: ( 0 = 12345 ) : attempted assignment to non-variable (error token is "= 12345 ) ")
AMIGA:barrywalker~> echo "$(( $# = txt ))"
-bash: 0 = txt : attempted assignment to non-variable (error token is "= txt ")
AMIGA:barrywalker~> echo "$[ ( $# = txt ) ]"
-bash: ( 0 = txt ) : attempted assignment to non-variable (error token is "= txt ) ")
AMIGA:barrywalker~> bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
Copyright (C) 2007 Free Software Foundation, Inc.
AMIGA:barrywalker~> _

At this pont I am going to bow out gracefully and keep a watchful eye on this thread.

Hi, Barry.

On a system like:

OS, ker|rel, machine: Apple/BSD, Darwin 9.8.0, Power Macintosh
Distribution        : Mac OS X 10.5.8 (leopard, workstation)
bash GNU bash 3.2.17

there is no mention of $[ in the man page. There might be in your version 57.

Interactive results there:

$ echo "$(( # = txt ))"
-bash: bad substitution: no closing `)' in "$(( # = txt ))"
$ echo "$[ ( # = txt ) ]"
-bash: ( # = txt ) : syntax error: operand expected (error token is "# = txt ) ")

The man page for a modern bash 4.3.30 notes:

       The  old  format  $[expression]  is  deprecated  and will be removed in
       upcoming versions of bash.

The man page for modern dash 0.5.7-4+b1 does not mention $[ at all.

For this code:

#!/bin/bash

# The  old  format  $[expression]  is  deprecated  and will be removed in
# upcoming versions of bash.
# -- man bash

x=12345

echo "$(( x == txt ))"

echo "$[ x == txt ]"

echo "$[ x == 12345 ]"

echo "$[ ( # = txt ) ]"

Both dash -n and bash -n return nothing on that file.

Executing the code first with bash on that file produces:

$ bash se-8
0
0
1
se-8: line 15: ( # = txt ) : syntax error: operand expected (error token is "# = txt ) ")

then dash:

$ dash se-8
0
$[ x == txt ]
$[ x == 12345 ]
$[ ( # = txt ) ]

So my conclusion is that $[ is not portable, it is not equivalent to $(( , and it will be removed from future versions of bash , as has been done with dash currently.

Judging from the output comparing $(( to $[ , I would say that they have either: a) differing syntax requirements, or b) old code for $[ was not updated. I tend to think the former.

So I would say, in addition to my suggested sequence in a previous post for bashcheck / bashrun / shellcheck, I would advise changing $[ to $((

If you are writing to the lowest common denominator, say sh (which may link to dash as it does in Debian), then you might want look at script checkbashisms to identify bash-only constructs.

Yeah, me too

Best wishes ... cheers, drl