Handling filenames with spaces

Shell Programming and Scripting
1

I'm trying to handle some files with spaces in their name using "" or \ . Like "file 1" or file\ 1.

My current confusion can be expressed by the following shell script:

#!/bin/bash

touch "file 1" "file 2"
echo -n "ls: " ; ls

echo ---

for file in "file 1" "file 2" ; do 
  echo $file 
done

echo ---
filesToShow=`ls -Q`
echo filesToShow=$filesToShow     #shows filesToShow:"file 1" "file 2" on the screen
echo ---


for file in $filesToShow ; do
   echo $file
done

I create two files whose names have a space in it. Then I list the directory. Files exist.

Then I echo the filenames to the screen using a for-loop with "file 1" "file 2" as the list. That results as expected in two separate lines with the filenames.

Then I list the directory using ls -Q and save the output in the variable filesToShow and echo the variable to the screen. Result: filesToShow="file 1" "file 2"

Then I use another for-loop providing the list via that variable. The result is that the filenames are split into two parts even though they are enclosed in double quotes. Hence I can't do anything with the files like cp or mv. However, cp * .. works, for example.

(I also tried to use ls -b.)

What am I missing?

2

The problem is in

for file in $filesToShow

The unquoted expansion of $filesToShow with the default IFS means the fields will be split by using white space.

What you could do instead is to use an array:

filesToShow=( file\ * )
for file in "${filesToShow[@]}" 
do
  echo "$file"
done

--
An alternative while using strings:

filesToShow=`ls -Q`
oldIFS=$IFS
IFS=$'\n'
for file in $filesToShow ; do
   echo "$file"
done
IFS=$oldIFS
1 Like
3

Both versions worked fine. Why do you write IFS=$'\n' ? It worked for me without the $. But shouldn't now a newline be used to split the fields? (There's no newline in the variable.)

4

You seem to be making this harder than it needs to be and are depending on non-portable features (such as ls -Q or shell arrays). The simple way to process all files in the current directory (no matter what characters are included in the filenames [even leading, trailing, and/or embedded <space>s, <tab>s, and <newline>s]) is

for file in *
do	printf '***%s***\n' "$file"	# or whatever you want to do with each file
done

or, if you just want to process files whose names contain one or more <space> characters:

for file in *" "*
do	printf '***%s***\n' "$file"	# or whatever you want to do with each file
done
1 Like
5

Thanks. But why are shell arrays a non-portable feature? Do not all shells have them. I'm mainly familiar with Bash.

6

Shell arrays are not defined by the POSIX standards and do not appear in all shells (even in all shells based on Bourne shell syntax). The Bourne shell (on which most current shells other than the csh based shells) did not include arrays. A future revision of the POSIX standards may include arrays, but no one has proposed adding that feature yet.

Note also that to declare an indexed array in bash , you use:

declare -a array_name

.
With a 1993 or later Korn shell you declare an indexed array with:

typeset -a array_name

and you declare an associative array with:

typeset -A array_name

.
I think some recent versions of bash also have associative arrays, but the version of bash available on macOS High Sierra version 10.13.6 (the OS I use) does not.

I don't know what syntax other shells that support arrays use when declaring them.

7

I'm using Bash 4.4 on Raspbian and on Kali Linux. Both have associative arrays. They must be declared using

declare -A array_name

. Indexed arrays can be declared by

declare -a array_name

but that's optional. Actually, the Bash Reference Manual (4.4) mentions associative arrays so I thought they (at least the indexed arrays) are a standard these days.

8

$'\n' is a way to denote a hard newline character. When this variable is set before the unquoted variable expansion of filesToShow , the variable is split into fields depending on the newline character.

After the assignment filesToShow=`ls -Q` there are certainly newlines present in the variable.

If you use IFS='\n' then the content of variable is field split on the n character or the backslash character. So if there happen to be no n-characters or \-characters then the variable will be split into a single field, that gets printed in a a single for loop iteration. What is then printed may look the same, but the process is entirely different..

To illustrate:

$ IFS='\n'; for file in $filesToShow ; do   echo "new line: $file"; done
new line: file 1
file 2
file 
new line: umber 1
file 
new line: umber 2
file 
new line: umber 3
$ IFS=$'\n'; for file in $filesToShow ; do   echo "new line: $file"; done
new line: file 1
new line: file 2
new line: file number 1
new line: file number 2
new line: file number 3

--
Note1: $'\n' only works in modern shells is also not a standard feature (like is the case with arrays). A portable alternative would be to assign like this:

IFS="
"

Note2: it is better to use filesToShow=$(ls -Q) instead of the outdated backticks method.

9

IFS=$'\n' is a bash invention.
IFS='\n' is plain wrong.?
Portable is

IFS='
'
10

One thing is special for IFS, if empty it defaults to a newline.
So

IFS=''

or

IFS=

is portable, too.