Treat Command Output as a File

Hi.

Before I've post this question, I have spent hours looking for the solutions but to no avail. Because I think it is possible but I just don't know what is the right keyword to search for.

Ok, basically what I want to achieve is really simple.

It's just that I don't want to write another file because it's merely based on the existing file content.

I have these two files:

$ pg source.list
CSM_CKC_INFO_20140925.csv@89049445
CSM_CKC_INFO_20140926.csv@74182043
CSM_CKC_INFO_20140927.csv@91001466
CSM_CKC_INFO_20140929.csv@454
CSM_CKC_INFO_20140930.csv@95508536
CSM_CKC_INFO_20141001.csv@67984672
CSM_CKC_INFO_20140928.csv@48531796

and

$ pg incomplete.list
CSM_CKC_INFO_20140927.csv
CSM_CKC_INFO_20140929.csv
CSM_CKC_INFO_20140930.csv

So, I just want to list out the missing line from incomplete.list against the modified version of source.list - source.lost

$ pg source.lost
CSM_CKC_INFO_20140925.csv
CSM_CKC_INFO_20140926.csv
CSM_CKC_INFO_20140927.csv
CSM_CKC_INFO_20140929.csv
CSM_CKC_INFO_20140930.csv
CSM_CKC_INFO_20141001.csv
CSM_CKC_INFO_20140928.csv

So, I have this command

grep -vxFf incomplete.list source.lost > missing.list

So is there any chance for me to avoid writing that new file source.lost which si simply based on this command:

cut -c 1-25 source.list > source.lost

Thanks a lot.

Do you mean like this?

cut -c 1-25 source.list | grep -vxFf incomplete.list

Certainly!

Thanks a lot.

Didn't think of passing the it that way.

Anyway, is there any other way? Embrace the command with some brackets for example?

What is I have the 3rd argument to pass in similar way (to be treated as a file)?

Thank you.

Yes what you describe is process substiution and it works like this:

grep -vxFf incomplete.list <(cut -c 1-25 source.list)

Thanks but I got this error:

sh: Syntax error: `(' is not expected.

This is a fairly new feature and is not available in a lot of shells yet. I know it's supported in the bash shell and ksh93.

I see.

Thanks for your info.

But I just read somewhere that it is due to posix limited environment.

How do I check if my system supports it or not?

Because I think it's really cool if I could use my script that way.

Thanks.

---------- Post updated at 11:48 AM ---------- Previous update was at 11:40 AM ----------

I am using Bourne shell I guess.

$ ps
   PID TTY       TIME COMMAND
  1114 pts/te    0:00 telnetd
  1115 pts/te    0:00 sh
 10318 pts/te    0:00 ps

I there anyway that I can do to use this process substitution feature?

Thanks.

---------- Post updated at 11:51 AM ---------- Previous update was at 11:48 AM ----------

I can use

 grep -vxFf complete.list $(cut -c 1-25 source.list)

But it will read the file content

What OS are you on, does your system have /dev/fd/?

If so, you could consider using bash or ksh shells. Basically if the <(command) works your system supports it.

This is not very portable and is typically avoided because of this. However if you know your only going to run the script on known platform(s).

$ cd /dev/fd/
sh: /dev/fd/:  not found.
$ uname -a
HP-UX system1 B.11.31 U ia64 0189138652 unlimited-user license
$ echo $SHELL
/sbin/sh
$

echo $0

$ echo $0
-sh

---------- Post updated at 01:26 PM ---------- Previous update was at 01:22 PM ----------

I've read somewhere that I can use `! <command> ` for Bourne instead.

But the output is not what I expect. So I just would some clarification here.

Does this BASH shell script

grep -vxFf incomplete.list <(cut -c 1-25 source.list

equivalent to this BOURNE shell script

grep -vxFf incomplete.list `! cut -c 1-25 source.list`

Thank you.

For HPUX /sbin/sh is a POSIX-compliant shell. POSIX does not require process substitution, which is what you are asking for. Out of the box 11i HPUX has /bin/ksh which does support it.

This explains it:
Process substitution [Bash Hackers Wiki]

Whether or not it is allowed on a given OS depends on the existence

 /dev/fd

If you get a return from

ls /dev/fd

ex:

$>  ls /dev/fd
0 1 2 3 5 9

then process substitution will work in ksh.

To enable ksh, put a shebang on the first line of a script in the leftmost column:

#!/bin/ksh

No, it is not. The first, although missing a closing parenthesis, is called "process substitution" and offers cut 's results as a stream on stdin for grep .
The second offers cut 's results as a list of parameters (i.e. file names) to grep . The "!" will invert cut 's exit code in sh , it will perform history expansion in bash .

Thanks Jim & Rudic.

Btw, I just noticed that in reality this code should be the other way around actually:

cut -c 1-25 source.list | grep -vxFf complete.list

that is I have to find the missing item from source.list against complete.list

So, is it possible without having to create a new file to get the cut -c 1-25 source.list part?

Thanks.

---------- Post updated at 10:26 AM ---------- Previous update was at 09:23 AM ----------

At the moment, I use this command to avoid multiple lines command:

cut -c 1-25 source.list > source.temp && grep -vxFf source.temp complete.list > missing.list && rm -f source.temp

But the issue here is, the rm -f source.temp is only executed whenever the missing.list is not 0 in size.

Please help.

Thank you.

If that is an issue then try:

cut -c 1-25 source.list > source.temp && grep -vxFf source.temp complete.list > missing.list; [ -e source.temp ] && rm -f source.temp

---------- Post updated at 09:30 PM ---------- Previous update was at 08:49 PM ----------

I wonder if this will produce the same result you are looking for:

awk -F"@" 'NR==FNR {a[$1];next} {if($1 in a){next;}print $1}' incomplete.list source.list

Thanks a lot!!

Yep this awk -F"@" 'NR==FNR {a[$1];next} {if($1 in a){next;}print $1}' source.list complete.list works! And it is so flexible and thus the difference could be done both ways. Thank you so much.

The temp file removal also works. Thanks.

So, could you explain both of the codes? What does the [ -e source.temp ] acts for?

Thanks again.

If was possible to just do:

cut -c 1-25 source.list > source.temp && grep -vxFf source.temp complete.list > missing.list; rm -f source.temp

which will try to delete source.temp regardless of any other previous command, however, that would display an error if it was not found.

[ -e source.temp ] (notice the spaces after [ and before ], very important) is test. (take a look at man test), -e evaluates to true if exists, so if that is true then continue to rm -f, avoiding the error message.

Awk

-F"@" make the field separator an `@', that takes care of the format of source.list, dividing each line into two fields, before the `@' and after the `@'

NR==FNR {a[$1];next} this part is only for the first file, in this case source.list. The build-in variable NR is set to the total number of records found from all files, the build-in variable FNR is only the number of records in the current reading file, so as long as both are the same, we are reading only from the first file, if so perform the instructions on the first set of {a[$1];next} which only keeps an array named `a[name of the field]' and skip the rest of the instructions.

Now, we have the whole first file indexed in memory.

{if($1 in a){next;}print $1}' The rest is just an evaluation for the second file, if any of the records are seen already, do nothing (skip to next), otherwise, display it because is missing.

Thanks a lot @Aia

So in conclusion, there is no way for me to mimic the process substitution by any other means right?

Because...

$ ls /dev/fd
/dev/fd not found

Thanks a lot to all.

Well, there might be. Try

cut -c 1-25 source.list | grep -vxFf - complete.list

The -f - will use stdin as the pattern file.

Thanks.

But I got this error:

$ cut -c 1-25 source.list | grep -vxFf - complete.list
grep: can't open -