Extract distinc sequence of letters

Shell Programming and Scripting
1

Hallo,
I need to extract distinct sequence of letters for example from 136 to 193
Files are quite big, so I would prefer not to use "fold -w1"
Thank you very much

Input file look like this:

       1 cttttacctt catgtgtttt tgcagatatt tgttcataat aacatcttct ttttaagtta
      61 ttaaaatctt ttttaaagtt attaacattt ttttgtcttt tagatcctat atagatccta
      121 aaagatccta aaagatccta aaagatcccc gtttttgtta aagcatatgt gataaggttt
      181 tatagtactt taagattcac tatagtcagt aaaacgttca ctatagtcag taaaacgttc
2

I'm lost.

  1. Please use CODE tags.
  2. From 136 to 193 what? You have shown us lines with a variable length digit string followed by 5 or 6 groups of 10 letters. The only thing you have shown us between 136 and 193 is the digit string 181 marked in red above.
  3. What do you mean by "Files are quite big"? How long will the longest lines be in your input files? What is the maximum size (in bytes) of your input files?
  4. What output are you expecting from the above sample input?
  5. I understand that you don't want to use fold -w1 , but I don't understand why you would say that. I don't see how using fold -w1 to put each character in your input files on a single line would help solve this problem. (But, maybe that is just because I can't figure out what you're trying to do.)
  6. What OS and shell are you using?
3
  1. Input file is in fact one sequence of letters which is separated into lines and numbered (and each lines is separated into sequences of 10 by space). If it would be better I can create one long line from whole input file at first. I want to create shorter sequence from each file such that Nth letter (eg.: third or 136th) of the sequence will be first letter of new sequence and Mth letter (e.g.: 196th) is the last one. (Just imagine all letters are numbered and I want all letters which has numbers greater or equal than 136 and smaller than 196)
    I use zsh but is no problem to use bash; OS: Biolinux (Ubuntu)
  2. All lines are of this length (just length of number is variable), but there can be about 1 000 000 lines in each file
  3. I wanted to a) remove spaces and numbers b) put each character on new line c) select lines containing characters I wanted using awk NR d) join all lines into one
4

Still not clear. If you want to remove the line count, all white space including new lines, and then extract some chars, this might help:

awk '{$1=""; gsub(/ /,"")}1' ORS="" file | dd bs=1 skip=135 count=58 2>/dev/null
tcctaaaagatccccgtttttgttaaagcatatgtgataaggttttatagtactttaa

This works for the sample you gave us; not sure how it behaves on large files.

5

You could also try something like:

#!/bin/ksh
IAm=${0##*/}
if [ $# -lt 3 ]
then	printf 'Usage: %s first_char# last_char# file...\n' "$IAm" >&2
	exit 1
fi
first="$1"
last="$2"
shift 2
awk -v fc="$first" -v lc="$last" '
BEGIN {	fl = int((fc - 1) / 60) + 1	# first line # containing data to copy
	ll = int((lc - 1) / 60) + 1	# last line # containing data to copy
	flc1 = fc % 60 ? fc % 60 : 60	# first character # to copy on line fl
	llcl = lc % 60 ? lc % 60 : 60	# last character # to copy on line ll
}
FNR >= fl && FNR <= ll {
	s = $2 $3 $4 $5 $6 $7
	printf("%s%s", substr(s, FNR == fl ? flc1 : 1,
		FNR == ll ? FNR == fl ? llcl - flc1 + 1 : llcl : 60),
		FNR == ll ? "\n" : "")
	if(FNR == ll) nextfile
}' "$@"

I prefer ksh over bash , but this script will work with either shell. This script allows you to specify the starting character number, the last character number, and a list of one or more files to process. It should work fine on any Linux system, but the awk nextfile command is an extension to the standards. If your version of awk does not have nextfile :

  • if you only want to process one file at a time, change nextfile to exit ,
  • otherwise, remove the entire line shown in red (it will still produce correct output, but will run slower; especially on large files). Note that the code shown in blue can be removed as long as this line remains in your code (with either exit or nextfile .)

If someone else reading this thread wants to try this on a Solaris/SunOS system, change awk in the script to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or /usr/bin/nawk .