# Shell script to merge the lines between two patterns and print the entire file

**URL:** <https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335>\
**Category:** Shell Programming and Scripting\
**Created:** [October 22, 2023, 10:20am UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335 "2023-10-22T10:20:38Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kshitij](https://community.unix.com/letter_avatar/kshitij/32/6_449c533f586939b6e5f246632a46421f.png) [@kshitij](https://community.unix.com/u/kshitij)\
**Post date:** [October 22, 2023, 10:20am UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/1 "2023-10-22T10:20:38Z")

</div>

Hi All ,

I am having a CSV file as stated below  
File1

```plaintext
Module Statements,Module,Collection of primitive building blocks and instances of other modules that comprise a network and or instrument. Two types handoff or internal,A redefinition of a module_def shall overwrite a previously defined module_def having the same module_name within the same nameSpace_name.,6.4.5.a),
Module Statements,,,A module_def shall have at least one port_def unless it is the top module and contains an AccessLink statement,6.4.5.b),
Module Statements,,,"The following objects shall have unique names within a module_def:
port_name
? scanRegister_name
? dataRegister_name
? one_hot_scan_group_name
? scanMux_name
? dataMux_name
? clockMux_name
? one_hot_data_group_name
? logicSignal_name
? alias_name",6.4.5.c),
Module Statements,,,"The following objects shall have unique names within a module_def:
instance_name
? scanInterface_name",6.4.5.d),
Module Statements,,,"An inputPort_name shall be one of the following
scanInPort_name
? shiftEnPort_name
? captureEnPort_name
? updateEnPort_name
? dataInPort_name
? selectPort_name
? resetPort_name
? tmsPort_name
? tckPort_name
? clockPort_name
? trstPort_name
? addressPort_name
? writeEnPort_name
? readEnPort_name
",6.4.5.e),
Module Statements,,,"An outputPort_name shall be one of the following:
? scanOutPort_name
? dataOutPort_name
? toShiftEnPort_name
? toUpdateEnPort_name
? toCaptureEnPort_name
? toSelectPort_name
? toResetPort_name
? toTckPort_name
? toTmsPort_name
? toClockPort_name
? toTrstPort_name
? toIRSelectPort_name",6.4.5.f),
Module Statements,,,Multiple inputPort_name port functions that are of type vector_id may share the same SCALAR_ID as long as the indexes form a contiguous range and every index range is of the same either ascending or descending type.,6.4.5.g),
Module Statements,,,Multiple outputPort_name port functions that are of type vector_id may share the same SCALAR_ID as long as the indexes form a contiguous range and every index range is of the same either ascending or descending type.,6.4.5.h),

```

Output file : Search patterns  
Pattern1 : Module Statements  
Pattern 2 : 6.\*

Output file

```plaintext
Module Statements,Module,Collection of primitive building blocks and instances of other modules that comprise a network and or instrument. Two types handoff or internal,A redefinition of a module_def shall overwrite a previously defined module_def having the same module_name within the same nameSpace_name.,6.4.5.a),
Module Statements,,,A module_def shall have at least one port_def unless it is the top module and contains an AccessLink statement,6.4.5.b),
Module Statements,,,"The following objects shall have unique names within a module_def: port_name ? scanRegister_name ? dataRegister_name ? one_hot_scan_group_name ? scanMux_name ? dataMux_name ? clockMux_name
? one_hot_data_group_name ? logicSignal_name ? alias_name",6.4.5.c),
Module Statements,,,"The following objects shall have unique names within a module_def: instance_name ? scanInterface_name",6.4.5.d),
Module Statements,,,"An inputPort_name shall be one of the following scanInPort_name ? shiftEnPort_name ? captureEnPort_name ? updateEnPort_name
? dataInPort_name ? selectPort_name ? resetPort_name ? tmsPort_name
? tckPort_name ? clockPort_name ? trstPort_name ? addressPort_name
? writeEnPort_name ? readEnPort_name ",6.4.5.e),
Module Statements,,,"An outputPort_name shall be one of the following:
? scanOutPort_name ? dataOutPort_name ? toShiftEnPort_name ?
toUpdateEnPort_name ? toCaptureEnPort_name ? toSelectPort_name
? toResetPort_name ? toTckPort_name ? toTmsPort_name ? toClockPort_name
? toTrstPort_name ? toIRSelectPort_name",6.4.5.f),
Module Statements,,,Multiple inputPort_name port functions that are of type vector_id may share the same SCALAR_ID as long as the indexes form a contiguous range and every index range is of the same either ascending or descending type.,6.4.5.g),
Module Statements,,,Multiple outputPort_name port functions that are of type vector_id may share the same SCALAR_ID as long as the indexes form a contiguous range and every index range is of the same either ascending or descending type.,6.4.5.h),

```

I tried to use below stated code

`awk '/Module State/{a=1;next}/6.4/{a=0}1' FILE.csv`

Thanks and Regards  
Kshitij Kulshreshtha

---

<div class="post-metadata">

**Author:** ![MadeInGermany](https://community.unix.com/letter_avatar/madeingermany/32/6_449c533f586939b6e5f246632a46421f.png) [@MadeInGermany](https://community.unix.com/u/MadeInGermany)\
**Post date:** [October 22, 2023, 4:09pm UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/2 "2023-10-22T16:09:35Z")

</div>

> [@kshitij](#):
>
> awk '/Module State/{a=1;next}/6.4/{a=0}1' FILE.csv

I think you want to use the `a` as a state variable, and act according to its state.  
First, it must be `6\.4` in a regular expression, because an unescaped dot means "any character".  
The action is missing.  
If it's to append the current line, then it could be

```plaintext
awk 'a{buf=(buf FS $0)} /Module State/{a=1; buf=$0} /6\.4/{a=0; print buf}' FILE.csv

```

I use a buf variable:

- if `a == 1` then add the current line to `buf` with a leading space (append)
- if `Module State` is met then set `a` to `1` and store the line in `buf`
- if `6.4` is met then set `a` to `0` and print `buf`

In this case it's doable without buffering:

```plaintext
awk 'a{printf " %s", $0} /Module State/{a=1; printf "%s", $0} /6\.4/{a=0; printf "\n"}' FILE.csv

```

- if `a == 1` then print the current line with a leading space and no ending newline (append)
- if `Module State` is met then set `a` to `1` and print the current line without a newline
- if `6.4` is met then set `a` to `0` and print an ending newline

I see opening and closing quotes, so maybe you just need my tip

> [@Tip: Remove newlines in quoted fields in a csv](https://community.unix.com/t/tip-remove-newlines-in-quoted-fields-in-a-csv/390809):
>
> A couple of times I had to remove newlines in quoted fields in a csv. (Once because the Excel import failed in certain locales. Was a bug of course.) I did not find a robust script solution on the Internet. So I created the following script with embedded sed code: #!/bin/sh # Remove newlines in quoted fields in a csv input sed ' # Save input in hold buffer h :Q1 # Substitute an opening " s/"// # If success jump to :Q2 tQ2 # Restore input; jump to the end (next cycle) x; b :Q2 # S…

---

<div class="post-metadata">

**Author:** ![kshitij](https://community.unix.com/letter_avatar/kshitij/32/6_449c533f586939b6e5f246632a46421f.png) [@kshitij](https://community.unix.com/u/kshitij)\
**Post date:** [October 22, 2023, 7:23pm UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/3 "2023-10-22T19:23:19Z")

</div>

Thanks a lot for the nice solution , it worked,

---

<div class="post-metadata">

**Author:** ![kshitij](https://community.unix.com/letter_avatar/kshitij/32/6_449c533f586939b6e5f246632a46421f.png) [@kshitij](https://community.unix.com/u/kshitij)\
**Post date:** [October 22, 2023, 7:26pm UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/4 "2023-10-22T19:26:35Z")

</div>

Now I am having a requirement to create perl script for the same issue  
I developed something like this but its not giving me the right solution

Perl script

`#!/usr/bin/perl`

`use strict;`  
`use warnings;`  
`use Data::Dumper;`  
`use feature 'say';`

`my @collect;`  
`my $file = $ARGV[0] or die; `  
`open(my $DATA, '<', $file) or die;`

`while (<$DATA>) {`  
` chomp;`  
` # If we're between our markers...`  
` if (/Module Statements/ .. /6.4*/) {`  
` # At the start marker, empty the array`  
` if (/Module Statements/) {`  
` @collect = ();`  
` # At the end marker, print the array`  
` } elsif (/6.4*/) {`  
` say join ' ', @collect;`  
` # Otherwise, push the line onto the array`  
` } else {`  
` push @collect, $_;`  
` foreach my $m (@collect) {`  
`print $m;`  
`}`  
` }`  
` # Otherwise, just print the line`  
` } else {`  
` say;`  
` }`  
`}`

Thanks and Regards  
Kshitij Kulshreshtha

---

<div class="post-metadata">

**Author:** ![MadeInGermany](https://community.unix.com/letter_avatar/madeingermany/32/6_449c533f586939b6e5f246632a46421f.png) [@MadeInGermany](https://community.unix.com/u/MadeInGermany)\
**Post date:** [October 22, 2023, 8:55pm UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/5 "2023-10-22T20:55:40Z")

</div>

> [@MadeInGermany](#):
>
> it must be `6\.4` in a regular expression

Why do you ignore my hint?

You can directly translate `awk` code to `perl -lne` code. Both loop around the input file.  
But if you prefer an explicit while loop...

---

<div class="post-metadata">

**Author:** ![vgersh99](https://community.unix.com/user_avatar/community.unix.com/vgersh99/32/14851_2.png) [@vgersh99](https://community.unix.com/u/vgersh99)\
**Post date:** [October 22, 2023, 9:13pm UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/6 "2023-10-22T21:13:08Z")

</div>

> [@kshitij](#):
>
> Now I am having a requirement to create perl script for the same issue

I wonder why?  
What's wrong with `awk`?  
But either way, consider [a2p](https://perldoc.perl.org/a2p).

---

<div class="post-metadata">

**Author:** ![MadeInGermany](https://community.unix.com/letter_avatar/madeingermany/32/6_449c533f586939b6e5f246632a46421f.png) [@MadeInGermany](https://community.unix.com/u/MadeInGermany)\
**Post date:** [October 23, 2023, 7:02am UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/7 "2023-10-23T07:02:45Z")

</div>

Didn't know about the a2p. It's written in C not in perl. Anyway, good to know.  
For the exercise, do a manual translation, and you learn awk **and** perl!

Apropos perl,  
I am suprised the `..` range operator works in a while loop.  
For each iteration the `$_` is just the current line; obviously the `..` maintains an invisible state variable.

Regarding your perl code:

- the for loop should not be there.
- it misses printing the `Module Statements` and the `6.4` because it is in an `else` branch.
- it misses printing the `6.4` if on the same line as the `Module Statements`, because it is in an `elsif` branch.

---

<div class="post-metadata">

**Author:** ![MadeInGermany](https://community.unix.com/letter_avatar/madeingermany/32/6_449c533f586939b6e5f246632a46421f.png) [@MadeInGermany](https://community.unix.com/u/MadeInGermany)\
**Post date:** [October 24, 2023, 6:55am UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/9 "2023-10-24T06:55:10Z")

</div>

Your perl code corrected:

```plaintext
use warnings;
use Data::Dumper;
use feature 'say';
 
my @collect;
my $file = $ARGV[0] or die;
open(my $DATA, '<', $file) or die;
 
while (<$DATA>) {
  chomp;
  # If we're between our markers...
  if (/Module Statements/ .. /6\.[0-9]/) {
    # At the start marker, empty the array
    if (/Module Statements/) {
      @collect = ();
    }
    # Always(!) push the line onto the array
    push @collect, $_;
    # At the end marker, print the array
    if (/6\.[0-9]/) {
      say join ' ', @collect;
    }
  # Otherwise, just print the line
  } else {
    say;
  }
}

```

The border case, `Module Statements` and `6.4` are on the same line:  
it clears the array, then adds the line to it, then prints it.

---

<div class="post-metadata">

**Author:** ![system](https://community.unix.com/uploads/default/original/2X/c/c7e9e16a595259edafe88615719d5c5dcda035c2.png) [@system](https://community.unix.com/u/system)\
**Post date:** [August 19, 2024, 6:55am UTC](https://community.unix.com/t/shell-script-to-merge-the-lines-between-two-patterns-and-print-the-entire-file/392335/10 "2024-08-19T06:55:45Z")

</div>

Closed on August 19, 2024, 6:55am UTC

This topic was automatically closed 300 days after the last reply. New replies are no longer allowed.