Awk add filter value to the result

Hello,

A couple years ago i posted the thread that i linked below. In this thread i had a quite slow filtering with awk and thanks to the help of the community and more specifically RudiC there was a script made that was quite alot faster then the original one (factor 21 if i recall correctly).

This script always worked flawlessly but now we got to the point that we need a (hopefully) very small addition to this script to be able to keep using it without any problems.

The problem we are facing is that we have no idea on which filter the script triggers.

awk -F";" '
NR==FNR                 {SRCH=SRCH DL tolower($0)
                         DL = "|"
                         next
                        }
tolower($0) ~ SRCH      {print > "'"removed_woord_tmp.csv"'"
                         next
                        }

                        {print > "'"filtered_winnaar_2.csv"'"
                        }
' prijslijst_filter.csv lowercase_winnaar.csv

I know the addition needs to happen at line
tolower($0) ~ SRCH {print > "'"removed_woord_tmp.csv"'"
because that is where it writes the hits it got from reading the unfiltered file (lowercase_winnaar.csv).

i tried to add SRCH to the print statement but the script went complete bezerker and gave me a file which is about 150x as big as the original file that it spews out. Then i tried $0 (gave me the original result), FNR or NR (only gave a linenumbers) and even DL (which gave me only |).

Does anyone know how i can get the original result with the filtervalue like
print $0, filtervalue > "'"removed_woord_tmp.csv"'"
so i know which value triggered the removal of that line?

Not quite clear.
Pls post samples of the two input files reduced to the bare minimum (i.e. 2 - 3 items in file1, and a handful of matching and non-matching lines in file1) and desired output(s) from those two samples.

prijslijst_filter.csv (everything that needs to be removed)

poly voyager
poly sync

lowercase_winnaar.csv (unfiltered file)

E0017229173866;14.9700;100;0.00;104067;consumer headset zubehör ;104067;poly voyager legend ladebox schwarz;89036-101;poly;0017229173866;15;1
E0017229174153;96.8400;10;0.00;315347;headset schnurgebunden;315347;poly bt headset voyager 4310 uc mono usb-a;218470-01;poly;0017229174153;15;1
E0017229174177;103.5900;20;0.00;1779011; headsets;1779011;poly voyager 4310 uc usb-a mono cs | with charge stand;218471-01;plantronics;0017229174177;4;1
E0017229174184;106.0200;10;0.00;315350;headset schnurgebunden;315350;poly bt headset voyager 4310 uc mono usb-a teams mit stand;218471-02;poly;0017229174184;15;1
E0017229174306;114.5800;25;0.00;315356;headset schnurgebunden;315356;poly bt headset voyager 4320 uc stereo usb-a mit stand;218476-01;poly;0017229174306;15;1
E0017229174382;114.1800;10;0.00;315362;headset schnurgebunden;315362;poly bt headset voyager 4320 uc stereo usb-c teams mit stand;218479-02;poly;0017229174382;15;1
E0017229175082;160.8000;25;0.00;309822;headset pc´s ;309822;poly sync 40+ ms usb-a bt600ww;218764-01;poly;0017229175082;15;1
E0017229175099;159.8000;100;0.00;309823;headset pc´s ;309823;poly sync 40+ usb-a bt600ww;218765-01;poly;0017229175099;15;1
E0021363810006;30.5000;10;0.00;1544844; gartengeräte;1544844;gartenkralle garden claw | gc 821-06;gc 821-06;joseph enterprises inc.;0021363810006;4;1
E0023942438236;15.0000;20;0.00;1170909; zubehör;1170909;bd-r m-disc   4x jc  25gb            5st;43823;verbatim;0023942438236;4;1
E0023942438885;81.2000;20;0.00;1538574; bd-brenner;1538574;verb blu-ray bdxl recorder slim  u3 bk r;43888;verbatim;0023942438885;4;1

removed_woord_tmp.csv (current output)

E0017229173866;14.9700;100;0.00;104067;consumer headset zubehör ;104067;poly voyager legend ladebox schwarz;89036-101;poly;0017229173866;15;1
E0017229174177;103.5900;20;0.00;1779011; headsets;1779011;poly voyager 4310 uc usb-a mono cs | with charge stand;218471-01;plantronics;0017229174177;4;1
E0017229175082;160.8000;25;0.00;309822;headset pc´s ;309822;poly sync 40+ ms usb-a bt600ww;218764-01;poly;0017229175082;15;1
E0017229175099;159.8000;100;0.00;309823;headset pc´s ;309823;poly sync 40+ usb-a bt600ww;218765-01;poly;0017229175099;15;1

removed_woord_tmp.csv (new expected output)

E0017229173866;14.9700;100;0.00;104067;consumer headset zubehör ;104067;poly voyager legend ladebox schwarz;89036-101;poly;0017229173866;15;1;poly voyager
E0017229174177;103.5900;20;0.00;1779011; headsets;1779011;poly voyager 4310 uc usb-a mono cs | with charge stand;218471-01;plantronics;0017229174177;4;1;poly voyager
E0017229175082;160.8000;25;0.00;309822;headset pc´s ;309822;poly sync 40+ ms usb-a bt600ww;218764-01;poly;0017229175082;15;1;poly sync
E0017229175099;159.8000;100;0.00;309823;headset pc´s ;309823;poly sync 40+ usb-a bt600ww;218765-01;poly;0017229175099;15;1;poly sync

Of course i forgot the most important part of adding some minor details. When you check the 2 outputs (new and old) you see that on the new one it has an extra column with the value that makes it that it will be filtered (poly voyager for the first 2 and poly sync for the 2 other lines). I hope this makes sense now?

Try

awk -F";" '
NR==FNR                         {SRCH=SRCH DL tolower($0)
                                 DL = "|"
                                 next
                                }
match(tolower($0), SRCH)        {print $0, substr ($0, RSTART, RLENGTH) > "'"removed_woord_tmp.csv"'"
                                 next
                                }

                                {print $0 > "'"filtered_winnaar_2.csv"'"
                                }
' OFS=";" prijslijst_filter.csv lowercase_winnaar.csv

I just checked and it seems to work. Tomorrow i will do a proper check because i did notice a difference between the old and new style of search.

The old result had 7896 results and the new one 7874 results with about max 5min in between so the data should be the same. Not sure yet where this is coming from but as soon as i know more i will post it.

There shouldn't be a difference.
The match function works like the ~ operator, but sets RSTART and RLENGTH in addition.

Post the "missing" lines, and your criteria file, so we can analyse. Or, are those 22 en plus lines erroneous?

Both replies actually made me curious just as you are so i took the results i saved (it runs every hour again with new data) and found that there where "2 problems". 1 problem is that one of our suppliers tends to add newlines in the data so i have to sanitize that some more, i thought it was removed before but both the old and new one have 72 lines that do match each other but did not have the same columns as the rest of the file because of the newlines (i manually checked those).

Then there is the second problem which is the 22 lines that the old one has more then the new one.

These lines are as follows (Do note that i had to copy them over from excel so the 3rd from the right looks weird but that one does not have any data in it that could be filtered)

E4027383400100;231.000;7;30.00;34275;packaging/bubble envelopes;34275;bubble envelopes brown size k 370x480mm (100 pcs.);;kein hersteller;4,02738E+12;22;1
E4027383500107;231.000;23;30.00;34276;packaging/bubble envelopes;34276;bubble envelopes white size k 370x480mm (100 pcs.);;kein hersteller;4,02738E+12;22;1
E4260135963250;899.900;23;30.00;19388;sport & fitness/vibration plates;19388;fitness body magnetic therapy vibration plate + music 73cm (black-red);td006c-9 red;kein hersteller;4,26014E+12;22;1
E4260135963267;899.900;13;30.00;19389;sport & fitness/vibration plates;19389;fitness body magnetic therapy vibration plate + music 73cm (black-gold);td006c-9 gold;kein hersteller;4,26014E+12;22;1
E4260135963274;899.900;25;30.00;19390;sport & fitness/vibration plates;19390;fitness body magnetic therapy vibration plate + music 73cm (black-silver);td006c-9 silver;kein hersteller;4,26014E+12;22;1
E4260135967067;999.500;34;30.00;28037;sport & fitness/massage;28037;vibrating foot and leg massager with heating function (gold, td001f-8);td001-f8;kein hersteller;4,26014E+12;22;1
E4260272286342;2.490.000;13;30.00;38272;toys/table tennis;38272;tenniswall training net;mh591917;kein hersteller;4,26027E+12;22;1
E4260272287639;1.249.900;71;30.00;31459;sport & fitness/vibration plates;31459;vibration plate 3d mode/dual with bluetooth speaker (78cm, td006c-4);td006c-4;kein hersteller;4,26027E+12;22;1
E4010995420697;11.274.000;10;18.00;1608634;aufbewahrung;1608634;wiha tool set electrician compet. xxl ii | 42069;42069;wiha;4,011E+12;4;1
E4050026031430;7.990.000;10;0.00;153620;system tk anlagen grundsysteme;153620;unify openscape business v2 x5w.;l30251-u600-g657;unify;4,05003E+12;15;1
E4242003859049;10.241.200;10;0.00;311003;gastartikel;311003;siemens tp707d06 kaffeevollautomat;tp707d06;siemens;4,242E+12;15;1
E4270002813113;9.020.500;25;0.00;311651;gastartikel;311651;rocket appartamento - kompakte siebträger espressomaschine - chrom weiß;re501a1w11;rocket;4,27E+12;15;1
E4710886761587;8.395.000;10;4.05;1778616; <=20.1;1778616;acer an517-52-59l2       i5  8 n bk w11h | nh.qdvev.006;nh.qdvev.006;acer;4,71089E+12;4;1
E4710886765196;10.916.000;20;2.70;1778686; <=20.1;1778686;acer an517-41-r9kh       r5 16 n bk w11h | nh.qarev.00t;nh.qarev.00t;acer;4,71089E+12;4;1
E4710886765332;18.479.000;20;4.10;1778692; <=20.1;1778692;acer an517-41-r6xm       r9 16 n bk w11h | nh.qbhev.00p;nh.qbhev.00p;acer;4,71089E+12;4;1
E4710886831396;15.117.600;20;4.10;1808422; <=20.1;1808422;acer an517-54-77g8       i7 16 n bk w11h | nh.qfcev.002;nh.qfcev.002;acer;4,71089E+12;4;1
E5035048722671;14.370.000;10;0.00;1588057; schrauben;1588057;dewa akkuschr dck865p4t   set        18v | dck865p4t-qw;dck865p4t-qw;dewalt;5,03505E+12;4;1
E5060359289292;12.282.000;10;0.00;1789043; kleinelektro;1789043;irobot roomba s9+;roomba s9+;irobot;5,06036E+12;4;1
E7394001011077;9.050.500;10;0.00;190428;konferenzsysteme;190428;konftel c50300ipx;951401084;konftel;7,394E+12;15;1
E7394001011145;8.446.600;10;0.00;190425;konferenzsysteme;190425;konftel c50300wx;951401078;konftel;7,394E+12;15;1
E8004399331037;8.878.000;20;0.00;1399488; kleinelektro;1399488;delo vollautom ecam 656.75.ms      ed-bk;ecam 656.75.ms;delonghi;8,0044E+12;4;1

And the full filter file is as follows:

0889842014273
0889842014402
0889842014532
0889842014662
0889842016734
0889842019322
0889842019452
188057
19" gehäuse 
19" server barebones
19" server-gehäuse & server-racks
19" serverschrank-zubehör
19" zubehör
231772
26 bis 40 zoll
28 zoll
2k games
318971
32 zoll
37 zoll
40 zoll
43 zoll
48 zoll
49 zoll
4k fernseher
50 zoll
55 zoll
65 zoll
absenkrahmen
abzugshaube
acer aspire
acer pc
acer switch
acer veriton
achtergrond
action- & spielfiguren
adblocker
adventskal
aktenvernichter
all-in-one
all in one pc
all in one systeme
analoge gateways
antennen -sat-
anti-virus
antwoordapparaten
anwendung
apple care protection
apple cto
apple-desktops
apple imac
apple mac
apple macbook
apple-notebooks
apple-server
arabic
arbeitsspeicher / ram
arbeitszeit und ersatzteile
asus e-box
asus pc
audio-/ musik cds und hörspiele
azerty
baby-/ kleinkindspielzeug
backo
backofe
bandsaege
basteln und malen
bauelemente
bauknecht
bcher
bekleidung
beko
benzin
bladblazers
blomberg
blu-ray-filme
bluray-recorder  -stand-alone-
bodenstandhalter
brochures
bruder
buch
bügelbrett
bugeltische
built-to-order
büromobiliar
bürostuhl
business notebooks
b-ware
candy
carepack
care pack
cat5 kabel
cat6 kabel
cat7 rohkabel
cd-medien
ci module
cisco
cleanair
collect and return
configurate-to-order
coverplus
dampfgarer
datensicherheit - lizenzen
datensicherung / streamer
deckenlüftung
dect
dect telefon
desktop pc
deutsch
dienstleistung
digital-decoder
digital signage
dokumentenscanner
druckerpapier / kopierpapier
duits
dunstabzugshaube
durchlauferhitzer
dvd & blu-ray recorder
dvd-medien
dvd-recorder  -stand-alone- hdd
education
einbau
eingabeger
einkoch
einkochautomaten
ek water blocks
elektro grossgeraete
elitebook
energieeffizienzklasse
enterprise
epack
ergonomische tastatur
ersatzlampe
ersatzteile fujitsu
e-support
exquisit
fahrzeug
fan-wear
fax
fernseher
festnetz
festnetztelefone
fimo kids
fisher-price
flachschirmhaube
fleecejas
flipchartpapier
flüssiggas
fotopapier -rollen-
french
fritz
fuehrungsschienen und untergestelle
gabeln
gaming chairs / spielsitze
gaming pc
gaming stuhl
gaming tastaturen / keyboards
garantie
garantierweiterung
garantieverlängerung
gartenspielgerat
gasgrill
gasherd
gefrier
gehrungssaegen
gehrungssge
generator
german
german
geschirrsp
geschsp
gesellschaftsspiel
glaskeramik
gorenje
grasmaaier
greek
griffleiste
groelektro
großelektro
großformatdrucker
grundig fernseher
guthabenkarte
habro
handgrondboor  
handschoenen
hardware-applience
hasbro
hdmi-adapter
hdmi-kabel
hd+ receiver
heckenschere
heckträger
heggenschaar
heißluftofen
helm
herde
herdset
herd-set
herstellergarantien
holzlaptop
holzspalter
hp elitepad
hp enterprise
hp pavilion
hp spectre
hyrican pc
ideacentre
ideapad
imac
inaktiven
indesit
induktions
industrie built-to-order
inselesse
inselhaube
internet security
isdn gateways
isdn kabel und adapter
isdn karten
isdn-produkte
italian
italian
it-schrank
jugendliteratur
jumbo
kabel
kabelbinder
kaminverlängerung
kapp- & gehrungssage
kappsaegen
kapseln
kaspersky
kassensysteme
kehrmaschine
kehrmaschinen
khlbox
khlgefko
khl-/gefrierkombination
khlschr
khlschrank
kickertisch
kinder multimedia
kiosksystemen
kleidung
klimagerat
klimagert
kochfeld
kochfelder
kombi-mikrowelle
komplett pc
komplettsysteme
kondenstrockner
kosmos
kuehlboxen
kuehlschraenke
kuhlbox
kühlbox
kuhl-/gefrierkombination
kuhlschrank
kühlschrank
kühlschränke
kultivator 
kvm kabel
laserdrucker farbe
laserdrucker multifunktion
laserdrucker s/w
laubblaser/-sauger
laubblser
laubsauger
lcd-fernseher
lcd-tv
led fernseher
led-tv
leinwand
lenovo miix
lernbuch
lg fernseher
lichtwellenleiterkab
lichtwellenleiterzubehör
lichtzelte
lizenz
lüfterbaustein
macbook
maher
mainboards server
maus & tastatur sets
mbel
medizinische tastaturen
merchandising
mher
microsoft xbox360 spiele
microsoft xbox one spiele
mini backoefen
mini itx
minikuche (pantry)
mini-ofen
minipc
mini pc
mischgetrank
mobel
montagerampe
montageturmverlängerung
motorschubkarre
multifunktionsgeräte
multimedia pc
neff
nintendo 3ds spiele
nintendo ds / 3ds spiele
nintendo wii spiele
nintendo wii u spiele
notebook
onsite service
opnametafels
optionales druckerzube
panasonic fernseher
paneelsaegen
papierrollen
parabolantenne
paytv-decoder
pc spiele
pc-system
philips fernseher
plasma-tv
playseat
playstation 4 spiele
plotter
plotterpapier
plüsch
potzugiesisch
prepaid-karten
projectierack
projectieschermen
projectietafels
projector lampen
projektorlampe
promotie materiaal
public display
pumpentrockner
puppen & zubehör
puzzle
qwertz
qwerz
raceroom
rasenm
rasenmaher
rasenmäher
rasenmher
rasentrimmer
refurbished
replacement lampe
repro standaard & accessoires
rollwerkbank
russian
rwe
saeulenbohrmaschine
samsung fernseher
sat-anlagen
sat antenne
satelliet
satellite
sat-installation
sat-komplettanlagen
sat-receiver. digital
sat-schsseln
sat signalumsetzer
sat spiegel
sat-verteiler
sat-zubeh
sat zubehoer
schaufeln
schetrockner
schlauchwagen
schmidt spiele
schneefrase
schnepel
schrank
schweissgeraete
selfsat
server
server zubehör
service
set-top-boxen
severin
sharp fernseher
side by side
side-by-side
sirup
sitzgruppe
smoby
smoker grillwagen
sockelverkleidung
software
solo-mikrowelle
sony fernseher
sony playstation 3 spiele
sony playstation 4 spiele
sony playstation portable filme (psp)
sony psp / sony vita spiele
spanish
spaten 
speedphone
speichermodule
standherd
stand lautsprecher
stofftiere
streuwagen
stromerzeuger
stromkabel
studioverlichting
stuhl
support
tc4
telefon anlagen
telefone
telefonzubehör
telefoon -bedraad-
telefoon -draadloos-
telekom
teleskop-vollauszug
temperaturschraenke
theo klein
thermotransferrollen
thin clients
tintendrucker farbe
tintendrucker multifunktion
tintendrucker s/w
tischbackofen
tischgeschirrspüler
tischkreissaege
tk-anlagen module
toebehoren  kiosksystemen
toner
toplader
toshiba satellite
transformer pad
tresore
tretfahrzeuge
trolleys
türfront
türverkleidung
tv combi -groter dan 46 cm-
tv geräte
tv-karten
tv -lcd & groter dan 46 cm-
tv-möbel
tv-receiver
type cover
typee cover
ultrabook & convertible
ultrabooks
umluftbetrieb
umluftkamin
unterbau
unterbauhaube
usb-adapter
usb-kabel
usb-speicher
userupgrade
usv garantie- und serviceerweiterung
usv systeme
vcm
veegmachines
verkoopondersteunende materialen
verticuteer apparaten
vertikutierer
video/vga-adapter
video/vga kabel
vtech
wagenheber
wandesse
wandhaube
warranty
wartungskit
waschmaschine
waschmaschinen
waschtrockner
waschvollautomat
wasdkeys
werkbaenke
werkstatt
werkzeugtrolley
wertschutzbehälter
wippsaegen
wireless desktop
workstation
xbox one spiele
zanussi
zubehör apc
zwischenbauhaube
trennen & stemmen
sen & hobeln
laserdrucker>xerox
vaatwasmachine
koelkasten -stand-
koel-vries-combinaties
vaatwasser -volledig integreerbaar-
py9-00006
ps4 spiele
spiele
strijkplank
adventskalender
brettspiel
geschicklichkeitsspiel
bucher
bücher
kartenspiel
gascilinder
windows 10
sgs3
sgs1
sgc1
sgk3
-de
k70 rgb
gaming chair
pc-spielstuhl
tastaturen
akracing
home & business
windows server
lto-7
tastatur
4044951021475
4044951021451
4260195391536
photoshop
slim folio
5707119025119
vertagear
bügeltisch
ecosys 
versalink 
paintshop
parallels
id8bsf-bbg
student 2019
renewd
4000810314201
4006094033302
4006094032800
4006094033302
4006094036006
4006094037577
peter bausch
lizenzen
patrone
8806088990064
6970244526816
office-pakete
office 365
sw_office
office-pakete
4012700802682
keyboard
4048164600359
ink cartridges
tintenpatronen
druckerpatronen
tintenpatrone
0882780869820
0884962986431
zubehã£â¶r tintenstrahldrucker
0740617260533
0766623545075
0848412012750
0884962925348
0885370720693
0889842095883
4260464224145
4948382940063
5031199043054
5031713352006
kaffeebohnen
0088698541630
lexware
esdptr2019mlug
corel
officeconnect
hpe aruba
download
p00183715
buhl
acronis
absaughaube
4078500028332
<=
p00184849
0192018284814
0192018284913
p00062213
p00187064
p00063592
p00139197
p00097681
p00187496
p00185346
p00051898
p00167096
p00147194
p00120682
p00157130
p00188394
0600346252028
4960999977287
privacyfilter
privacy-filter
4005556275908
tonies
wyse
0778988194034
0088381835770
088381835770
0088381689120
0088381693561
6907502583564
roadstar
4038986124508
4038986124775
4004943567268
5035048470596
beats
4078500889001
4008423869996
3165140562935
trotec
trotec
4078500042864
0815820020103
4078500300100
6970100371215
p00188622
p00188628
6970100371536
6970100371468
4008423861181
4000810314010
4000810114221
3138522070649
3138522070618
3138522109950
3138522072452
gasgrill
avalon
4000810141012
miton
gasgrillküche
gasgrillwagen
gasgrill
feuerkorb
4000810114023
4001690101363
4008423861389
4078500012638
5703887107255
4078500012652
4078500042611
4078500033589
4078500012676
4078500033565
baumscheren
8806084842190
4031541727817
4006825603842
0021615020146
191376403660
4548736036369
5060059049714
8712581730314
4260079822675
4260094730139
4260094730146
8715946454245
6944904418162
6944904418131
6944904418063
6944904418179
6944904418049
5060059049714
3165140824064
0688334003227
4004943565608
8806088820255
3165140705554
6941059614654
730143307659
0730143307659
3165140821629
8806086772853
8004399512351
mobiles klima
8806088730332
0088381619066
4044951021727
skiller
5901912283817
4004470738872
4713883475594
5035048656563
6970100371543
p00188622
0088381699013
4008423878554
3165140488563
4078500033688
4009269709187
4009269306522
3165140649544
4078500031592
4078500031592
3165140742085
0888793859940
4078500017985
p00139358
8718696408926
8718696541241
4005435104756
5035048335055
3165140349383
5010994836108
4015704266009
tropicool
mikrowelle
4008423869743
4717964407788
4052095640490
6411501150127
schneidgiraffe
4078500360593
4078500371308
4078500371506
4078500373401
4008423877793
4015867199077
4008423877854
4078500017404
4008423861174
4009269303347
4009269305761
4009269303330
4008423861167
4039784518209
4009269709507
5035048356180
4013674004270
4711148599979
6970995780239
3253560436797
3138522086343
4895106287969
4895106287778
8005125590971
gigaset
optiplex
0095205838985
4021801564030
4260074975291
p00189329
8801643684570
8801643692919
p00189320
5010994887841
4712900848960
4895173610172
leinwände
4001627012540
4001627012540
0193638879732
4710180578829
0193386897477
5397184349588
4718017515511
4713392252372
0191376150281
4250812434173
eizo
eizo
p00170683
4078500042789
5010994887889
4001627012533
4001627012540
p00188351
1sgha00q
4003336938296
4054681978994
3253561703249
0778988151525
778988151525
4719331305406
0778988151525
ddr2
ddr3
0846651025760
4004943567107
0778988151532
4005380454180
8715946616292
4005998761281
778988151556
4260182765715
3165140705417
8715946610122
0718037773421
5011423190181
4078500558600
5035048657652
sigma
4004943567336
4009269305594
4078500042437
0088381153508
4008423882421
4078500555401
4078500498500
4009269331180
4009269305587
3138522109554
puddle jumpers
6942138940008
puddle jumper
schwimmflügel
schwimmlernhilfe
0606449136616
606449136616
5905568347107
4008423882452
0733538867295
4006160639421
4009269724258
4078500042697
6942138925579
5099383005172
4260236271896
5028252351898
pnzk49
4030454003896
0088381688321
088381688321
4078500886604
5035048706824
0094338513806
8028153103081
3165140767682
6941057414201
4078500042598
4260182765692
4078500025539
0733538850792
0810354025464
kitchenaid
4981254028971
4058154079911
8003437237652
6941057414324
0752113653146
4030454004831
4030454004831
duplex module
0885370996357
0885370996357
0889842013719
885370996357
4004470708806
5099206080379
3045386356355
4078500048156
4006825601275
4039621003110
4001690102483
moederbord
cpu - prozessoren
plustek
netzteile
amazon
3660619402052
8710364052431
4001797270306
8021098270016
0698813001194
4054278130842
jabra
jabra
5706991013733
5706991016512
5706991017090
5706991019827
5706991020717
5706991017045
5706991017106
5706991017632
5706991016659
5706991004151
5706991010824
5706991005622
5706991018318
5706991018387
5706991010961
5706991010978
5706991013245
5706991020366
5706991020700
5706991019858
5706991017168
evolve
uc duo
ms duo
4006501724397
strijkplank
4006501724397
4006501724472
4006501724458
4006501724533
4006501724526
4006501724519
4039784977655
4054278459844
fiskars
8717973932468
4078500028370
5035048554081
4078500012713
4210201211075
8021098774828
6438158764194
6438158763869
8003437237669
7611160405265
5099206086449
4078500873802
4078500302401
4040849915871
5099206086456
gaming seat
4078500351706
5025232867073
4078500301503
4006501853523
4078500300407
3165140465366
3165140662208
4005998148433
4002667356250
7393033096137
singer
4078500016483
4005998835135
5397184491508
5397184472767
4210201211075
4210201211082
0000000000000
owc
sieraden
durex
dreamtoys
pornhub
cook expert
8718696687581
4078500054287
8717973931829
3875001740067
klimagerät
airco
8712184054909
8712184055647
8717973938910
8717973935100
8717973931812
4004943565509
4006501852854
8717973931843
8717973934899
4078500032698
pfaff
501937
4025112071379
speichermedien
laserdrucker
flachbettscanner
tablet
>=2000 ansi lumen
digitale bildbearbeitung
lã¼fter & kã¼hler
am4
usv
office
mit touchscreen. speicher >=8gb
tablet pcs
lga 1200
s1151
combos
ethernet-karten
atx big tower
onboard cpu amd
onboard cpu intel
s1150
gasflaschen
toiletten
atx midi tower
 gehã¤uselã¼fter
 mini-pc-gehã¤use
 mit netzteil
 ddr4
 <=13.3
 i/o-karten
 sata
 s-ata
 seriell
 displayport
 hubs
 ip kameras
 wlan
elektr. beschriftungsgerã¤t
drucker
vga
android
tft-touchscreen
module
versand
homematic ip
<=12.1
<=17.4
brillen
flachbettscanner
tft-displays
gaming
multiplattform
barebones
s2066
sockel 1200
tischladestationen
19gehã¤use
atx desktop
einplatinen
sat-schã¼sseln
spezialspeicher
<=14.1
<=16.0
notebookzubehã¶r
lautsprecher
netzgerã¤te
switches
wireless lan
farbbã¤nder
nadeldrucker
mit hdd
netzwerk
bd-brenner
dvd-brenner
pci-express
mikrofone
hybrid
dvi
soundkarten
media-pc-gehã¤use
ohne netzteil
sonstiges
nas/ndas
dvb-t
cooling
ethernet-over-power
printserver
repeater
router
speichermedien
festplatten
laserdrucker
<=750 ansi lumen
aerocool
apple
biostar
brother
digitus
goobay
hp
htc
intenso
microsoft
netgear
sharkoon
tp-link
transcend
verbatim
yealink
zebra
corsair
be quiet!
logitech
mediarange
silverstone
steelseries
thermaltake
delock
noctua
hyperx
asus
nzxt
cooler master
coolermaster
fractal design
gehã¤uselã¼fter
0088381894715
4078500054478
4078500372800
4078500032278
4078500555609
4008423877779
4008423877847
4078500374507
3165140888042
3165140888059
5035048470633
4260556140094
4078500054263
8717973937234
droogmolen
4006501852861
wäschespinne
4059952526782
4078500054232
4005998233962
4260182767924
4078500053037
4005998201435
4013674003983
4260341185583
4260182761205
4006501724403
4006501724465
5025232675371
4078500054249
4006501724410
4260556140049
4260556140087
4260556140063
4260556142500
4260556142647
5025232815180
4260059582797
4260059582780
6947273701774
5025232825516
5025232815067
5025232731794
5025232829644
5025232675401
5025232815135
5025232871896
5025232871940
5025232731787
4009839276071
5025232866199
5025232886647
5415182042505
4078500054300
5025232731770
4044763008060
6937295600681
5025232815166
5025232866182
9120042772824
405002600366
4019377903476
snom
9005613149154
inter-tech
msi
gigabyte
papst
fractal
steelseries
tower
itx
atx
5025232644926
4005998769096
pc-systeme
notebooks
tablet-schutz
cpu
mainboards
festplatten intern sata
maus-tastaturkombinationen
multifunktionsdrucker
farblaserdrucker
poly
fractal design
plantronics
6941057418148
4048164102884
erotic
6942138949735
8004399020061
8004399512337
4048164102891
4053035820019
4260328611630
b-ware
refurbished
geoeffnet
qwertz
serviceerweiterung
care pack
arbeitszeit
kingston
kingston
kingston
lizenz
6942138987126
6941057414461
6941057414409
e6942138987126
4015704254860
4015704254860
6941057420486
502523268061
6941057414362
6941057414362
6941057408514
4008423877786
4008423877861
4008423882407
4078500555005
5025232675463
4078500031585
4250435700198
saeco
5025232675326
grandstream
3253561431098
5025232644919
5025232731800
6970357852833
4021121468827
8435484056069
addi
3138522092955
galaxy
4028982005796
cyb-bb02-150
6941057418544
4260135967111
0887961904123
4008423877816
0035051576747
5025232675432
5025232644896
0035051571452
8806092123311
4002395003174
7630013825739
voip
epos
8806092119390
8717973936992
4052926069001
8718696705315
4078500033664
0045496420314
4019631671547
tigerbox
grandstream
4006528022223
p00191874
powercube
allocacoc
designnest
designnest
allocacoc 
powercube
8858782109617
8858782109648
8858782105749
8858782105954
8858782107149
8858782107163
8858782107255
8858782107286
\n
8858782108887
\n
8858782111009
\n
8858782113904
8858782114611
8858782111290
8858782111931
8858782112440
8858782114284
\n
8858782114307
\n
0045496424749
0045496424749
8886419379324
0887961974782
887961974782
8886419379324
8886419379379
3165140557023
8718696123096
smartphones
handy
norton
symantec
arbeitsspeicher
ersatzteil drucker
grafikkarten
kabel &amp; adapter
kvm/splitter/umschalter
netzwerkkabel
netzwerkkameras
netzwerkkarten
powerline
public displays
schnittstellenkarten
speicherkarten
ssd
sw_hybrid
sw_lizenzen_app
sw_lizenzen_system
sw_mobility_design
sw_office
sw_organisation
sw_os
sw_security
tastatur-mauskombinationen
usb-hubs
voip voice over ip
wlan/wireless lan
workstations
zub. notebooks win
nas
bohnen
0088381291736
4015704264654
4710483936067
sockel 
intel
0045496452599
ekwb
ipevo
0088381883023
088381883023
8806092122178
0088381894654heggenschaar 
0778988397718
8806092121980
8806092122154

This one i took straight out of the database to have no excel screwups. I did notice some /n in there and i will take them out to check again but i somehow doubt these where the cause of those 20 lines.

[Addition]
I just sanitized the whole filter database but it did not change the output. Its still the same 20 lines as posted above. I cannot wrap my head around the part why there is a difference between the 2 even though they use exact the same data except for the filter part which of course has a minor addon.

Now - THAT'S an enormous "filter" regex in the SRCH variable - I wouldn't have dared to try that on my behalf. I'm surprised it worked that well.
Caveat: it contains several characters that might be interpreted as regex meta characters ("+", "(", ")", not sure about the "&"), as well as leading and trailing spaces, which all might lead to erroneous results (in either version).
But:
Either version doesn't like those 21 lines, nor does a grep approach with the "filter file". So : are you SURE the two versions were operatiing on identical input (filter AND data) when delivering different results?

EDIT: (and side remark)
You are talking of a database. Wouldn't it make more sense to do the filtering inside the DB? There are powerful SQL commands in them there hills...

The leading and trailing spaces should be gone now as i sanitized those yesterday. The same for the new line characters that where present in some lines.

From what i can see only the old version removes those 21 lines but the new one does not. Since i am not sure what triggers those 21 lines its hard to say if they have to be removed. They are operating on identical output as i run the new one about 1 min after the other one has run so the input is precise the same. To be sure i double check the timestamps each time (when its not xx:31, i wait since it has not run yet).

We indeed have this in an MySQL database. I would guess it would make a lot more sense but i am not sure how to do a full text search through (most) columns in 1 go when i am running the script itself. I do know how i can do it with a single column and i would guess that i could copy that part and just change the column it has to use but i am pretty sure that would not be the correct way even though it would probably work.

Too be honest the filtering is not that big as it runs over the complete input of about 30k lines at this moment. We want to slim it down a bit so each supplier has its own filtering before the big boy filtering comes around the corner but that will be done in the next version of the script later this year. Until then i could use the added value line so i can easily spot errors in the filtering (hp for example needs to be filtered but this also gets used a lot for power tools from Makita for example so i have to change that to make it better). The special chars never gave any problems btw as we always double checked the output after it had run and this puppy has run very good the last few years for which i am super grateful.

[Addition]
I am just checking again and i am noticing something:

E4710886761587;8.395.000;10;4.05;1778616; <=20.1;1778616;acer an517-52-59l2       i5  8 n bk w11h | nh.qdvev.006;nh.qdvev.006;acer;4,71089E+12;4;1
E4710886765196;10.916.000;20;2.70;1778686; <=20.1;1778686;acer an517-41-r9kh       r5 16 n bk w11h | nh.qarev.00t;nh.qarev.00t;acer;4,71089E+12;4;1
E4710886765332;18.479.000;20;4.10;1778692; <=20.1;1778692;acer an517-41-r6xm       r9 16 n bk w11h | nh.qbhev.00p;nh.qbhev.00p;acer;4,71089E+12;4;1
E4710886831396;15.117.600;20;4.10;1808422; <=20.1;1808422;acer an517-54-77g8       i7 16 n bk w11h | nh.qfcev.002;nh.qfcev.002;acer;4,71089E+12;4;1

These 4 lines above should be filtered but i cannot see anything that would make them filter. I would even go as far as saying that the new filtering works better since it would not remove these (i just checked these since i noticed them first) as nothing of that line is added to the filtering from what i can tell. I will have to check the other lines also to be sure but for now i think the new one seems the better (correcter/safer?) one.

Path forward from my perspective:
Take copies of both files so you work on static data.
sort and split the filter file to files containing entries with manageable sizes (50?) and similar structures (all 4006501724397ers , all apple stuff, all vaatwasser, ...), run both versions to compare, and also

grep -f prijslijst_filter_split.csv lowercase_winnaar.csv

to verify.

Going to do that and will post the results as soon as i have them. Gonna run the results both with the old and new version and use the grep one to double check.

That should give me enough data to work with i guess.

[Addition]
Ok then. I finally sorted out all the files etc. It went faster then i thought so here are the results:

Old: 12878 hits
New: 12878 hits
Grep: 12877 hits

At first you might think "why so many hits?", but that is easily explained because some filters are overlapping and since i splitted the filter file into 27 chunks of 50 it of course overlaps. The weird thing however is that the grep one has 1 less hit then the others. Because i would have thought that the new and old one would be different i thought i might have done something wrong with the original files so i took both a new copy of the filter and input file and ran the test again which was an exact match.

Knowing that i tried the full filter in the old/new/grep way and all 3 got the same 7933 results. Again this is a bit confusing but i remembered you mentioned the part of special chars and just before i did the split and all i decided to recheck the whole filter file and remove duplicates and remove most special chars that could cause any issues.

Because of this i am thinking that the previous filter file had some special chars in it that screwed with the result somehow and make it that it gave back extra lines. For now it seems to have resolved and i am happy that the new one gives me back the value which was used to filter that line.

So we can call that case closed, now?

Yes sorry i forgot to set the solution which i did now.

grep by default uses BRE while awk uses ERE.
Use grep -E or egrep to work with ERE.
Use grep -F or fgrep to work with plain strings (no interpretion of RE-special characters).
Further grep takes a -i option (ignore case).