Continuous nc data acquisition fails ocassionally

Hi,

I'm receiving text data on a server trough UDP packets.

The data are encoded messages separated by blank lines, like this:

!AIVDM,1,1,,A,13FPE?PP08OG@cPH:Gi8OwwB0<0h,0*06

!AIVDM,1,1,,A,13Ebj60000wH2E:H:mKIF2GB2<17,0*49

!AIVDM,1,1,,B,4028nAAv9Kh0awESU0HRIP102D3f,0*5D

I'm using the following sentence to adquire that data| add a timestamp and remove the blank line| separate in 1MegaByte files:

nc -ul -p 56045 -q -1 -vv | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }'| split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

It seems to work but after a period of 5-8 hours the splited text files stop to increase their size and content, and I can't get more data till I kill the processes. When the adquisitions stops I get no error on screen and the processes appear if I call ps.

Any idea of what may be happening?

All the best

I wonder if it's network-related. Are the sender and receiver on the same subnet?

No, they are not. But if the sender ocassionally stops sending packets (this can happens), nc with -q -1 should listen and wait forever isn't?

It would be interesting to see where the data is being lost.

Why don't you throw some tee commands in those pipelines and when it happens again you will know if it's nc awk or split that's losing the data:

nc -ul -p 56045 -q -1 -vv | tee /tmp/pre-awk.dat | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }' | tee /tmp/pre-split.dat | split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

Are you using GNU or BSD netcat ?

With BSD netcat -k option will listen forever, with GNU i do not think it works.

Both should be available on linux distributions
On debian :

~$ apt-cache search netcat-
netcat-traditional - TCP/IP swiss army knife
netcat-openbsd - TCP/IP swiss army knife # this is the one you want with -k option.

Regards
Peasant.

Thank Chubler_XL i will try and tell you.
Yesterday i tried a simpler:

nc -ul -p 56045 -q -1 -vv >data.txt

and there was no problem, so it seems that it has to be the awk or split sentence.

Thanks Peasant, I'm using GNU but I guess that the nc -q -1 option should work and wait forever and the tried that I did yesterday seems to confirm this. At this moment I guess that the problem is in the awk or split sentence

All the best

I don't think you've said what operating system or filesystem type this is. Is there a possibility that you're hitting a maximum file size for the filesystem? When it stops writing to the file, is the file near or about the same size each time?

Another possible pitfall: It may be waiting in buffers between programs, rather than truly dead.

After testing chubler_XL suggestion:

nc -ul -p 56045 -q -1 -vv | tee /tmp/pre-awk.dat | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }' | tee /tmp/pre-split.dat | split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

I can say that it seems that the split command is stopping the whole sentence because there some data that arrives to pre-awk.data and not to pre-split.data

hicksd8 I'm working on GNU/Linux 9.4 and the file stream could be infinite, I'm dividing it with split in 1M files and curiously the process always ends between 3-4 files (3.5-4M). I dont think its a problem with the file size but maybe a buffer size or buffer waiting as Corona668 is pointing.

Any clues on how to trace this? I'm not having any error message

It can be a problem of the ssh session?

I mean, if I do something like

AISdecode.sh

#!/bin/sh
#CONTINUOUS AIS DECODE TO PERIODICALLY JSON FILES
nc -ul -p 56045 -q -1 | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0;fflush();  }' > awk.txt 2>&1 &

and then execute ./AISdecode.sh &
should I have any problem when the ssh session closes?

Note that in this case I've added and fflush(); to the awk sentence, eliminated the split sentence and reidirect stderror and stdout to a file just to tests by parts and see if I can find the problem. I'm my last try without 2>&1 & it stopped with a file of 3.8 M, now I'm trying this, lets see.

Any more ideas? This should be working...

To avoid issues with session timeout use screen

[LEFT]Run a screen session, detach, return to it later (attach).
There are, of course, other alternatives and utilities, but i find screen mature, so i recommend it.[/LEFT]

Is it a real pain to try out BSD nc variant to check if you are having same symptoms ?

To really detected what is happening one would need to tcpdump (or equivalent) and examine the generated files after, looking for suspected behavior.
I see no other way of telling what is going on, if network level is the problem of some sort.

When did the problem surfaced ?
Did it stop working in some point after working for some time or ?

Hope that helps
Regards
Peasant.

Would it make sense to start from the other side and flood the network and reception pipe from the sender side, by e.g. cat ting over MBytes and MBytes?

Thanks Peasant, I will check screen

What do you mean by BSD nc variant?
The problem is a new issue, I mean is a new problem, I recently obtained this data stream and I want to save it continuously but at the moment I can't get a satisfactory result, from time to time the pipe stop receiving data and saving to the file.

I guess that from time to time some packect get lost or the transmission temporarely stops or the pipe buffer if filled so even when the data still being received the data storing get stuck.

I will keep trying

The problem with buffers, if that is the problem, would be the buffer NOT filling, storing up data and waiting until its full.

-I've realized that the script only stops and 5:05 and 17:05 so I thought that maybe was a cron job in the server but I could not fin anything. I tried the same in my local area network and the same happens so maybe the data streams is doing something weird and that times.

-I use vim to take a look to the files when this happens, to see if theres any special mark. I found that at the end of the file there are several trailing characters:

556×796 1.21 KB

�Maybe this is causing any problem to the awk sentence?
�Any clues of waht is happening or how to solve it?

All the best

I know what is happening hopefully you can help me now:

-Two times a day the receiver from where I'm getting the data stops sending for a couple of seconds due to an internal cron job.

-Because of this my entire all the process of getting AIS sentences stops:

nc -ul -p 56045 -q -1 | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0;fflush();  }' > awk.txt 2>&1 &

but I'm listening forever in my nc sentences, so what is happening�?

All the best

Instead of why? how about using tail -f which is meant to solve this problem.

nc -ul -p 56045 -q -1 | tail -f | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0;fflush();  }' > awk.txt 2>&1 &

What you need to consider, if you do not like tail , is to develop some kind of daemon - usually written in C. You have an attached child process that leaves the parent there forever. If I understand what you have there.

PS: fflush() on every line forces disk I/O instead of buffered I/O. Since nc can produce an endless stream of data this change might help.

Thanks a lot Jim!

I will try with tail then and see what happens

About fflush(), I'm already using it inside my awk sentence do you mean that It better to avoid using it�?

All the best

Can you please try the bsd version
It should take little time and effort.
This was a well intent suggestion before, since i had issues with gnu nc while experimenting with go and stuff.

Similar issues as you describe here.

I did not debug further.
Only switched version which resulted in expected behavior..

Regards
Peasant.

Dear Jim,

your adding of tail -f is not working, the file its not created and nothing else happens..

Dear Peasant,

I can't try that in the server because I dont have that privilegies but I did that on my laptop with no changes...

All the best