Many processes running at the same time

UNIX for Beginners Q & A
1

Hello everybody ,

I launched cron to execute a task every hour but the job takes more than hour that's why I'm getting more than 1000 cron processes running at the same time !!!
My question is how to tell cron not to execute unless the job terminated in order to have only one process running .
Thanks in advance

2

Create your own 'flag'. For example, when the job finishes, the last thing it does is create a file called 'oktorun'. The first thing the job does when it starts is check for the existence of the file 'oktorun' and, if it does not exist then do nothing and terminate. If the file does exist then delete it and run the job. So the file does not exist whilst a job is running.

Anyway, if you're happy for a job to start immediately after the previous one finishes then why not get it to spawn itself (without the use of cron) ?

1 Like
3

first of all I wanna thank you for your answer ,
but to be honest I have no idea how to create this file , where to put it and what should I put inside.
same for the second hypothesis
thanks

4

Hmmm - something seems to be wrong here. 1000 jobs, started one per hour, would be 41 days worth of jobs - if not a single job finished in the meantime. Are you sure everything is correct with your script / program? Is it expected to finish in close to an hour? Does it finish at all? Are there locking problems?

1 Like
5

@RudiC......Good thinking.
@beautymind......Please post the script you are using.

6

One example of using a flag, or lock file.

#!/bin/bash
myname=${0##*/}
if [[ -f /tmp/${myname} ]] 
then
    myname_proc=$(</tmp/${myname})
    if ps -p ${myname_proc} >/dev/null
    then exit
    else rm  /tmp/${myname} # assumption: last iteration of program died
    fi
fi
echo $$ > /tmp/${myname}
cleanup() {
    trap '' 0 HUP INT QUIT ABRT TERM # clear traps
    rm /tmp/${myname}
    # other cleanup here
}
trap 'cleanup' 0 # always run cleanup on exit
trap 'exit' HUP INT QUIT ABRT TERM # exit on one of these signals
# the rest of your script goes here

A good place to put such a file is the /tmp directory. The cleanup function should ensure that the file is deleted on exit, so you don't have to remember to put

rm /tmp/${myname}

everywhere you exit the program. Putting the PID in the file and then checking whether the process exists is optional, and there in case the program exits abnormally, leaving the file in place.

Andrew

1 Like
7

@RudiC , I don't know what is the problem because is it expected to finish in close to an hour.

8

Use a lockfile. In bash, one can do this:-

#!/bin/bash

# Use a lock file.  Lock released on exit unless earlier.
mylockfile="/var/lock/whatever"

[ ! -f $mylockfile ] && touch $mylockfile

exec 77>> $mylockfile       # Direct file descriptor 77 (or whatever) to the lock file

flock -w 30 -x 77           # Wait for up to 30 seconds to acquire the lock on fd 77

if [ $? -ne 0 ] 
then
   logger "$0 bypass due to lock"
   exit 99
else
   logger "$0 has the lock"
fi

# Carry on with the rest of the script.....

Does this help?
Robin

9

That's why hicksd8 asked you to post the script you are using, including the environment it is executed in. As is, your waiting "until the job terminated in order to have only one process running" might wait for 41 (or even more) days.

10

@apmcd47 Thanks for the help
@hicksd8 and @RudiC Unfortunately I can't post the script ( I am intern and I do not have the right to release the code)

I think that the script takes so long unlike in the past for the folowing reason :
The script is supposed to download files each hour from 12 sites . In the past there was only one site so the job was done in less than an hour but now with 12 sites it's no longer the case.

11

How come you're having 1000 processs?

12

@Rudic 1000 process running because there are many scripts each script calls an other ,
Now what is changed . we don't have only one site as before , we have 12 sites so it takes more time

13

You have been asked to show us your script twice. If it needs to run over 800 processes to process one site, there is something seriously wrong. If you are unwilling to show us your script there isn't much we can do to help.

Please show us your script (in CODE tags) or close this thread.

14

Unfortunately I can't post the script because it is strictly forbidden as I'm working on a confidential project.
My question is : how can I do so that the process takes less time ? knowing that the script load files on HDFS
Thanks in advance

15

Since we don't know what your script is doing, there is absolutely no way that we can guess how to speed it up.

This thread is closed.

1 Like