One more idea, check the crontab file for strange characters or missing linefeed or funny permissions:
sed -n l /var/spool/cron/crontabs/report
Normal lines will be displayed with a dollar sign to show where the end of line is.
And check permissions on the crontab file (should be 400)
ls -lad /var/spool/cron/crontabs/report
Ps. In post #1 you posted that you had stop/started the cron daemon, but not how this was done.
ehehe everything is fine:
mceldev[308]/var/spool/cron/crontabs #sed -n l /var/spool/cron/crontabs/report
45 02 * * * /home/report/duo_subscribers.ksh > duo_subscribers.log$
45 03 * * * /home/report/postpaid_subscribers.ksh > postpaid_subscriber\
s.log$
59 23 * * * /home/report/daily_open_amount.ksh > open_amount.log$
20 0 9 1 * /home/report/ista_hlr.ksh > lista_hlr.log$
10 1 9 1 * /home/report/dpworld_execure.ksh > dpworld_output.log$
00 1,13 * * * /home/report/opn_amt_gestores_credito.ksh > opn_amt_gesto\
res_credito.log$
#00 13 * * * /home/report/opn_amt_gestores_credito.ksh > opn_amt_gestor\
es_credito.log$
00 4 15 * * /home/report/run_arpu_previous_month.ksh > run_arpu_previou\
s_month.log$
mceldev[309]/var/spool/cron/crontabs #
The file does indeed look clean. Were the permissions ok?
The last crontab line (04:00 on the 15th) wasn't in the crontab -l report you posted before.
Strange. Perhaps two users are editing the same crontab?
I once had issues with malformed cron entry constantly spawing new process which never end (wrong syntax for instance iostat or alike,bad scripts ), reaching queuedefs limit (maximum concurent cron/at jobs limitation).
So roots cron was malformed hitting the limit, and all other users cron's didn't work.
If this works the other way (i don't see why not), a user could compromise cron of the entire system with a simple entry 
I had one recently where the disc cabinet containing some of the scripts mentioned in root crontab failed. The cron daemon soon reached the queue limit as it queued more and more impossible jobs with hung I/O.
Well worth speed reading the cron log.
As a minimum:
grep "max run limit" /usr/adm/cron/log
I assume that /home/report is on a normal mounted partition (not NFS or whatever).
Hi All
To all that contributed to this issue, many thansk for the support. Problem now solved, with a reboot. But it would help to know why this was happening
FR
Hi FR,
May be wrong, but have u checked locale of the server at that time and now.
Hi All
crontab again not working,
Amit_Chopra: I am not english native, sorry if its dumb question, what is "locale" of the server?
---------- Post updated at 08:54 AM ---------- Previous update was at 08:51 AM ----------
After running:
grep "max run limit" /var/adm/cron/log | more
I got the following result:
! c queue max run limit reached Fri Apr 13 14:50:00 SAST 2012
! c queue max run limit reached Fri Apr 13 14:50:00 SAST 2012
! c queue max run limit reached Fri Apr 13 14:50:00 SAST 2012
! c queue max run limit reached Fri Apr 13 14:50:00 SAST 2012
! c queue max run limit reached Fri Apr 13 14:51:00 SAST 2012
! c queue max run limit reached Fri Apr 13 14:51:00 SAST 2012
Is that something wrong?
But if I run
more /var/adm/cron/queuedefs
I have the following:
a.4j1n
b.2j2n90w
So no
c
is present
Please can you help
FR
---------- Post updated at 11:28 AM ---------- Previous update was at 08:54 AM ----------
Hi
I think now the problem is solved, what I did was to change the text "a.4j1n" to "a.10j1n" in file "/var/adm/cron/queuedefs", them stopt/start cron deamon. So
nothing appears now, when I run "grep "max run limit" /var/adm/cron/log | more" and entries in the crontab are being executed.
---------- Post updated at 11:30 AM ---------- Previous update was at 11:28 AM ----------
Hi
I think now the problem is solved, what I did was to change the text "a.4j1n" to "a.10j1n" in file
/var/adm/cron/queuedefs
, them stopt/start cron deamon. So
nothing appears now, when I run
grep "max run limit" /var/adm/cron/log | more
and entries in the crontab are being executed.
The following are the alphabet codes for the queue types:
a For "at" jobs
b For "batch" jobs
c For "crontab" jobs
I think you need to add a c queue line in the queuedefs file, not change the a queue line. Be careful because the default is already 100 concurrent jobs. That is a lot of concurrent jobs so we do wonder what is happening in the other crontabs ?
you could be right, because, the problem went away for a while only. Is back, I am having issues
tail /var/adm/cron/log
! c queue max run limit reached Mon Apr 16 13:51:00 SAST 2012
! rescheduling a cron job Mon Apr 16 13:51:00 SAST 2012
! c queue max run limit reached Mon Apr 16 13:51:00 SAST 2012
! rescheduling a cron job Mon Apr 16 13:51:00 SAST 2012
! c queue max run limit reached Mon Apr 16 13:51:01 SAST 2012
! rescheduling a cron job Mon Apr 16 13:51:01 SAST 2012
! c queue max run limit reached Mon Apr 16 13:51:01 SAST 2012
! rescheduling a cron job Mon Apr 16 13:51:01 SAST 2012
! c queue max run limit reached Mon Apr 16 13:51:01 SAST 2012
! rescheduling a cron job Mon Apr 16 13:51:01 SAST 2012
and cronjobs entries are not being runned
The letter c at the start of the error message shows you that it is the c (crontab) queue.
(I had one yesterday for the a queue after a script containing at jobs went wrong).
I repeat that over 100 concurrent crontab jobs is worrying. Perhaps one of your other crontab files is faulty?
or may be is the fact that I do not have an entry for
c
in the
more /var/adm/cron/queuedefs
mceldev[513]/tmp/fr #more /var/adm/cron/queuedefs
a.30j1n
b.2j2n90w
mceldev[514]/tmp/fr #
So shall I insert a c entry on the queuedefs file?
FR
Check what's being ran inside those 100 concurrent jobs.
Are you really running 100 jobs concurrent jobs ?
Are they finishing properly ?
If all above is correct then increase concurrent jobs limit, unless you want the same to happen with 200 (or any) limit in future time 
It's likely that some of that jobs are not exiting or hanging, occupying the one of the cron job slots forever ( until cron or machine is restarted/rebooted )
Peasant makes a vary good point.
My approach would be to look at every process in memory and account for each process, looking for processes which should not be there at that time of day. Particularly if there are lots of the same process which came from a crontab line which are still there long after they should have finished.
ps -ef | sort +8 | pg
The classic problem is where cron jobs connect to a remote database but never disconnect.
I am not having 100 crontab entries. there are 17 entries, 10 for root and 7 for a user.
What is intriguing is after I changed the line a.10j1n to a.30j1n, the entries in the cron, runned, BUT I beleive by changing "a" should not make a difference, only if I changed the line "c" for crontab, it should do something, but on my queuedefs file there is not a "c" line.
#ps -ef | sort +8 | pg
UID PID PPID C STIME TTY TIME COMMAND
root 18662 3370 0 07:54:56 ? 0:00 <defunct>
root 18671 18666 0 07:55:37 pts/3 0:00 -sh
root 18687 18671 0 07:55:40 pts/3 0:00 pg
root 27814 1 0 14:21:46 ? 0:00 /usr/sbin/cron
ora10g 18665 1 0 07:55:25 ? 0:00 ora_q000_bscsprod
ora10g 18669 1 0 07:55:31 ? 0:00 ora_q001_bscsprod
report 27726 19770 0 14:18:48 pts/1 0:00 -su
ora10g 10002 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10004 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10006 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10008 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10010 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10012 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10014 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10016 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 10018 1 0 23:34:31 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 18646 1 0 07:52:22 ? 0:00 oraclebscsprod (LOCAL=NO)
ora10g 18648 1 0 07:52:22 ? 0:00 oraclebscsprod (LOCAL=NO)
root 18686 18671 0 07:55:40 pts/3 0:00 sort +8
root 20895 1 0 09:10:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22193 1 0 10:00:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22290 1 0 10:01:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22303 1 0 10:02:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22320 1 0 10:03:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22350 1 0 10:04:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
root 22406 1 0 10:05:00 ? 0:00 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
Is that ping spawned by cron? Most unix-pings do not stop unless you tell them by using the -c flag.
No that is cron entry:
#* 10-11 * * * /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
That should not be running because is commented. but it appears in the ps command
ls -lrt | tail
-rw-r--r-- 1 root sys 19928 Feb 8 16:42 disk.080212
-rw-r--r-- 1 root sys 21058 Feb 8 16:45 present_new_disk.080212
-rw-r--r-- 1 root sys 3044648 Feb 17 08:57 mceldev.170212
-rwx------ 1 root sys 11914 Feb 17 11:09 vconf11
-rwxrwxrwx 1 root sys 254 Mar 23 10:17 fs
-rwx------ 1 root sys 1398 Apr 16 15:03 netscp
-rw-r--r-- 1 root sys 0 Apr 16 16:00 outfile
-rw-r--r-- 1 root sys 0 Apr 16 16:00 errorfile
-rw-r--r-- 1 root sys 2877696 Apr 17 09:00 downlog
-rw-r--r-- 1 root sys 4 Apr 17 09:00 abc
mceldev[607]/tmp/fr #
That cron alone was enough because it will try to start 119 concurrent jobs (one per minute between 10:00-11:59) .They do look faulty because they all started the day BEFORE you issued the ps -ef .
When you fix the process, note that the HP-UX ping command takes a parameter -n <number> to restrict the number of ping packets.
I can't explain why the ping jobs have PPID 1 unless cron has been stop/started since they started running or they were actually started with nohup command & .
A gash way of finding out what jobs are running directly from cron is to find out the PID of cron itself and then grep that as PPID from ps -ef . This will not tell you what is queued but might help diagnose where the 100 concurrent jobs are coming from (assuming it wasn't the faulty ping cron).
Thank you very much for your email, I did not noticed that by running between 10:00-11:00, I was puting so much "stress" in the crontab (running lots of jobs). I did start/stop the cron earlier. Because that entry was already commented out, can I kill those pids with related with ping command?
Yes you can kill the PIDs from the rogue "ping" processes.
Killing these processes may leave crout... files in /var/spool/cron/tmp . Just check that these files are not too big, otherwise they are harmless.
Footnote: Starting and stopping cron is the last resort and you need to be sure that cron is doing nothing at the time. I would only stop cron if I was doing something unusual like working on a database when I did not want any "interruptions".