Email broken

timgolding · December 15, 2010, 9:25am

Hi my email appears to be broken. Well the whole server is barley responding. I can log in through ssh and look at the system.log file

sudo tail -f -n 1000 /var/log/system.log

When i do this the errors being recorded is off the scale. E.g 100's of entries per second. The errors are complaing about the database. And nobody can connect to the server. Here is some of the log

Dec 15 14:07:40 localhost imap[2207]: auxpropfunc error generic failure 
Dec 15 14:07:40 localhost imap[2207]: DBERROR: opening /var/imap/mailboxes.db: Cannot allocate memory
Dec 15 14:07:40 localhost imap[2207]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:07:40 localhost imap[2207]: Fatal error: can't read mailboxes file
Dec 15 14:07:40 localhost master[427]: service imap pid 2207 in READY state: terminated abnormally
Dec 15 14:07:40 localhost imap[2211]: auxpropfunc error generic failure 
Dec 15 14:07:40 localhost imap[2211]: DBERROR: opening /var/imap/mailboxes.db: Cannot allocate memory
Dec 15 14:07:40 localhost imap[2211]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:07:40 localhost imap[2211]: Fatal error: can't read mailboxes file
Dec 15 14:07:40 localhost master[427]: service imap pid 2211 in READY state: terminated abnormally
Dec 15 14:07:40 localhost imap[2212]: auxpropfunc error generic failure 
Dec 15 14:07:40 localhost imap[2212]: DBERROR: opening /var/imap/mailboxes.db: Cannot allocate memory
Dec 15 14:07:40 localhost imap[2212]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:07:40 localhost imap[2212]: Fatal error: can't read mailboxes file
Dec 15 14:07:40 localhost master[427]: service imap pid 2212 in READY state: terminated abnormally
Dec 15 14:07:40 localhost lmtpunix[2210]: auxpropfunc error generic failure 
Dec 15 14:07:40 localhost lmtpunix[2210]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory
Dec 15 14:07:40 localhost lmtpunix[2210]: DBERROR: opening /var/imap/deliver.db: cyrusdb error
Dec 15 14:07:40 localhost lmtpunix[2210]: FATAL: lmtpd: unable to init duplicate delivery database
Dec 15 14:07:40 localhost master[427]: service lmtpunix pid 2210 in READY state: terminated abnormally

I can see there's something wrong with email but what should i do about it.

zaxxon · December 15, 2010, 9:28am

Your box seems to have run out of memory:

Dec 15 14:07:40 localhost imap[2211]: DBERROR: opening /var/imap/mailboxes.db: Cannot allocate memory

This may explain why the server is "barely" responding.
Maybe a memory leak somewhere or just a lot of memory consumption.

I am not sure if there is something like the

free

command. If yes, please issue it and post it's output using code tags, thanks.

timgolding · December 15, 2010, 9:52am

-bash: free: command not found

---------- Post updated at 06:38 AM ---------- Previous update was at 06:36 AM ----------

I have top if that's any help. but i don't know how to see processes below the page

---------- Post updated at 06:52 AM ---------- Previous update was at 06:38 AM ----------

i just run mailbfr -f with no luck.

No when i start the mail service i get this

Dec 15 14:50:04 localhost mbpath[3979]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:04 localhost mbpath[3979]: DBERROR: init() on berkeley
Dec 15 14:50:04 localhost mbpath[3979]: DBERROR db4: environment not yet opened
Dec 15 14:50:04 localhost mbpath[3979]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:04 localhost mbpath[3979]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR db4: fatal region error detected; run recovery
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR: init() on berkeley
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR db4: environment not yet opened
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:04 localhost mbpath[3980]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR db4: fatal region error detected; run recovery
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR: init() on berkeley
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR db4: environment not yet opened
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:04 localhost mbpath[3981]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR db4: fatal region error detected; run recovery
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR: init() on berkeley
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR db4: environment not yet opened
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:05 localhost mbpath[3982]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR db4: fatal region error detected; run recovery
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR: init() on berkeley
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR db4: environment not yet opened
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:05 localhost mbpath[3983]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR db4: fatal region error detected; run recovery
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR: dbenv->open '/var/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database recovery
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR: init() on berkeley
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR db4: environment not yet opened
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR: opening /var/imap/mailboxes.db: Invalid argument
Dec 15 14:50:05 localhost mbpath[3984]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error

zaxxon · December 15, 2010, 10:16am

Uhm no - you need a command that will show how much memory is used and hopefully how much swap space is used. The single command is for now not the problem. It seems to me that your box has a general memory usage problem at the moment.
Found this:
Quickly check Mac OS X�s virtual memory usage

Maybe give

vm_stat

a try.

timgolding · December 15, 2010, 10:38am

mail:~ timgolding$ vm_stat
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                   127057.
Pages active:                  25333.
Pages inactive:               148806.
Pages wired down:              26484.
"Translation faults":        5779577.
Pages copy-on-write:          888575.
Pages zero filled:            626569.
Pages reactivated:             84232.
Pageins:                      553935.
Pageouts:                       1166.
Object cache: 452236 hits of 501061 lookups (90% hit rate)

zaxxon · December 15, 2010, 10:55am

Hm ok, I guess I have to pass - do not know enough about Mac OS/X, sorry. Your box seems to have free pages left, but I am not sure if it shows them including any swap space or something.
I found some list of OS/X commands (OS X commands), where there was pointed, that the command

alloc

should list used and free memory - no clue how the output looks like. Also found another graphical tool:
Does my MAC need more memory?

If none of the output of the above commands/tools gives any clue, I have no more clue myself. Maybe checkout also Cyrus sites for the error you get.
Is there any system log you can also check for any entries that could relate to memory problems or loads of errors of other applications/programs?

timgolding · December 15, 2010, 11:05am

I ran that command after a restart. Here's what i have done.

I stopped the mail service ran mailbfr -f then reconstruct -m. Then i restarted the server. Now the mail seems to be working and the /var/log/system.log file isn't going off the scale. I don't think everything is fixed however. Just waiting patiently for another memory leak. I will leave mysql and httpd off over night. That way when i come in tommorrow and find it's all ok. I can restart mysql and httpd and see if the problem comes back. Then i will know that's it my inefficient codes that is causing the problem?

zaxxon · December 16, 2010, 3:36am

Can't you help much on this. I know a similar situation from a Linux box which was running out of memory, swapping heavily and to get some more memory free just killed randomly(?) processes, until we noticed something was not responding anymore. When this happened, there were entries in /var/log/messages (no clue on OS/X) which showed all the killing and I think there were also messages about memory problems.
You can either wait if someone with OS/X knowledge hopefully answers here in this thread too - or google if there is any how-tos or something to detect and analyze a memory leak on OS/X.

timgolding · December 16, 2010, 11:17am

Ok thanks. I found this leak command and run it on some processes.

mail:~ timgolding$ ps -ax | grep http
18780  ??  Ss     0:00.62 /usr/sbin/httpd
18781  ??  S      0:00.05 /usr/sbin/httpd
18897  ??  S      0:00.00 /usr/sbin/httpd

Gives the following

mail:~ timgolding$ sudo leaks 18780
Process 18780: 17422 nodes malloced for 1818 KB
Process 18780: 8 leaks for 336 total leaked bytes.
Leak: 0x02125540  size=128
        0x00103480 0x00103450 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
        0x00000000 0x00000000 0x00000000 0x00000000 
Leak: 0x00103450  size=48
        0x00103470 0x00103464 0x00000002 0x00000004 
        0x00103468 0x00000000 0x0010347c 0x00000000 
        0x6c6f6361 0x6c686f73 0x74000000 0x7f000001 
Leak: 0x001f7460  size=32       string 'mail.domainwise.co.uk'
Leak: 0x02125510  size=32       string 'mail.domainwise.co.uk'
Leak: 0x021123b0  size=32       string 'mail.domainwise.co.uk'
Leak: 0x021254b0  size=32
        0x00000000 0x0000003f 0x00000000 0x00000000 
        0x01000000 0x00000000 0x00000000 0x00000000 
Leak: 0x02125530  size=16
        0x00010010 0x02125540 0x00000000 0x00000000 
Leak: 0x00103480  size=16       string 'localhost'

Does that mean apache has memory leaks? If i try it with mysqld i get

mail:~ timgolding$ leaks mysqld
Process 18782: 40289 nodes malloced for 34878 KB
Process 18782: 0 leaks for 0 total leaked bytes.

---------- Post updated at 08:17 AM ---------- Previous update was at 01:25 AM ----------

I cleaned up some of my code. I think the memory leak was caused by tcpdf. Or might of been because i had too much of my program flow in one script. Either way everything seems to be working today. I made sure i revisited all the web pages i visited yesterday after the clean up and fingers crossed no memory leak. Thanks for your help zaxxon