Damn, I hate mail administration… And I really like google apps (gmail) or appriver mail hosting.
Today we have issue with our corporate qmail server, mail stalled.
To be clear – I change job a week ago and just 5 days as I take care of new infrastructure.
Corporate mail server powered by qmail. Thay have zabbix as monitoring software, but not enough configured, without queue monitorind and SMS escalation. Due to great number of urgent tasks, I really have no enough hands to fix all at once…
So Monday begins with great fuckup… 180 000 messages in queue…
First diagnostic steps:
Get queue length:
Read queue:
Find and read message by ID:
less /var/qmail/queue/mess/8/251640
We found a great number (more then 30 000) messages from one of our hosting servers. Looks like
Error message:
Function split() is deprecated
File: /opt/www/ms/...../www/src/programme.php
Line: 58
<code>
Nobody will patch this code - this web site still in production, but not supported anymore.
I just remove notification:
<code lang="php">
/opt/www/ms/......../www/src/functions.php
ms4# cat functions.php | grep sts
// @mail ( "webmaster@it.randomthemes.com", "PHP Error notification", $the_error
);
And continue investigetion. Found a lot messages marked as SPAM by our Kaspersky SPAM Filter.
[!! SPAM] Damn! They should be just killed by filter.
Looks like one of our engineers changed filter settings. When try to solve Jira mail delivery problem.
I change SPAM filter policy, stop qmail
Removed mail contained “webmaster@it.randomthemes.com” and “[!! SPAM]” from queue
Use find, Luke… 🙂
incoming queue here: /var/qmail/queue/mess
moved all messages to separate folder (to add them latter), recreate queue structure. And start qmail.
When level of messages in queue becomes low I add stored messages (managers start direct mail campaign and it`s heavy load our server).
In next 2 hours all come to normal!
But I still hate to run mail servers.
Same here, I try to avoid mail server administration when I can.