Sending large newsletter is slow

[Cybex]

Hi all,

This is the first time I am sending a large newsletter (over 520,000 recipients) with OpenEMM 2013. The performance isn't good, it takes too long to send all mails. The server has been sending mails for 22 hours now and only 133,000 mails are sent. It's a dedicated server with OpenEMM installed.

Server is an Intel Core i7 (2600) with 16GB memory installed.

Is there anything I can do to speed up the sending process?

Kind regards, John

[Fatalin]

Hi John,

Depending on your mailing-list quality it could be the case that your server get clogged by a large number of soft bounce related server messages (from the receiving mail-servers).

You may want to check your mailing-statistic. The bounce rate on the main statistic page shows only the hard bounces. If you click on the bounces-link you can also see the soft bounces. If there are several thousand soft bounces those server messages could block your server. It means that the send-out is slower than expected, but it still takes place.

You can also check your log files to see if there are many grey-listings and the like, that are additional causes for delay.

These things can always happen, if your mailing-list is not really clean. The problem will solve itself, through a number of future newsletter sendouts, because many of your current soft bounces will ultimately hard bounce and then be excluded from future send-outs. When this happens depends on your system settings. The default soft bounce -> hard bounce count is 7 (I believe). Scan the forum for further info on this bounce counter.

You can also check, if your server is on a black list. An easy way to check this is: http://zy0.de/q/YOUR_IP_ADDRESS

Example: http://zy0.de/q/46.163.77.12 for the Agnitas Newsletter.

I hope this helps you a bit.

Best Regards
Markus

[Cybex]

The bounces only work when the server is also receiving mail, right? I mean, the mx records in our DNS is not set to the OpenEMM server.

[Fatalin]

I believe the answer is yes and no.

The bounces, that the OpenEMM bounce-filter processes are EMAILS, return from a receiving mail server. Those may not be blocking your sending server.

However, there are a number of cases, were bounce situations do block your sending server. Some examplese are:

1. Grey listing
   The receiving mail server "does not know your server" and requires a resend in 5/15/60 minutes (whatever the receiving server is set to). These SERVER MESSAGES are processed by your sending server, because they are related to the send-out. They also cause additional mail send outs, that are not show in the statistic

1. Soft Bounce
   There are many different reasons why soft bounces occur. In some cases the server communication between sender and receiver can be clogged due to too many server messages.

Regards
Markus

[maschoff]

You should check Sendmail's maillog from time to time if you are not happy with the performance to make sure there are no nasty surprises.

[Cybex]

Thank guys, I'll check it to see how I can speed up this process the next time. It's still sending mail now

[Cybex]

Where can I find the sendmail log? When I monitor /var/log/maillog I don't see anything weird, except that lot's of mail being send.

Where can I see softbounces?

[maschoff]

This is the maillog. Softbounces are recorded in the database (see appendix in extension guide) and the workflow is like this:

- Sendmail log realtime scanner slrtscn.py parses sendmail's log files in /var/log/maillog for instant bounces and writes identified bounces in file ~/var/spool/log/extbounce.log and state of processing (cursor) to ~/var/run/slrtscn.save.
- if you have configured OpenEMM for processing of delayed bounces, email replies are forwarded to Procmail (via configuration in mailertable).
- Procmail analyses via control file bav.rc (bounce address validation filter) email replies with ruleset in file bav.rule and forwards result to bavd (bounce address validation daemon).
- bavd writes copies of all incoming replies to ~/var/spool/filter and filtered bounces in file ~/var/spool/log/extbounce.log.
- update.py reads and processes extbounce.log and writes result to bounce_tbl.
- Softbounces in bounce_tbl are processed by softbounce.sh and written to softbounce_email_tbl. Softbounce rules are in script softbounce.py, beginning at about line 200.