Performance of e-mail sending

[Stephen]

I started sending a newsletter of 222,442 generated e-mails over 48 hours ago.

So far the system has only sent 100,002.

Sendmail is not stressed at all and is certainly capable of sending e-mails at a much higher rate.

Is there throttling in place in OpenEMM that would account for this poor performance?

Can I fine tune it to get better delivery rates?

[maschoff]

Stephen,

could you please specify the (dedicated?) hardware you are using for OpenEMM?

Thanks!

[sysadmin]

Stephen,

could you please specify the (dedicated?) hardware you are using for OpenEMM?

Thanks!

Hi, master

With a Pentium IV 3.2 MHz, 512 Mb, SCSI320 hd, 2 Mb bandwidth (dedicated line). Despite the bandwidth and other external concerns, which would be the openemm's capacity for sending mails? (for example, how many mails per hour with a 3 kb mailing)

Thanks in advance
Ruben

[maschoff]

Good question - I don't know because we use OpenEMM only for small lists (10k max). Maybe other OpenEMM users can share their findings here?

[sfantul]

u said 10k max
What is the time that openemm sends that 10k mails?

[Stephen]

Afraid hardware isn't the issue here -

Problem here appears to be the list size, my list contains over 545k users. Scares me to no end that you say you've only used OpenEMM for lists of around 10k!

At this point I'm/we're not even able to pull up the recipient -> overview list. I've been waiting for the page to load for 4 hours now.

No errors in any of the logs that I can find.

Running -
OpenEMM v5.0.2
CentOS v4.4

Hardware -
dual proc. 2.1 or better, don't remember off the top of my head
1 Gb memory


Initial checking using mytop showed a slow query -

select distinct cust.customer_id, cust.gender, cust.firstname, cust.lastname, cust.email from customer_1_tbl cust where 1

Must admit I was a bit perplexed by the query as it's going to return the enitre list regardless of the 'distinct' clause. Adding distinct only adds 34+ seconds to the execution time.

Adding a few indices have reduced the query time by half (down to 16.94 seconds), but still no subscriber list.

One thing that may be important is that a small list was imported with out checking for duplicates so right now we have about 80k of duplicate e-mail addresses.

Stephen

[Stephen]

Ein Fehler ist aufgetreten
Ursache: javax.servlet.ServletException: javax.servlet.jsp.JspTagException: org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; Cannot get a connection, pool exhausted; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted

I'm not a Java hack[er] so other than the obvious I don't know where to start looking for a solution. I do know that MySQL isn't indicating any problems, it's been rock stable.

Any suggestions?

[ar]

Hi Stephen,

there is a problem with loading large lists of recipients. This problem will be solved in a future version. For now you might try the following solution:

Open the file webapps/openemm/htdocs/recipient/list.jsp in an editor.

Find the line:
sqlStatement=sqlStatement.replaceAll("cust[.]bind", "bind");

Add this line after it:
sqlStatement+=" LIMIT 1000";

It should now look like this:
sqlStatement=sqlStatement.replaceAll("cust[.]bind", "bind");
sqlStatement+=" LIMIT 1000";
System.err.println("Statement: "+sqlStatement);

This should solve the problem with listing large lists of recipients.

[Stephen]

Thanks for the quick reply.

Stephen

[jim_frey]

Here's my machine:
- Port Speed: 10 Mbit Port
- Processor: Athlon XP 2100+
- Memory: 1 GB
- Operating System: CentOS 4

I'm sending about 17k emails a day for www.duncecapjokes.com -- The problem I have is that it takes just over 24 hours to send the mailing, and it's suppose to be a "joke of the day" mailing.

It appears that all the RAM is being used, but not all the CPU --

Cpu(s): 4.0% us, 2.0% sy, 0.0% ni, 93.4% id, 0.7% wa, 0.0% hi, 0.0% si

Mem: 1003172k total, 988800k used, 14372k free, 41760k buffers

Swap: 2096472k total, 225888k used, 1870584k free, 331548k cached

Has anyone found a way to bump up the output? I think it might just be a matter of adjusting sendmail...?

[Stephen]

Adjusting sendmail is not really going to improve the delivery speed, you'd really just be moving the pending queue from OpenEMM to sendmail.

The mail won't really be delivered any faster.

One option that I have considerd is configuring sendmail on the OpenEMM box to use multiple smart hosts.

For example the OpenEMM box is configured to use x number of smart hosts, which can be nothing more than a Linux OS with Sendmail configured. Effectively OpenEMM would spool out the mail x times faster than when working by it self and pass the processing, and time overhead to the smart hosts.

The possible problems are the way that OpenEMM 'hooks' in to the maillog to process bounces. Since you've handed the mail off to another system OpenEMM doesn't have direct access to the maillogs on the smart hosts. It can only read the maillog on the server that it is running on.

In theory the smart hosts will relay some form of error message back to sendmail on the OpenEMM server and hopefully OpenEMM can and will process them correctly. But since I've left the company where I had deployed OpenEMM I've not had the chance to test this configuration.

Hope that helps and if you try this post a reply to let everyone know how it works for you.

Stephen

[jim_frey]

What's the hardware bottleneck? Can I throw more RAM at it and improve the speed?

[tgilbert328]

I have spent quite a bit of time using OpenEMM over the past number of weeks, and I have also found that it can be quite slow in sending messages. It seems to get jammed up on the deferred emails. This is what I did to improve performance:

1) I added more RAM to the machine. The more ram the more instances of sendmail it will create. On my 2 gb RAM OpenSuse box I currently have 40 instances (I have seen it go to 90). It seems like the more the merrier

2) I originally ran the machine from behind my firewall (Cisco PIX). With all of the requests, particularly concurrent connections, I found that the firewall was really slowing things down. So I secured the box, and put it out on the switch directly connected to the Internet. This resulted in *much* faster sends. Prior to this I was getting only 60 emails or so a minute. Now it is a few hundred. Obviously there is some risk to this, but I closed every port that I could, reviewed www-data priviliges and secured the box. I am reasonably confident that the box is secure.

3) For me I don't really care if one particular user doesn't get an email from my system. I send out 3 or 4 a week and they are time sensitive. Therefore I dropped the retry time period from 5 days to 1 day for sendmail. This really helps in unjamming sendmail from continuously retrying poor mail servers. My one concern with this is soft-bouncing them due to many temporary failures. I am keeping my eye on this and will report back if this causes problems.

Anyone else have any other tips or criticisms of what I did?

Tim

[tgilbert328]

I am not sure how typical this is, but for the record, here is the specs on my machine.

Its an single core 2.0 Ghz Pentium IV with 2 Gb of RAM running OpenSUSE 10.2. I have one network card connected. In my last three jobs of about 50,000 each, I have achieved an average of ~10,000 emails per hour.

I think I could squeeze more performance by adding a second network card and bridging them. More RAM might help, I don't feel its relatively low CPU speed is a factor.

Again, I am not sure that this is typical, but its what I have.

Tim

[maschoff]

According to our experience 10,000 mails per hour is a decent (i.e. above-average) performance for a state-of-the-art server with one dual-core-CPU in a real world environment (with real world providers at the other end of the line).