Using priority queues in OpenEMM 2013?

[shi]

Hi there,

I've seen, that there are priority queues in OpenEMM 2013, but I have no idea how I can use them.
Is it possible to set the priority for any mailing or what exactly do they do?

There are now 3 directory in /home/openemm/var/spool:
MIDQUEUE, QUEUE and SLOWQUEUE.

It would be great if anyone could tell me how to use them.

Thanks a lot!

[maschoff]

OpenEMM uses QUEUE to send out mails. If OpenEMM has tried to deliver a certain mail three times without success the mail is moved to the MIDQUEUE. There, it is tried only once per hour to deliver the mail. After a total of 10 unsuccessful delivery tries a mail is moved to SLOWQUEUE which tries to send the mail only every 4 hours. This process helps to keep the QUEUE clean for fast mail deliveries.

[atego]

Maschoff,

I have OpenEMM installed and running (having previously used version 2011). However, the 'priority' queues do not appear to be working. For example, I can see messages in QUEUE from 5 days ago and since that just appear to be stuck there. The MIDQUEUE and SLOWQUEUE directories are empty.

What can I do to make sure everything is working as it should and to ensure that the QUEUE doesn't get clogged up? It's not clear what settings and parameters need to be checked.

(Previously, we OpenEMM I increased the queue size by editing pickdist.py and setting:

Code: Select all

return len ([_f for _f in os.listdir (self.queue) if _f[:2] == 'qf']) < 20000

This helped to prevent the queue from backing up with undeliverable emails. I haven't done that since upgrading because pickdist.py has changed in version 2013.)

Thanks..

[maschoff]

Service qctrl is in charge of administrating the queues. Does it run? If so, does it write anything into the log?

[atego]

Maschoff,

Thank you for your response. The qctrl process is running so I assume the service is running. The QUEUE is very slow to clear with messages hanging around for four or five days but never seemingly entering the SLOWQUEUE or MIDQUEUE. Messages do appear to be leaving the QUEUE eventually so I don't believe there is a critical problem but the system doesn't appear to be running very efficiently.

[maschoff]

If you like you can tinker with the configuration in shell script mailer.sh:

Code: Select all

$HOME/bin/qctrl -d780   move $BASE/var/spool/QUEUE $BASE/var/spool/MIDQUEUE tries:3
$HOME/bin/qctrl -d3240  move $BASE/var/spool/MIDQUEUE $BASE/var/spool/SLOWQUEUE tries:10
$HOME/bin/qctrl -d20880 force move $BASE/var/spool/SLOWQUEUE /dev/null maxage:6d

Option "-d" defines the scan interval in seconds and option "tries" defines the number of devlivery tries.

[baibai]

Hi maschoff,

May I add more "QUEUE" (eg: QUEUEA, QUEUEB, ...) for parallel sending in the same time?
If it is possible, any suggest how to done it?

[maschoff]

Our own research has shown that the current configuration of mail queues is the optimum regarding sending performance.

[baibai]

Hi maschoff,

We have upgraded our production from 2011 to 2013 and also upgraded to a better environment (m1.xlarge) on Amazon:
1. High performance CPU
2. 16+G RAM
3. Storage with good IOPS (IOPS=1500+)
4. Unlimited bandwidth for up/downstream
And we have 3M+ recipients in our mailing list.

After upgraded, I blasted and only got 7 emails/second throughput at first time, it just took 10Mbps or lower. (no tweaked yet)
The bounce status in maillog was looking good, because our list has been sent few times with 2011, it was a clean list.
We also have a good domain/IP reputation and confirmed by SenderBase, Trend Micro and DNSBL.

My question is:
How could I increase the throughput greatly with the new function: multi-queue will be better?

I expect it could send 30+ emails/second, is that possible?

[nessyu]

Can I disable the priority queue or change the resend time of midqueue and slowqueue?

[pp84]

I am having the same issue. The MIDQUEUE and SLOWQUEUE are never populated. All the files are in QUEUE. Has anyone resolved this? I am using 2013 R2 release.

[Anton]

If you want to send lots of email fast, then you will need to invest in an smtp product that is made to send email fast and efficiently. There may be a way to tweak open source products to be faster, but you need to realise that they weren't designed for this purpose, so you are always going to be limited. That will also take time, and time costs money. The products I am familiar with are expensive (~$10k) but with these products you will only be limited by your reputation (and per-ISP limits), the number of IPs you have, and your bandwidth. With OpenEMM you can simply use the smtp proxy functionality (making sure the product generates async bounces that it sends to openemm for the sync bounces it receives) and reroute to your smtp product. Because we are talking about millions of emails, you need to ask yourself whether you are making money off this, or whether you are doing some kind of social work. If you are making money, then you should probably invest in a product - at a company I used to work at we moved from 8 physical postfix machines to one PowerMTA in a VM and saw a significant increase in throughput. Openemm can take care of everything else (or most of it) but if you want to send millions of emails then you unfortunately need to make sure you have an smtp that is fit for purpose.

[maschoff]

For clarification this is how the multi-queue backend works:
- after 3 unsuccessful attempts to deliver the mail it is moved from the QUEUE to the MIDQUEUE
- after - in total - 10 unsuccessful attempts to deliver the mail it is moved from the MIDQUEUE to the SLOWQUEUE
- after 6 days the mail is removed from the SLOWQUEUE (fallback for Sendmail's standard configuration of deleting mails after 5 days)

The mails are moved by external process qctrl. If qctrl is stressed it may react later, and it won't move mails which are processed by Sendmail right now.

Reasons, why MIDQUEUE and SLOWQUEUE are not used:
- Sendmail was configured in such a way that the number of 3 or 10 attempts is never reached
- a smart-relay is use which accepts all mails initially
- process qctrl is not executed with root permission and, therefore, is unable to move mails

If these reasons do not fit, you should searchthe log files ($HOME/var/log/*qctl*log) for error messages.

Hope this helps!

[pp84]

You are right, i see tons of these in *qctrl error log, how should I go about resolving this?
ERROR/move: qf00000E000000439FA cannot be opened (13, Permission denied)

[maschoff]

"Permission denied" could indicate that qctrl does not have root permission. Did you unpack OpenEMM's binary tarball with option -p ?