One Openemm - 4 computers. Is this possible?

[sfantul]

I want to broke openemm in modules that stay on different computers.
Mashine 1: the database
Mashine 2: python ,c scripts and sendmail
Mashine 3: openemm interface
Mashine 4: openemm redirection server

Is this possible?
What i want to know to be more exact is how this 4 entities interact.
What i know:
The database is easy to separate from others .
-for this we have to properlly modify emm.properties ,agn.py and Mailgun.ini (By the way who uses this file beside webapps? and if nobody else uses it why cant be used just "emm.properties")

Both webapps write some log files to disk in the openemm file system (but i dont think this is a big problem they could write them anywhere. no other script use them).

Console app write some xml to disk when sending mails that the other scripts fetch. This is the first big problem.

Are there any other problems?
The redirection app for example have any contact with the filesystem ? (beside log files of course)

[sfantul]

I'll answer myself .
Openem can be relatively easy split on 3 mashines .
1) Openemm user, filesystem , python and c script, sendmail and openemm interface
2)database
3)Redirection server.
This is important because the redirection server could be put in a location where a good uptime is ensured and also the database.
The interface doesnt need to be online all the time.

[marton]

hi,

does it work? do you run openemm on three machines?

thanks in advance, marton

[sfantul]

yes, I do , i have :
- the database on a mashine
- the interface , python scripts, log files , etc on an other mashine
- redirection (public) server on another mashine

[maschoff]

sfantul, I am curious: How many mails per hour can you send with this configuration?

[sfantul]

Currently i have a mailing list of 600k subscribers and i limit the mailing to 50k per hour (but it can do better).I did this limitation because of the notifications that i receive from some domains that my mailing rate is too high.
But the mailing rate has nothing to do to with this configuration , its all about configuring sendmail (when i started i had a rate of 10k per hour max). I tried a lot of things to make the sending rate higher.

First big breakthrough was to split the queue folder in 10 folders with persistent runners on every queue
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE0
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE1
...
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE9
changing pickdist.py and mailer.sh accordingly and sorting the queue after the host. here are my defines in sendmail.mc:
define(`confQUEUE_LA', `50')dnl
define(`confREFUSE_LA', `51')dnl
define(`confCON_EXPENSIVE',`true')dnl
define(`confTO_ICONNECT', `5s')dnl
define(`confQUEUE_SORT_ORDER', `Host')dnl
define(`confMIN_QUEUE_AGE', `3h')dnl

This was a big improvement, but finally i'm using an extra postfix mailer as a smarthost which give me the results i was talking about.
The disadvantages of this last solution are that u need to also configure an postfix mailer and that the first type of bounce management does not work properlly.

[joedj]

Hi sfantul,

We are interested in doing something similar. Can you tell us what kind of changes you made to pickdist.py to use the multiple queues, or provide a copy? I know this was a while ago now... :)

[dcunniff]

Hi guys,

I'm looking for the same information if it could be provided. I too understand this was a long time ago.

[tpalmieri]

I am trying to host openemm pages, etc on one server and the database on another.

I edited both cms.properties and emm.properties to point to the remote database server, but openemm times out when it tries to connect.

cms.properties:
#####################################################
# Database settings
#####################################################

cmsdb.driverClassName=com.mysql.jdbc.Driver
cmsdb.url=jdbc:mysql://internal.c3psypivq9bi.us-east-1.rds.amazonaws.com:3306/openemm_cms?useUnicode=true&characterEncoding=UTF8&jdbcCompliantTruncation=false
cmsdb.dialect=org.hibernate.dialect.MySQLDialect
cmsdb.username=agnitas
cmsdb.password=openemm
cmsdb.maxCount=30
cmsdb.maxWait=10000
cmsdb.validationQuery=SELECT 1 FROM DUAL

#####################################################
#Remote webservices settings
#####################################################
cms.ccr.url=http://openemm.simpractice.com:8080

emm.properties:
# Property file with OpenEMM settings - targeted at system administrators

###############################################################################
# Database Connection Settings (dataAccessContxt.xml, etc.)
###############################################################################
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://internal.c3psypivq9bi.us-east-1.rds.amazonaws.com:3306/openemm?useUnicode=yes&characterEncoding=UTF-8&useOldAliasMetadataBehavior=true
jdbc.dialect=org.hibernate.dialect.MySQLDialect
jdbc.username=agnitas
jdbc.password=openemm
jdbc.maxCount=30
jdbc.maxWait=10000
jdbc.validationQuery=SELECT 1 FROM DUAL

###############################################################################
# System Default Values
###############################################################################
system.url=http://openemm.simpractice.com:8080
system.logdir=/home/openemm/var/log
system.upload=/tmp
system.updateserver=http://www.openemm.org/
system.mail.host=localhost

###############################################################################
# Feature Default Values
###############################################################################
password.expire.days=-1
fckpath=fckeditor-2.6.4.1
ecs.server.url=http://openemm.simpractice.com:8080

Any help anyone can provide would be greatly appreciared!

Thanks!

[tpalmieri]

Disregard my last post, I got it all working:

Web Server runs the pages
Database Server holds all the data

The web server is running on Amazon EC2, and the Database server is running on Amazon's RDS service.

I'm uploading about 75,000 records now and its going pretty fast (although I have no frame of reference!)

LOVE OpenEMM!

[maschoff]

Perhaps is would be interesting for other OpenEMM users to share your settings (in case you want to disclose your knowhow).

[unhuman]

Pretty sure we're going to explore this as well. I currently envision:
1. OpenEMM itself (UI + Mail sending)
2. Database
3. Bounce Detection
4. Click detection
5. We'll either use Amazon SES or stand up our own STMP server(s). I might add support for multiple configurations so this data can be sent round-robin style.

To note, I'm not using OpenEMMs send logic (with the subprocesses). I'm using my own mail building code (for SES and/or SMTP) and will be writing a client to detect bounces.

We may combine 3 & 4 since they should be lightweight processes, however, we'll be sending out mail in the hundreds of thousands / millions, so, we might want more girth there. We could easily use a VIP to allow us to map multiples of those.

-H

[Infomess]

This is a great post! I was wondering if you could help me.

I have installed OpenEMM 2012 on an EC2 instance and it looks like it's working fine. Now created a RDS instance which I intend to use for the DB. I have
updated the emm and css.properties file as tpalmieri has indicated. OpenEMM works fine, I can logon to CMS but when starting openmm.

It says "Failed to setup database interface".

I have also updated the address from localhost to the rds endpoint on agn.py but still no luck.

Running with the database on the same instance everything is working fine.

Any idea how to debug this?

Many many thanks!