A General error occurred

[lgnch]

I have everything running nicley on the server. The machine is specd at dual Xeon processors with 16GB ram. All requirements are covered well above teh minimum specs and requirements for Linux. redhat Enterprise in operation.

The systems performs seemlessly until I put above 300k users in the database. When the database hits this limit the recipients Overview page throws an error as follows:

! A general error occurred. Please check your input.

Loading stopped

I have checked the data, all is valid and clean in MySql PHPAdmin.

I have increased the java stack size to 4GB and locked innodb to 5GB of memory usage. No swap has been hit. Can someone advise what causes this error so i can track it? I need to get 15 million rows to work on the server. I fthis is not possible then I need to understand the limit so I can duplicate servers.

Thanks in advance for your help.

[maschoff]

Please have a look at Tomcat's log catalina.out.

[lgnch]

All works perfectly until I upload above 300k records. Here is the catlina log

SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1097)
at org.apache.catalina.core.StandardService.start(StandardService.java:457)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Feb 27, 2012 5:01:43 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 552 ms

It says that port 8080 is in use, assuming this is by OpeneEMM, but the site works so i am unsure what this means. No other errors are reported.

[lgnch]

When I do a netstat it shows me that port 8080 is bound to java

tcp 0 0 :::8080 :::* LISTEN 30891/java

[lgnch]

Oh, nevermind I found the error. It was in the options, I had a typo in the max memory. DOH ! Thanks for the help

[lgnch]

Nope its back *sigh*

Seems it is a Tomcat problem though as it happens everytime the database is increased. I will need to read up on it. Any suggestions will be greatly appreciated however.

[lgnch]

Well I have checked everything for tomcat and I cannot see whats causing it. I can keep uploading no problem but canot view recipients. Clearly something is not quite right but I am stuck where to look next as the error logs show nothing.

I have also found that the args= options in http.conf are overridden by the same -J-Xms setting in core.sh. Fixed this by removing the http.sh args and updating the core.sh ones so that 4GB is available to JAVA. Checked the processes with ps-ax and confirmed teh new settings are active.

However, the problem still persists.

Any ideas?

[maschoff]

Why don't you slice and dice your import files into chunks of less than 300k lines? Our default setting in emm.properties for import.maxRows is 60,000 to limit usage of ressources.

[lgnch]

Yep that was my idea too. Did that and imports are not an issue. It is when you click recipients then overview that it fails. The overview will not list users when it gets over 400k users.

As said my issue is not import but viewing users that have been uploaded into teh database.

[maschoff]

Did you adjust property recipient.maxRows in file emm.properties?

[lgnch]

Yep. Set it to 20million. No change.

[lgnch]

Here is a snippet of the java statement from ps -ax

java -Xms4096m -Xmx8192m -Xss1m -Dresin.home=/home/openemm -Dserver.root=/home/openemm/webapps/core....

So I know the memory is being allocated correctly. Although I am unsure what the Xss1m is.

Here is the emm.properties file

# max number of recipients in database
recipient.maxRows=20000000
# max size of bulk import
import.maxRows=500000

So I cannot see why the overview does not list the users based on all the forums and docs you have written.

I can upload 100000 users at a time, just cant view them after 400k.

[maschoff]

What if you set filters on the recipient page (since you probably don't want to browse 20 mio. recipients, anyway)? Does the page work for you if you narrow down your search?

[lgnch]

I do not get the ability to do so as it just errors. Or do you mean edit the java code of Openemm?

The page I have trouble with does not provide the ability to edit the search before execution. If it did this would not be an issue. Can you advise which file executes the recipient.do command and what paramater can be changed to prevent it from doing an immediate execute?

[maschoff]

Ok, got it. My hunch: Due to the long recipient list the JVM runs out ot memory and Tomcat restarts. We will have a fix in OpenEMM 2012 which limits the list down to a maximum (set via company_tbl) so that a memory shortage will not occur.