Not so many - there are 3 rooms, 2 of thea are empy and in one there is only 10-20 messages per hour and currently there are only 15 users in that room.
I keep history for last 100 messages - so as you can see server is slightly loaded.
Not so many - there are 3 rooms, 2 of thea are empy and in one there is only 10-20 messages per hour and currently there are only 15 users in that room.
I keep history for last 100 messages - so as you can see server is slightly loaded.
Sounds like a nasty memory leak. I suspect that there are a couple of them in the Openfire core code, but nothing that should have this impact. What plugins are you using? Are you using custom plugins, or third party plugins? Do you have problems restarting one (or more) plugins properly?
Here is a list of plugins we use: Broadcast (Jive Software), Monitoring Service (Jive Software), Registration (Ryan Graham), Search (Ryan Graham).
No custom plugins.
I’ve just tryed to resart all of them and didn’t notice any problems with it.
We have exactly the same problem. We have about 100 User online and we use the Client Control, IM Gateway (with only MSN and ICQ), Monitoring Service and Search Plugins. I use a Mysql Database and Authenticate over our Active Directory. All users are shared to each other in seperate groups. If the memory usage reach 100% i get this error:
2008.10.22 09:41:02 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.OutOfMemoryError: Java heap space
After i restart the service all Users reconnect and work like before. Openfire use at this time only 20MB but the usage increase permanently…
I try to disable all plugins but the memory usage grow up as before…
Everytime I get some error messages like this:
2008.10.22 09:41:49 [org.jivesoftware.openfire.roster.Roster.(Roster.java:177)] Groups ([IT Support]) include non-existent username (info)
And:
2008.10.22 09:32:07 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java:313)
at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:392)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageReceived (AbstractIoFilterChain.java:499)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.fireMessageReceived(Abstra ctIoFilterChain.java:293)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.j ava:228)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcesso r.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProce ssor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
One crash was early in the morning where definitely nobody was online. So its very strange that the memory increase but nobody doing something. We use the server only in our internal network.
It seems like our error, but we’ve got other error messages:
2008.11.14 17:50:09 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)]
java.lang.ArrayIndexOutOfBoundsException
But we use embedded db.
So for the moment the only one solution I’ve got is - to use more stress-stable client software, and restart opernfire every 2-3 days.
Sounds like a pretty vanilla setup.
It would be interesting to generate a heapdump of a Openfire suffering from the problem. That way, you’d be able to analyze it in a tool like JProfiler or JHat. I’m not sure if you’re in a position to create one?
A quick google showed at least two ways of obtaining a heapdump:
java -agentlib:hprof=file=
file,format=b
application; on Control-Break, VM generates a heap dump.-XX:+HeapDumpOnCtrlBreak
; on Ctrl-Break, VM generates a heap dump. (Introduced in Java SE 5.0 update 14.)The way I like to get heapdumps when running out of memory is by adding -XX:+HeapDumpOnOutOfMemoryError. That setting will automatically create a heap dump when Java runs out of memory. I would be very interested in seeing your heap dump. We have been using Openfire on jivesoftware.com for more than 3 months without a single restart. IOW, I’m curious.
– Gato
Please reduce the total size of the heap if you do use ‘on OOM’. Trying to work with a heapdump that’s 128MB is easier than working with one that’s 1024MB
great! I’m going to send it to you after I’ll get it.
Hi,
did you create some thread dumps? To create a stack trace use ´jstack openfire-pid >/tmp/javacore.txt´ and make sure to use different names if you create more than one stack trace. A quick way to count the used threads is ´ps -T -p $pid_of_openfire | wc -l´
http://www.igniterealtime.org/community/thread/35959 could be related to your problem. In any case it would be nice to know that you do not have a similar problem.
You should use a valid path within “-Xloggc:$/WF/logs/gc.log” like “-Xloggc:/tmp/gc.log” otherwise the info will go to STDERR (likely nohup.out).
LG
whats going on? Have you find a solution for your problem? I still have to restart openfire every weekend to prevent a crash. I updated allready to 3.6.2 with no problems but the problem is still there. Currently we’re planning to provide it to all of our employees so it grow up from 100 users up to 250 over the whole world. So i have no free time to restart the service because then every time users have to use it. It would be very nice if we can solve the problem. I try to start some extra debug info and so on.
regards
Lunks
Hi,
I am having the exact problem. My memory usage is 99.7%!!! I’m restarting Openfire every 2 hours as users cannot login. Could someone please help me>!?!?!? I’m not a programmer, so are there any other solutions to this problem? I can’t even log inot the Admin Console. Thanks.
First off, try running the server with no plugins installed. If the server works fine then add the plugins back one at a time and make sure that the server keeps running fine. Eventually you should identify which plugin was causing this problem.
If you are running with no plugins then most likely (if you are running 3.6.2) is that some client is flooding the server with messages? In this case, I would recommend using the load stats plugin to collect information about the MINA queues/buffers. Finally, you should add -XX:+HeapDumpOnOutOfMemoryError to the list of command line parameters when starting Openfire. That new parameter will instruct Java to generate a dump of the memory and store it in a file. This file can then be analyzed to identify what was consuming so much memory.
– Gato
HI!!!
Thank you so much for responding to my cry for help!!! I’ve deleted all the
plugins and I’m still running out of memory. There are 2 processes running:
openfire.exe that’s avg. about 6,908K worth of CPU Memory and openfired.exe
that’s averaging a whopping 112,816K worth of memory. I believe the second
one is the problem. Since we’ve upgraded to 3.6.2, we have been having a
lot of problems with memory. I do believe that I need to run the
HeapDumpMemoty command you sent me, but could you tell me EXACTLY where to
go to add this to the command line and do I need to add the “**”. And where
do I go to get the load stats plugin?? Is this the same as the monitoring
plugin? Thanks so much again!!
Regards,
Nicolyn C. Parks
Business Systems Support
IT Department
Belize City, Belize
Phone: 011-501-222-4757 ext. 274
Fax: 011-501-222-4759
CONFIDENTIALITY NOTICE
The information contained in this message and any attachments are intended
solely for the use of the individual or entity to whom they are addressed.
It may contain privileged or confidential information. If you are not the
intended recipient you must not copy, distribute or take any action in
reliance on it. If you have received this e-mail in error please notify the
sender and delete the email. Thank You.
Hi Nicoly,
http://www.igniterealtime.org/builds/openfire/docs/latest/documentation/install- guide.html reads
Advanced users may wish to pass in parameters to the Java virtual machine (VM) to customize the runtime environment of Openfire. You can do this by creating vmoptions files in the bin/ directory of your Openfire installation. For the Windows service, you’d create a new text file called openfire-service.vmoptions. Each parameter to the VM should be on a new line of the file. For example, to set the minimum heap size to 512 MB and max VM heap size to 1024 MB, you’d use:
-Xms512m
-Xmx1024m
To create parameters for the normal launcher, create a file called openfired.vmoptions (since the openfire.exe launcher invokes the openfired.exe executable to actually start the server).
So you may want to increase the Java memory to 256, 512 or 1024 MB by adding a line like
-XmX512m
to openfired.vmoptions.
LG
Hi LG,
That seemed to have helped. Can I just confirm the name of the file??
Should it be openfired.vmoptions or openfired.vmoptions.txt?? Also, the
little meter that shows the Java Memory is still reading 99.3%. How do I
decrease that? Also, how do I decrease the size of the Current Search
Index, the Archived Message Count and the Archived Conversation Count??
Thanks.
Regards,
Nicolyn C. Parks, MBA
Business Systems Support
IT Department
Belize City, Belize
Phone: 011-501-222-4757 ext. 274
Fax: 011-501-222-4759
CONFIDENTIALITY NOTICE
The information contained in this message and any attachments are intended
solely for the use of the individual or entity to whom they are addressed.
It may contain privileged or confidential information. If you are not the
intended recipient you must not copy, distribute or take any action in
reliance on it. If you have received this e-mail in error please notify the
sender and delete the email. Thank You.
Hi Nicolyn,
the file name must be openfired.vmoptions without .txt. You need to restart Openfire to read the new settings. The JVM does not allow to alter the max memory (-Xmx) setting during runtime.
As fas as I know one needs to run SQL statements to delete archived messages. Are you using the embedded database or an external one?
LG
I think I got it to work as Openfire is not crashing anymore. The only
thing is that the archive is not working. The message is saying that the
Conversation was not archived. I’m using the embedded dbase. Where or how
do I execute the SQL statements?? I just came across it in a msg that was
previously posted…It’s shown below.
DELETE ofMessageArchive.*
FROM ofMessageArchive, ofConversation
WHERE ofMessageArchive.conversationID = ofConversation.conversationID
AND ofConversation.startDate <= 1192224791872;
DELETE ofConParticipant.*
FROM ofConParticipant, ofConversation
WHERE ofConParticipant.conversationID = ofConversation.conversationID
AND ofConversation.startDate <= 1192224791872;
DELETE ofConversation.*
FROM ofConversation
WHERE ofConversation.startDate <= 1192224791872;
Regards,
Nicolyn C. Parks
Business Systems Support
IT Department
Belize City, Belize
Phone: 011-501-222-4757 ext. 274
Fax: 011-501-222-4759
CONFIDENTIALITY NOTICE
The information contained in this message and any attachments are intended
solely for the use of the individual or entity to whom they are addressed.
It may contain privileged or confidential information. If you are not the
intended recipient you must not copy, distribute or take any action in
reliance on it. If you have received this e-mail in error please notify the
sender and delete the email. Thank You.
Hi Nicolyn,
you should open a new thread for new questions as this has usually little to do with memory. But in your case it has a lot to do with it. The embedded database uses “MEMORY TABLES”, this means: “Memory tables are the default type when the CREATE TABLE command is used. Their data is held entirely in memory but any change to their structure or contents is written to the <dbname>.script
file. The script file is read the next time the database is opened, and the MEMORY tables are recreated with all their contents.” (ref: http://hsqldb.org/doc/guide/ch01.html)
As more log messages you store there the more memory Openfire will consume. So it’s a bad idea to use the monitoring plugin with the embedded database.
To edit the database you need to stop Openfire. Make a backup of the embedded-db folder. Then you can use the bin/extra/embedded-db-viewer.sh or .cmd script to access the database via SQL.
You can also use vi or notepad to open the embedded-db/openfire.script and embedded-db/openfire.log files and delete there random SQL INSERT statements.
If you like the risk you can change the three table from MEMORY to CACHED tables. Anyhow you can no longer edit content of CACHED tables with vi / notepad.
LG
I have no headdump yet but i create a stack with jstack and a seperate file with a reduced stack with only the threads who use more cpu time. Maybe calling looping threads?
I alsa include the script wich i create the both stacks… not perfekt but it works I think
regards,
Philip
get_threads (751 Bytes)
openfire_loopingthreads.txt (22715 Bytes)
openfire_stack.txt (109473 Bytes)