Memory usage on Openfire 3.6.3

Hi!

I am using openfire 3.6.3 on a CentOS 5.2 x64 server, running with JRE 6 U12 (java 1.6.0_12) x64. Openfire was installed in /opt from the tarball. It runs fine, however, about once a week, the server starts behaving very slow, openfire will fail to accept new client connections, the administration web interface is very slow (about a minute or more just to open the user/password prompt). There I usually have to stop openfire (I have to stop it several times since it will say “Timeout. Daemon did not shutdown yet.”. After a few times, it will stop successfully. Then I restart it and it will go OK for about a week. My system has about 20-30 simultaneous users, the server has 2GB RAM assigned (It’s a virtual machine running under VMware ESX 3.5) and the openfire.sh.vmoptions file contains only this line: “-Xms256m -Xmx1024m” (no quotes). All the clients use spark 2.5.8.

Any help on optimizing my openfire installation is welcome.

Thanks in advance.

See http://www.igniterealtime.org/community/docs/DOC-1033

Looks like you’ll need to analyze some debugging information. Suggestion is to get a Java heap dump and use jhat to see what is consuming resources.

Hi,

which database and which plugins are you using?

LG

Hi!

The database is mySQL, the one that comes with CentOS:

[root@messenger ~]# rpm -qa | grep mysql
mysql-5.0.45-7.el5
mysql-server-5.0.45-7.el5
[root@messenger ~]#

and the only plugins I use are:

asterisk (not configured, but active)

broadcast

clientControl

gateway (configured to connect only to MSN network, only for certain users)

search

Now, it’s possibly very unrelated, but my users complain that the gateway plugin keeps disconnecting very frequently (I use it only to access the MSN Messenger network). It’s just in case it matters.

Regards.

Hi,

you may want to try to restart one plugin after the other (wait 5 minutes beetween restarts) and monitor the memory usage on the server information page.

If you have the JDK installed you could start jconsole (requires an X-server) or jconsole.exe and connect to the local Openfire JVM. There the “memory” tab allows you to perform a GC to free non-allocated memory.

LG

Thanks, I have installed only the JRE, not the full JDK, and the server does not have X server at all, so running graphical programs will not work I’m afraid. However I’ll try installing the JDK and doing some of the restaring plugins thing. I disabled the asterisk plugin (deleted it) since it wasn’t being used anyway. I’ll see if it gets better. BTW: what should be the memory usage of openfire for only 30 concurrent users? the server is dedicated for it (no other applications installed) and the DB y MySQL 5.0.

Thanks!

Hi,

for 30 users 30 MB are usually fine, even when using plugins. Either you do a lot of messaging or a plugin has a problem.

At least at night when no one is online the memory usage should drop to very low values.

Without JDK you could try VisualVM on the client side and run Openfire with JMX debugging enabled. See https://visualvm.dev.java.net/ and https://visualvm.dev.java.net/jmx_connections.html for details.

LG

Thanks for your suggestion LG. I just wanted to point that I did two things in the server and memory usage seem to have improved:

  1. I replaced JRE 1.6 U12 with JDK 1.6 U13 (so I have all the JDK tools to debug the memory usage)

  2. I disabled IPv6 in the server

Now, the server is using only 700MB of RAM, and openfire console reports usage of about 250MB (It used to be about 900MB, and the server used full 2GB of RAM with almost 1GB swap used) so, it seems these things had some effect. However, I’ll continue monitoring the server and post again if the issue arises.

Regards.

Hi!

I did all the things suggested in this link: http://www.igniterealtime.org/community/docs/DOC-1033 but the machine still has to be restarted about every third day, otherwise it will stop accepting requests and no client can connect. It produces logs, but I am not sure how to analyze them, or what do they mean.

The server runs CentOS Linux 5.3 64 bits, with JDK 1.6.0U13, and openfire 3.6.3. I have not upgraded to 3.6.4 because it mentions nothing about memory usage.

Thanks!

I would try analysing the heap dump resulting from adding that JVM flag using jhat. This will provide some better clues. See which objects are consuming all of the memory. Another suggestion is use less JVM memory, and use a tool like pmem to plot it’s consumption. When knowing it will soon crash a priori, issue kill -QUIT to see what the state of OPF is. See also the guide on the web, “Java Troubleshooting and Diagnostic Guide”.

HTH

Post the tail of info.log, warn.log and error.log so we can see if it says anything useful.

Thanks for replying. Here are the last 50 lines of every log:

error.log:

at org.jivesoftware.openfire.gateway.protocols.msn.MSNSession.syncUsers(MSNSession .java:274)

at org.jivesoftware.openfire.gateway.protocols.msn.MSNListener.contactListInitComp leted(MSNListener.java:276)
at net.sf.jml.impl.AbstractMessenger.fireContactListInitCompleted(AbstractMessenge r.java:481)
at net.sf.jml.protocol.outgoing.OutgoingPNG.receivedResponse(OutgoingPNG.java:65)
at net.sf.jml.protocol.MsnSession$1.messageReceived(MsnSession.java:112)
at net.sf.cindy.impl.AbstractSession$6.doRun(AbstractSession.java:341)
at net.sf.cindy.impl.AbstractSession$DispatchObject.run(AbstractSession.java:395)
at net.sf.cindy.impl.SimpleDispatcher.dispatch(SimpleDispatcher.java:35)
at net.sf.cindy.impl.AbstractSession.dispatch(AbstractSession.java:249)
at net.sf.cindy.impl.AbstractSession.dispatchMessageReceived(AbstractSession.java: 335)
at net.sf.cindy.impl.StreamChannelSession.recognizeMessageAndDispatch(StreamChanne lSession.java:77)
at net.sf.cindy.impl.StreamChannelSession.readFromChannel(StreamChannelSession.jav a:53)
at net.sf.cindy.impl.ChannelSession.onReadable(ChannelSession.java:212)
at net.sf.cindy.impl.ChannelSession.onEvent(ChannelSession.java:134)
at net.sf.cindy.impl.SocketSession.onEvent(SocketSession.java:119)
at net.sf.cindy.impl.SimpleEventGenerator.processKey(SimpleEventGenerator.java:212 )
at net.sf.cindy.impl.SimpleEventGenerator.run(SimpleEventGenerator.java:192)
at net.sf.cindy.impl.SimpleEventGenerator.access$000(SimpleEventGenerator.java:54)
at net.sf.cindy.impl.SimpleEventGenerator$1.run(SimpleEventGenerator.java:115)
2009.05.08 08:05:52 [org.jivesoftware.openfire.ldap.LdapGroupProvider.getGroup(LdapGroupProvider.ja va:91)]
org.jivesoftware.openfire.group.GroupNotFoundException: Groupname Clientes not found
at org.jivesoftware.openfire.ldap.LdapManager.findGroupDN(LdapManager.java:855)
at org.jivesoftware.openfire.ldap.LdapManager.findGroupDN(LdapManager.java:782)
at org.jivesoftware.openfire.ldap.LdapGroupProvider.getGroup(LdapGroupProvider.jav a:83)
at org.jivesoftware.openfire.group.GroupManager.getGroup(GroupManager.java:278)
at org.jivesoftware.openfire.group.GroupManager.getGroup(GroupManager.java:257)
at org.jivesoftware.openfire.roster.RosterItem.setGroups(RosterItem.java:361)
at org.jivesoftware.openfire.gateway.BaseTransport.addOrUpdateRosterItem(BaseTrans port.java:1330)
at org.jivesoftware.openfire.gateway.BaseTransport.addOrUpdateRosterItem(BaseTrans port.java:1257)
at org.jivesoftware.openfire.gateway.BaseTransport.addOrUpdateRosterItem(BaseTrans port.java:1415)
at org.jivesoftware.openfire.gateway.BaseTransport.syncLegacyRoster(BaseTransport. java:1568)
at org.jivesoftware.openfire.gateway.protocols.msn.MSNSession.syncUsers(MSNSession .java:274)
at org.jivesoftware.openfire.gateway.protocols.msn.MSNListener.contactListInitComp leted(MSNListener.java:276)
at net.sf.jml.impl.AbstractMessenger.fireContactListInitCompleted(AbstractMessenge r.java:481)
at net.sf.jml.protocol.outgoing.OutgoingPNG.receivedResponse(OutgoingPNG.java:65)
at net.sf.jml.protocol.MsnSession$1.messageReceived(MsnSession.java:112)
at net.sf.cindy.impl.AbstractSession$6.doRun(AbstractSession.java:341)
at net.sf.cindy.impl.AbstractSession$DispatchObject.run(AbstractSession.java:395)
at net.sf.cindy.impl.SimpleDispatcher.dispatch(SimpleDispatcher.java:35)
at net.sf.cindy.impl.AbstractSession.dispatch(AbstractSession.java:249)
at net.sf.cindy.impl.AbstractSession.dispatchMessageReceived(AbstractSession.java: 335)
at net.sf.cindy.impl.StreamChannelSession.recognizeMessageAndDispatch(StreamChanne lSession.java:77)
at net.sf.cindy.impl.StreamChannelSession.readFromChannel(StreamChannelSession.jav a:53)
at net.sf.cindy.impl.ChannelSession.onReadable(ChannelSession.java:212)
at net.sf.cindy.impl.ChannelSession.onEvent(ChannelSession.java:134)
at net.sf.cindy.impl.SocketSession.onEvent(SocketSession.java:119)
at net.sf.cindy.impl.SimpleEventGenerator.processKey(SimpleEventGenerator.java:212 )
at net.sf.cindy.impl.SimpleEventGenerator.run(SimpleEventGenerator.java:192)
at net.sf.cindy.impl.SimpleEventGenerator.access$000(SimpleEventGenerator.java:54)
at net.sf.cindy.impl.SimpleEventGenerator$1.run(SimpleEventGenerator.java:115)

warn.log:

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:619)
2009.05.07 16:56:50 Failed admin console login attempt by administrator from 10.10.15.89
2009.05.07 17:34:58 Error occured while processing packet:
java.lang.IllegalStateException: session is not available
at net.sf.cindy.impl.ChannelSession.write(ChannelSession.java:329)
at net.sf.jml.protocol.MsnSession.sendAsynchronousMessage(MsnSession.java:181)
at net.sf.jml.impl.BasicMessenger.send(BasicMessenger.java:119)
at net.sf.jml.impl.AbstractMessenger.send(AbstractMessenger.java:141)
at net.sf.jml.impl.SimpleMessenger.newSwitchboard(SimpleMessenger.java:36)
at net.sf.jml.impl.BasicMessenger.sendText(BasicMessenger.java:223)
at org.jivesoftware.openfire.gateway.protocols.msn.MSNSession.sendMessage(MSNSessi on.java:408)
at org.jivesoftware.openfire.gateway.BaseTransport.processPacket(BaseTransport.jav a:262)
at org.jivesoftware.openfire.gateway.BaseTransport.processPacket(BaseTransport.jav a:201)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.MessageRouter.route(MessageRouter.java:97)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:72)
at org.jivesoftware.openfire.net.StanzaHandler.processMessage(StanzaHandler.java:3 55)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processMessage(ClientStanzaHa ndler.java:91)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:209)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:619)

info.log

2009.05.05 09:18:16 User Login Failed. PLAIN authentication failed
2009.05.05 09:18:35 User Login Failed. PLAIN authentication failed
2009.05.05 09:18:41 User Login Failed. PLAIN authentication failed
2009.05.05 09:18:42 User Login Failed. PLAIN authentication failed
2009.05.05 09:20:36 User Login Failed. PLAIN authentication failed
2009.05.05 09:20:46 User Login Failed. PLAIN authentication failed
2009.05.05 09:20:52 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:06 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:11 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:26 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:31 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:39 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:42 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:48 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:50 User Login Failed. PLAIN authentication failed
2009.05.05 09:21:56 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:05 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:12 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:39 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:47 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:53 User Login Failed. PLAIN authentication failed
2009.05.05 09:22:58 User Login Failed. PLAIN authentication failed
2009.05.05 14:22:32 User Login Failed. PLAIN authentication failed
2009.05.05 14:22:43 User Login Failed. PLAIN authentication failed
2009.05.05 16:00:09 User Login Failed. PLAIN authentication failed
2009.05.05 16:07:50 User Login Failed. PLAIN authentication failed
2009.05.05 16:08:00 User Login Failed. PLAIN authentication failed
2009.05.05 16:49:31 User Login Failed. PLAIN authentication failed
2009.05.05 16:49:40 User Login Failed. PLAIN authentication failed
2009.05.05 16:49:48 User Login Failed. PLAIN authentication failed
2009.05.05 16:50:02 User Login Failed. PLAIN authentication failed
2009.05.05 17:33:23 Packet sent to unreachable address



2009.05.06 10:58:06 User Login Failed. PLAIN authentication failed
2009.05.06 16:37:08 Publish-Subscribe domain: pubsub.gtscolombia.com
2009.05.06 16:37:08 Multi User Chat domain: conference.gtscolombia.com
2009.05.06 16:37:09 Openfire 3.6.3 [May 6, 2009 4:37:09 PM]
2009.05.06 16:37:12 Admin console listening at:
http://gtscolombia.com:9090
https://gtscolombia.com:9091
2009.05.06 16:37:13 Starting transport service: msn
2009.05.06 16:37:13 Started plain (unencrypted) socket on port: 5222
2009.05.06 16:37:13 Started SSL (encrypted) socket on port: 5223
2009.05.07 08:38:20 User Login Failed. PLAIN authentication failed
2009.05.07 11:12:33 User Login Failed. PLAIN authentication failed
2009.05.07 11:12:38 User Login Failed. PLAIN authentication failed
2009.05.07 11:12:43 User Login Failed. PLAIN authentication failed
2009.05.07 15:47:20 User Login Failed. PLAIN authentication failed

Thanks for any hint. There is also a garbage collector log, which I can post if it seems like it would be useful.

Re: your error log. I don’t see what the exception is associated with the error log.

Re: the plugins you are using. Also it looks like you’re using the openfire gateway plugin, 1.2.4d. That project has migrated to Kracken (http://kraken.blathersource.org/).

Re: Isolating the root cause. Another clue is the try and use a vanilla default installation of openfire, and see if the issue goes away, then progressively make customizations until you identify the culprit.

I’m running Openfire 3.6.4 in an ESX VM. Openfire memory usage is ballooning out of control. I’ve installed JDK 1.6.0_16 and VisualVM. VisualVM cannot profile memory usage of openfire. Jconsole cannot connect to the startup.jar process.

I’ve modified the heap settings as described at http://www.igniterealtime.org/community/docs/DOC-1033 and I’ve got heap dumps when memory ran out from three different instances. If one of the Openfire developers will email me, I can send the links to the help dumps.

It’s better to start a new thread. Which plugins installed, which OS? Try running Openfire in debug mode. Set the JVM args to dump heap on crash/failure (I forget the exact arg), and analyse it with Jhat.

P.S. Start a new thread!