Roster disappearing

Hi all,

We’'re currently trying out Spark (2.0.8) and Wildfire (3.2.3-1 on Linux) within our company.

We’'re using active directory/LDAP as the user database, with groups managed within Wildfire (not LDAP). Users are added to a Wildfire group based on their primary role (eg: Helpdesk, Management, Sales, etc), and contact lists are shared.

What we are finding is that after Wildfire has been running for a few days, users rosters in spark will disappear at their next login. This problem is fixed by restarting wildfire or clearing the caches (not sure which specific cache, as we’'ve been clearing all of them). Once this has been done, contact lists will reappear at the next login until they disappear again a few days later.

I was wondering if anybody else has experienced this problem before, and what can be done to fix it?

Thanks,

Shaun

Just to update (and as a bit of a bump) - this problem is fixed by clearing the roster cache and logging in to spark again until it happens again a day or so later.

Unfortunately no solution, but issue JM-1016 to track this problem.

LG

Thanks LG.

On a related note, like that other thread, we’'re running CentOS 4 (equivalent of RHEL 4, as used in the other thread).

Just to bump this thread.

The issue is still occurring with Openfire 3.3.0-1 on Linux (with groups now managed by Openfire).

I completely removed and reinstalled the software, but it’'s still occurring (and as before, clearing the caches fixes it temporarily).

I’'m going to try installing the software on Windows 2003 x64 to see if that makes a difference.

We have this issue as well. We have a mixed deployment of spark versions from 2.0.8 to 2.5.1 and we have now upgraded Wildfire from 3.2.2 to 3.2.4 and are still having the issue. We are Windows 2003 AD/Ldap.

I am seeing this same problem as well and clearing the caches also works for me. Like everyone else it is only a temporary fix. I am following progress of this issue in the bug tracker but hope that posting helps to expedite a fix.

Server Info:

Openfire 3.3.0 running on Solaris 10 x64

MySql 5 as the database on a remote machine

Active Directory SBS 2003 for the Users and Groups Authentication.

Java Version on server:

java version “1.5.0_06”

Java™ 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)

Java HotSpot™ Server VM (build 1.5.0_06-b05, mixed mode)

Client Info:

Psi 10 - both windows and linux

JAJC - windows

This is a great jabber server…just would like to see this problem resolved…as well as most of my users.

Hey guys,

Since you have seen this problem when using Spark and other clients then we can say that this is a server problem. From your posts it seems that it only happens when users are being read from LDAP. Groups may or many not be coming from LDAP.

Have you checked in the log files for related errors? I’'m wondering if Openfire is failing to load data from LDAP (sometimes), prints an error and continues but with an empty group.

Regards,

– Gato

Gato,

This is the only error I see in my log that I can not explain, could be unrelated. I comb my logs daily and I dont see anything other then this and vcard errors due to people trying to change them.

java.lang.NullPointerException

at org.jivesoftware.wildfire.ldap.LdapUserProvider.findUsers(LdapUserProvider.java :490)

at org.jivesoftware.wildfire.user.UserManager.findUsers(UserManager.java:272)

at org.jivesoftware.wildfire.plugin.SearchPlugin.processSetPacket(SearchPlugin.jav a:282)

at org.jivesoftware.wildfire.plugin.SearchPlugin.handleIQ(SearchPlugin.java:196)

at org.jivesoftware.wildfire.plugin.SearchPlugin.processPacket(SearchPlugin.java:1 48)

at org.jivesoftware.wildfire.component.InternalComponentManager$RoutableComponent. process(InternalComponentManager.java:490)

at org.jivesoftware.wildfire.IQRouter.handle(IQRouter.java:250)

at org.jivesoftware.wildfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.wildfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.wildfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:131)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

We are seeing this problem too. Just upgraded from Wildfire 3.0.0 to Openfire 3.3.0. We have a mixed spark install from 2.0.5 to 2.5.1.

Gato,

Thanks for replying. I am seeing a few things in my logs. Fist thing I see is fairly common and occurs often:

  • BEGIN NESTED EXCEPTION **

com.mysql.jdbc.CommunicationsException

MESSAGE: Communications link failure due to underlying exception:

  • BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2739)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.openfire.PrivateStorage.get(PrivateStorage.java:166)

at org.jivesoftware.openfire.handler.IQPrivateHandler.handleIQ(IQPrivateHandler.ja va:66)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:48)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:300)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:132)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

This next entry happens less often but it looks like it may have something to do with the roster:

  • BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2739)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.openfire.PrivateStorage.get(PrivateStorage.java:166)

at org.jivesoftware.openfire.handler.IQPrivateHandler.handleIQ(IQPrivateHandler.ja va:66)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:48)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:300)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:132)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

Last packet sent to the server was 1 ms ago.

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2757)

at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2650)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1581)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1695)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3004)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1128)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1222)

at org.jivesoftware.openfire.PrivateStorage.get(PrivateStorage.java:166)

at org.jivesoftware.openfire.handler.IQPrivateHandler.handleIQ(IQPrivateHandler.ja va:66)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:48)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:300)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:132)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

  • END NESTED EXCEPTION **

at com.mysql.jdbc.Connection.checkClosed(Connection.java:1844)

at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4306)

at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4272)

at org.jivesoftware.database.AbstractConnection.prepareStatement(AbstractConnectio n.java:36)

at org.jivesoftware.openfire.group.Group.getSharedGroupsNames(Group.java:76)

at org.jivesoftware.openfire.group.GroupManager.getSharedGroups(GroupManager.java: 284)

at org.jivesoftware.openfire.roster.RosterManager.getSharedGroups(RosterManager.ja va:167)

at org.jivesoftware.openfire.roster.Roster.(Roster.java:93)

at org.jivesoftware.openfire.roster.RosterManager.getRoster(RosterManager.java:92)

at org.jivesoftware.openfire.user.User.getRoster(User.java:289)

at org.jivesoftware.openfire.handler.IQRosterHandler.manageRoster(IQRosterHandler. java:200)

at org.jivesoftware.openfire.handler.IQRosterHandler.handleIQ(IQRosterHandler.java :105)

at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:48)

at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:300)

at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:104)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:67)

at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:289)

at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)

at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:153)

at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:132)

at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:703)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:62)

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:200)

at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:362)

at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:54)

at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:800)

at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :266)

at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:326)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java: 650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

2007.05.01 09:24:54 org.jivesoftware.openfire.roster.RosterItemProvider.getItems(RosterItemProvider. java:313) Internal server error

java.sql.SQLException: No operations allowed after connection closed.Connection was implicitly closed due to underlying exception/error:

  • BEGIN NESTED EXCEPTION **

Thanks looking into this.

Message was edited by: mmelbert

Message was edited by: mmelbert

Gato,

Is there any more information that I can provide to help expedite a resolution to this? Is the information provided what you needed? Let me know.

-Matt

Hi all. I’‘ve been experiencing the same problem. I think i’'ve solved it by seting database connection timeout to less than that on the server.

Seems that the openfire’'s database connection provider does not handle server closed connections well. Perhaps there are connection in the pool that are used when they are in fact closed…

Hope this helps… Regards

Juan.

How did you solve this problem exactly?

What did you modify?

Thanks in advance,

-Matt

Hi Matt

I’‘m not sure if this will resolve the problem. Until monday i can’'t make all the tests i want to do. But if you or someone else try it and works… problem solved (please, let us know )

I saw the same exception that you posted over and over again in my openfire server. I’‘ve shared groups stored in a mysql server and realized that the timeout set in mysql for idle connections was lower than that set in openfire configuration file (by default one day). So i changed connetionTimeout property in openfire.xml to a value less than mysql’'s wait_timeout. I had not much time to test it but seemed that worked fine.

It seems to me that problem is that if mysql closes an idle connection from openfire connection pool, the latter doesn’‘t realize of it being closed and tries to reuse it, resulting in an exception. But i’'m not sure of that… ok?

Regards,

Juan.

Hi all again.

All things go ok with the configuration i’'ve written in the previous post.

Good luck!!

Juan.

Juan,

Thanks for replying. I changed my connetionTimeout value to 0.01 to get under the MySQLl timeout. This does seem to have fixed the issue. I have not seen a single error in the error.log file since I made the change. I am going to keep my eye on this for another day or so before I declare this fixed…but it is looking good so far.

Thanks again for the help

-Matt

I am using the internal database, and I dont see such timeout settings…

After a week things here have been running very smoothly. I have not had to clear my cache or restart Openfire since making my connetionTimeout value 0.01. Thanks for the post Juan and I hope that it helps many others.

-Matt

I am still having the issue, and I use in builtin database and I don’'t see where I can manipulate the settings mentioned.

Anyone know where I can go to do it?