This Question is Answered

30 Replies Last post: Oct 8, 2008 2:33 AM by Ben Lye  
Coolcat KeyContributor 585 posts since
Mar 19, 2007
Currently Being Moderated

May 2, 2008 5:24 AM

Openfire 3.5.1 / 100% CPU / DoS on it self...

Since a few days I saw a extremely high count of XMPP-packets send by Openfire in my statistics. The is no abnormal traffic on the network device, nothing in the logs, only 100% CPU load.

Today I wrote a small plugin, which is able to count packets per user per time. I found out, that Openfire does send 7000-8000 packets per second from an JID that is offline to another unavailable JID.

 

  • The first JID is a normal user account which is offline. I'm sure about this, because its my own account and I'm online with another resource.

  • The second JID is chat.yahoo.jabber.rwth-aachen.de, which belongs to our PyYIMt transport. This JID does is also unavailable, because it's the yahoo MUC, which does not work.

 

Server runs stable, so I will wait until tomorrow morning (08:00 CEST) until I try a restart.

 

I will try to investigate more about this weird bug.

Daniel Henninger Jiver 2,933 posts since
Aug 10, 2005
Currently Being Moderated
May 2, 2008 7:52 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hrm.  Does the packet appear to be dancing back and forth between the two entities?  (even though they aren't available)  I've seen this occur before, but I was never able to track down a way to reproduce it, and it vanished and never showed up for me again.  Are you able to easily reproduce it?

Daniel Henninger Jiver 2,933 posts since
Aug 10, 2005
Currently Being Moderated
May 2, 2008 8:18 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

=)  But with a fresh restart, are you able to easily trigger it "going nuts" so to speak again?

 

So one way...  so its' as if it tries to route the packet, fails, tries again, fails, over and over again?

Daniel Henninger Jiver 2,933 posts since
Aug 10, 2005
Currently Being Moderated
May 2, 2008 9:37 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Ah, of course  =)  (not restarting right now)  I misjudged that this was on a test server.

LG KeyContributor 4,984 posts since
Dec 13, 2005
Currently Being Moderated
May 3, 2008 12:58 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hi Coolcat,

 

if you are on Linux you may be able and want to identify the thread which is causing the high cpu usage. See JVM Settings and Debugging.

 

LG

LG KeyContributor 4,984 posts since
Dec 13, 2005
Currently Being Moderated
May 3, 2008 5:31 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hi,

 

you need a stack trace, kill -3 PID should produce one. Windows guys may want to use the Stacktrace tool. One may want to take a look at http://www.adaptj.com/main/tracehowtos#ht1

 

LG

LG KeyContributor 4,984 posts since
Dec 13, 2005
Currently Being Moderated
May 3, 2008 7:00 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hi,

 

my JVM 1.6.0_02 writes the stack trace to standard out, Openfire will likely log this in nohup.out.

 

LG

LG KeyContributor 4,984 posts since
Dec 13, 2005
Currently Being Moderated
May 3, 2008 9:32 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hi,

 

do you have enough memory available? The JVM thread with 3:33 hours seems to be much to high.

 

LG

LG KeyContributor 4,984 posts since
Dec 13, 2005
Currently Being Moderated
May 4, 2008 2:49 AM in response to: Coolcat
Re: Openfire 3.5.1 / 100% CPU / DoS on it self...

Hi,

 

do you write a GC log? I wonder if one thread is allocation and releasing objects in a loop and thus the garbage collector runs every second or more often. Of if your PermSize is too small causing also a lot of useless garbage collections.

 

LG