This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (5 pts)
16 Replies Last post: Feb 11, 2008 3:10 AM by Guus der Kinderen  
vovans Bronze 5 posts since
Dec 20, 2007
Currently Being Moderated

Dec 20, 2007 2:36 AM

Problem with cyrillic in messages

 

Good afternoon!

 

 

While using Openfire we had some problems in messages with cyrillics. Some symbols (cyrillic) displayed incorrectly. It looks as one entered cyrillic symbol, after the sending, replaced with two rectangle symbols. It happens not more than for two symbols per message. Bad symbol each time different. And if you repeat your message - this symbol will display correct, but possible another symbol will play this joke again.

 

The same results for gentoo(Java Version: 1.6.0_03 Sun Microsystems Inc. -- Java HotSpot(TM) Server VM) and for debian testing (1.6.0 Sun Microsystems Inc. -- Java HotSpot(TM) Server VM)

 

 

Attachments:
LG KeyContributor 5,051 posts since
Dec 13, 2005
Currently Being Moderated
Dec 20, 2007 3:09 PM in response to: vovans
Re: Problem with cyrillic in messages

Hi,

 

this sounds a lot like JM-1175. Or did you already downgrade the MINA version?

 

LG

Gaston Dombiak Jiver 3,803 posts since
Sep 26, 2001
Currently Being Moderated
Dec 21, 2007 4:59 PM in response to: vovans
Re: Problem with cyrillic in messages

Hey vovans,

 

I would like to confirm that this is a MINA problem so I can report it to MINA developers. If you no longer see this problem when using an old version of MINA then let us know and we will talk with MINA devs.

 

Thanks,

 

  -- Gato

Eugene Kinyapin Bronze 4 posts since
Dec 22, 2007
Currently Being Moderated
Dec 22, 2007 1:17 AM in response to: vovans
Re: Problem with cyrillic in messages

 

Confirm.

 

 

Built openfire with mina 1.1.0 and everything is clear (tested with 2 clients only sending each other large messages in russian).

 

 

Built with mina 1.1.1 and got this bug again.

 

 

Eugene Kinyapin Bronze 4 posts since
Dec 22, 2007
Currently Being Moderated
Feb 6, 2008 4:23 AM in response to: Eugene Kinyapin
Re: Problem with cyrillic in messages

 

Hello everyone!

 

 

I've enabled additional debugging in read method of XMLLightWeightParser:

 

 

/*

   

  • Main reading method

    */

    public void read(ByteBuffer byteBuffer) throws Exception {

        invalidateBuffer();

        // Check that the buffer is not bigger than 1 Megabyte. For security reasons

        // we will abort parsing when 1 Mega of queued chars was found.

        if (buffer.length() > 1048576) {

            throw new Exception("Stopped parsing never ending stanza");

        }

 

        Log.debug("XMLLightWeightParser: " + byteBuffer.toString());

 

        CharBuffer charBuffer = encoder.decode(byteBuffer.buf());

 

        Log.debug("XMLLightWeightParser: " + charBuffer.toString());

 

        char[] buf = charBuffer.array();

        int readByte = charBuffer.remaining();

 

 

to see what happens.

 

 

So, Openfire reads messages by parts. Cyrillic letters are double-byte. If first byte at the end of one part and second byte at the beginning of next one, each byte decodes in CharBuffer charBuffer = encoder.decode(byteBuffer.buf()) separetaly and replaces with Unicode replacement symbol 0xFFFD. Users see two 0xFFFD (square or question mark) symbols instead of one cyrillic (and other double-byte i think) letter.

 

 

It's happens with mina 1.1.0 in my case too, but more rare. It's uncommon for secure connections (i could reproduce this behaviour once or twice in all my tests).

 

 

I can send my debug logs, if you need them. 

 

 

Gaston Dombiak Jiver 3,803 posts since
Sep 26, 2001
Currently Being Moderated
Feb 7, 2008 2:14 AM in response to: Eugene Kinyapin
Re: Problem with cyrillic in messages

Hey Kinyapin,

 

Excellent analysis. I can now totally see why this problem is happening. I'm now trying to figure out the correct way to fix it but the problem has been identified now. Just as you mentioned this problem happens when each byte of a double-byte char arrives in 2 different iterations but what we are storing in XMLLightWeightParser is a corrupted/incomplete conversion of the byte into a character.

 

Thanks a lot for this information and the analysis. I will try to find a fix for this tonight or tomorrow.

 

Regards,

 

  -- Gato

Gaston Dombiak Jiver 3,803 posts since
Sep 26, 2001
Currently Being Moderated
Feb 7, 2008 7:54 PM in response to: Gaston Dombiak
Re: Problem with cyrillic in messages

Hey Kinyapin,

 

I'm no longer able to reproduce it locally so I was not able to test the fix. If you or anyone else is willing to test it I can send you a modified version of Openfire 3.4.4 or 3.4.5 with the patch so you can try it out.

 

Thanks,

 

  -- Gato

Eugene Kinyapin Bronze 4 posts since
Dec 22, 2007
Currently Being Moderated
Feb 7, 2008 11:01 PM in response to: Gaston Dombiak
Re: Problem with cyrillic in messages

 

Hi Gato!

 

 

Yes, i can test it this weekend and send results on monday or tuesday.

 

 

Thank you for quick fix.

 

 

Gaston Dombiak Jiver 3,803 posts since
Sep 26, 2001
Currently Being Moderated
Feb 8, 2008 1:29 AM in response to: Eugene Kinyapin
Re: Problem with cyrillic in messages

Hey Kinyapin,

 

Drop me an email with the Openfire version that you are using. Tomorrow morning (PST) I will send you back a modified version that you can try.

 

Thanks,

 

  -- Gato

Guus der Kinderen KeyContributor 403 posts since
Sep 8, 2005
Currently Being Moderated
Feb 8, 2008 3:36 AM in response to: Gaston Dombiak
Re: Problem with cyrillic in messages

 

Hey Gato,

 

 

I'm also interested in that patch. We're still using 3.4.2 though. Could you send me the patch itself (instead of a completely modified server)? We already applied other (unrelated) changes that I'd rather not override. Thanks!

 

 

Ivan Bronze 30 posts since
Feb 7, 2007
Currently Being Moderated
Feb 9, 2008 2:52 PM in response to: Guus der Kinderen
Re: Problem with cyrillic in messages

Guus so you will not upgrate your openfire? It's not good...

if your changes are interesting for others send your patchs to developers of thr openfire

else make them as plugins

Guus der Kinderen KeyContributor 403 posts since
Sep 8, 2005
Currently Being Moderated
Feb 11, 2008 3:10 AM in response to: Ivan
Re: Problem with cyrillic in messages

 

Thanks Gato, I've applied the patch to our source. I'll try to deploy it to the one machine that gives us the best chances of reproducing the problem.

 

 

Vanaf, I've actually sent in patches for a lot of the non-plugin modifications that I have. I believe serveral of them are scheduled for the upcoming 3.5.0 release. We're currently running four domains with Openfire, three of them in a clustered setup. We have been integrating a lot of custom systems (through 20+ plugins, client connections and external component implementations. Each time we update Openfire, we need to do regression tests on all of them, to check for compatibility issues. This entire process (which has to be repeated in several test environments) just eats up to much of our time to do it with ever minor Openfire release - that's why I'm still running 3.4.2

 

 

wroot KeyContributor 3,467 posts since
Jan 24, 2005

More Like This

  • Retrieving data ...