Monday, after a long period of heavy development, I finally put out version 1.1.0 of the IM Gateway plugin! A total of 85 issues from JIRA were closed along the way and I'm quite pleased with the results. Along the way there were a number of stumbling blocks where I would just about be ready to release and something major would come up, and I certainly did not want to release anything with serious issues going on. As development continued, more and more features became interesting to me and were implemented. Since 1.0's release, I've had a number of helpful folk step up and offer patches, testing, code, translations, and help with libraries I depend on. I want to take a moment to thank everyone who contributed in any way! You are all invaluable to me!  There are a number of big plans coming for the next major release, but I wanted to highlight some of the things from 1.1.0's release and even comment on some of it!

First off, there's XMPP/Google Talk support. One might ask, why do you want XMPP support when there's s2s? That's a question that's been fought many times in the past and typically results in nothing being decided. Well I decided to implement it and then another helpful person (thanks Mehmet!) took my piddly start with it and turned it into a full on transport for the plugin. After implementing it I found myself using it with some accounts I had seen no reason to add to my Adium X config but decided hey, if I can handle it server side, then I'll just carry it around with me. Has been working out really well!

Then there's Gadu-Gadu support. This is a protocol I did not expect to ever implement. Why? Everything about it is Polish. I couldn't find my way around the web site enough to even download a copy of the client.  However, I said early on that if someone would translate or help me download or generally help me get it set up and there was a good API out there, I'd do it. So thanks to Marcin for stepping up and helping me get this started! The API itself was amazingly enough the easiest API to work with yet!

And what about SIMPLE support? Thanks to Ravin and Patrick for writing this support as I wouldn't have even known where to begin! I still don't understnad SIP/SIMPLE. Who knows if I ever will. But the transport sure works with my OpenSER server! I'm hoping some folk will take a look at it so I can get a feel for what it does and does not work with and maybe I can work with them to tweak it to work correctly with various implementations.

Another interesting thing that was added is an XMLRPC interface so site administrators can write their own web front ends for users to register with the various transports. That way folk could use some various standard web look and feel for the registration, and/or their own authentication mechanism, or even just something they consider "nicer" for their users, and dodge around typical requirements of registering through a client or requiring the admin to do it for them.

A lot of the inner workings of the plugin were redone, making things a lot more efficient in terms of network traffic and overall coding structure. I don't know that anyone but me will appreciate the reworkings of the code, but hey. =)

One of the interesting things that came about is that I ended up as the lead developer for Martyr, a very cool IRC library. IRClib just wasn't getting it done for me so I tried out Martyr, adored it's structure, and offered to help improve it. It's already led to a far better IRC transport than what I had before.

Beyond that I want to thank the folk from nimbuzz for creating the wonderful project OpenYMSG (a fork of YMSG) that fixes a number of problems I kept running into in the past! Through their improvements Yahoo support in the IM Gateway plugin was promoted to being considered stable!  On top of that they've helped me out some with JML, the library that handles MSN support. Great work folk!

Well that's enough of me, I'm excited about things to come though and I hope you all will join me in that excitement and continue to help me with testing, ideas, and whatever! =)

The engineering team at Jive Software is growing fast (btw, we're hiring!). I thought it might be interesting to talk about some of the engineering process changes we're making to cope with that growth, especially since they directly affect product releases. First, a bit of history. We've always had a fairly agile development process -- lots of iterations, tools like unit tests and continuous integration, etc. But, we've consistently had a lot of pain around our current process:

* Not enough time for QA* . It's always scheduled, but gets squeezed due to lack of time. For example, the "official" QA time period for the Spark 2.5.0 release got crunched down to almost nothing. * Stress when we don't need it* . The stress is caused by having to cram a ton of work into a short period of time to make internally set product release dates. It's also caused by scheduling extra features into releases at the last minute and by having to do emergency patch releases due to bugs we missed. We like working really hard, but there should be a way to do that without excessive stress. * Unpredictable release dates* . The bigger a release is, the harder it is to make accurate work effort estimates. That means that dates slip and that it's hard to communicate to the outside world exactly when a release will ship.

So, there are some clear problems that we want to fix. The trick for us was to come up with a process that fixes those problems but that's still light-weight enough to avoid a huge administrative burden. We're now experimenting with a strict release train model for Spark and Smack. Assuming it goes well, the same model will be applied across all our products, including Clearspace. A visual summary of the model is below (click the image for a larger version):

[Clearspace|http://www.igniterealtime.org/blog/wp-content/uploads/2007/04/train.png|train_sm all.png]

The key points to this model:

• We put out a new release every three weeks (although each release will have gone through a nine week process total).

• Three weeks before each development cycle is reserved for product management; three weeks after each development cycle is reserved for QA. But notice that all three processes are all happening at the same time.

There are all sorts of subtleties to the system like how to deal with new features that take more than three weeks of engineering time, but so far everything is going quite well.

The start of the three week cycle is always on a Monday, and we have a big meeting to determine exactly which features will go into the release. Product management has already done the work of prioritizing the features so we just need to choose exactly what we're going to work on and come up with the plan about how to get it done over the next three weeks. On the second and third Mondays during each cycle we go through more detailed updates than in our SCRUM meetings to see what's on track and what's not.

At the end of the cycle, we:

• Do the product release coming out of QA on Thursday.

• On Friday, finish all development and create a branch ("spark_2_5_2_branch" for example).

• Switch the builds in our continuous integration environment, TeamCity. The new branch becomes the "stable" release of the product and trunk becomes "development".

• Release a beta version out of the new branch. In other words, the world generally always sees a stable official product release and a beta release that will become official in three weeks.

There's way more that I could talk about with this process (including some of the potential drawbacks), but I'll save that for future blog entries. Things we're jazzed about so far include solid release dates, having a better process in place to deal with new feature requests and bug reports, more stable quality in each release, and always having a place to check in new feature development (trunk). We're still early on in the new system, but we'll all know how exactly how well this works in the next two months. Of course, we'd love your feedback as time goes on about how well you think we're doing.

We are getting closer and closer to the final release of Spark 2.5, and I thought I would give a little insight on some of the stuff that's new, stuff that's coming, and stuff that's hidden. It doesn't do anyone any good when we have features within the client that no one knows about.  So why don't we begin...

The New Stuff

1. Jingle VOIP Calls: This is the one big feature that we have been working on for quite some time, and it's really starting to kick some serious booty. Besides being one of the only, if not the only client using Jingle right now, we are coming up with all sorts of ideas. From conferences to just better interaction with others, it's just a blast to finally get to talk with some of the people who we have been chatting with daily.  Give it a try, you can make a call from either the roster or inside any of the chat rooms. Basically anyone using Spark 2.5.0 Beta 4 should be able to talk with one another.

2. Recent Conversations: Try ctrl+e inside of the Contact List, this allows you to see the last 10 users you have chatted with. I use it all the time.

3. My Favorites: Try ctrl+t inside of the Contact List, you can see the most "Popular" people related to you. Now granted, it may be a person that you just have to put up with constantly, but I like to be optimistic about it.

4. Copy To and Move To: I actually added this because I'm still having troubles scrolling down the contact list using drag and drop mechansims. This feature allows you to do mass move or copies of contacts into other contact groups.

The Hidden Stuff

1. Task List: Using ctrl+F5, you can add tasks that are persisted directly onto your server, so they are mobile. This feature, as is the next one, have been in the product forever, but yet, no one seems to know about them.

2. Notes: Using  ctrl+F6, allows a simple notepad that is persisted directory onto your server. I use it for daily reminders, etc.

Still Brewing (But should get in the release)

1. I have a shiny new Mac on my desk in order to get the OSX build up to snuff.  So, two things that I'm going to be jamming on. The first one being better growl notification. The second one being better dock notifications via text and animated icons.

2. Jingle  Discovery - we need to see who can actually talk.

I hope you find this helpful. Always fun to discover new things

Cheers,

Derek

Smack uses an XML Pull Parser (XPP) to parse XMPP stanzas and build custom Packet objects. Wildfire represents XMPP stanzas as DOM objects wrapped by Packet objects. Each approach has its own pros and cons. Moreover, in Wildfire we can use different parsers to generate DOM objects.

For Wildfire 3.2 we needed to change the way we were parsing XML to work with the new, more scalable, networking layer built using MINA. We wanted to keep building DOM objects wrapped by Packet objects so our parsers were still useful. However, we needed a way to parse stanzas received in an asynchronous way. We ended up reusing a contribution made by a community member. We know that it is a custom solution that we might need to replace at some point since the parsing is not 100% efficient and some cases may potentially break the parser as was seen in Wildfire 3.2.1.

Anyway, while doing some heavy load testing on Wildfire and measuring performance we noticed that the parser we were using to generate DOM objects was becoming a serious bottleneck. At that point we were using a SAX parser (SAXReader) so we decided to try with other parsers and compare results. The other parser that we had in Wildfire was XMPPPacketReader that uses an XML Pull Parser (XPP).  To my surprise the performance improvement was substantial with the new parser: around 30% faster with the XPP parser compared to the SAX parser.

Architecturally it is obvious that an XPP parser will be much faster than a SAX parser, but since both parsers were being used to create DOM objects I initially thought  it wouldn't make much of a difference which one was used since building DOM objects was the expensive operation. Well, I was simply flat wrong and I learned it the hard way. I'm happy, though, that we found this bottleneck during our performance tests prior to release. It is yet another proof of how important it is to include QA as part of your development cycle.

How does building software for an open source community compare to the commercial world? It has been three years since I started working on an open source project. Before this exciting work I spent ten years building commercial software and I can tell you that the differences are really big both on the final product and the way the product is being built.

Here at Jive we are very focused on solving business problems rather than building cool technical solutions just because they help increase our ego. A consequence of this philosophy is that the community feedback is key to building a successful solution. This is one of the reasons I would like to thank you all for helping us build Wildfire into the product that it is now: a successful product that is growing and becoming more popular with each new revision.

Now back to the initial question. Is it really that different to build open source software compared to commercial software? As I said the answer is yes and let me give you a concrete example:

There was a time when Wildfire didn't have support for SASL and I have to say that I had no idea about SASL. Fortunately, there was a community member that provided the first implementation and that was a great opportunity to review his work to ensure quality standards and at the same time learn about the subject. Moreover, a few month later we heard that other community members were adding Single-Sign-On and also GSSAPI as SASLextensions. I said: Wow, this is great. The community keeps providing some cool new features that are much needed and I'm not an expert on this area.

It was clear to me that having a big community was much more powerful than just having several developers and a few product users. Another way to understand the difference is to split the question in two areas: 1) product quality, and, 2) product development and support.

Product Quality

Wildfire has an incredibly large community behind it and we can safely assert that, due to statistics reasons, the more people use a product the more chances a product has for maturing and becoming a solid solution. The number of downloads we are registering is immense and the number of problems we are receiving is proportionally inverse. Of course, this was not the way things used to be. I think that the key to becoming a mature product, in our case, was being strict in running automated or manual test cases before delivery code to the community. In addition, the number of setups that are out there helps ensure that someone has to have already used the software the same way someone else is doing it. In summary, the bigger the community the more chances a product has for becoming a solid product. Let's leave the analysis of the reasons why our community grew out the way it has grown for another post.

Product Development

Supporting an open source community is radically different from supporting commercial companies. The first thing that an open source product has to build, besides the product itself, is a community. For that it is key to have a good site that allows community members to meet up, collaborate, share experiences, problems, etc.. The second challenge is to organize your time so that you can keep thinking and implementing new product features or bug reports and answer questions posted in the forums. While building commercial solutions there is naturally a bigger distance between developers and end users and usually customer support handles customer requests. In open source products, the developer is much closer to the end users so it is always a challenge to focus on work and the community at the same time. I can tell you that this is a very hard challenge but it pays off at the end.

Wildfire 3.2.2 is out now and  I guess that this long post is just a way to thank you all for building Wildfire the product that we are all proud of.

I'm excited to announce that my first article on Wildfire plugin development has been published. What originally started off as a blog post grew into something much larger (and longer), so at Matt's urging the content was reworked and a space for it on the Ignite Realtime site was carved out.

The article walks the reader throught the development of a "Message of the Day" plugin that will send users a greeting each time they sign-in to Wildfire. Over the course of the article a number of plugin specific API's are examined along with other, core Wildfire API's. The completed plugin is fully functional but leaves room for a number of additional features.

As always, any and all feedback is welcome.

See you on the forums,

Ryan

A new beta of Smack (3.0.0 Beta 1) was released today. It's been a long time since the last release, which is easy to tell by the length of the changelog. The beta represents major new functionality. Some highlights:

• Much better control over connections, including the ability to re-connect after a dropped connection.

• Many new XEP's supported and better XMPP RFC compliance.

• Major re-factoring of how chats work.

All this is very good news. The bad news is that this new release isn't a drop-in replacement for the old version of Smack. We had to make API changes in order to support the features above and we decided to make the move to Java 5 at the same time (might as well break a lot if you're going to break a little). The switch to Java 5 in particular will be a deal-breaker for some. We'll likely continue making bug fixes in the 2.x branch as compensation.

All these changes have been driven by work being done on Spark. In fact, Jingle support should officially land in Smack before the 3.0 beta period is up (you may have seen Jingle in the roadmap already). Let us know what you think of the beta. We'd also love your feedback on what we can do to make the transition smoother.

  !http://www.igniterealtime.org/blog/wp-content/uploads/2007/01/2007_finalist_bw.g if! Wildfire has been selected as a finalist for this year's CODiE awards in the Best Communications Solution category. We'll find out on April 17 just how much the judges like the most cutting-edge Open Source communications server on the market. Going through the judging process for awards can be time consuming, but they're a great validation for the project. Of course, don't forget about the  Open Source Reader's Choice voting -- we're just nine votes away from first place!