As you may know we have been running load tests for the last 2 weeks. The numbers that we are getting are really awesome so I wanted to share with you some practices and optimizations we done in order to improve scalability of our next Wildfire version.
Out of the box DB connection pool has a min size of 5. This was the first bottleneck we found. The min and max size was set to 100.
Cache size is also a key factor to get good performance. A small cache size or low effectiveness implies more db queries. The main cache that we needed to increase was the Roster cache. To hold the roster of 25K users that contain 20 contacts each we needed a 48MB cache.
Offline timestamp is now stored in its own table (and uses a new Cache). Before it was being stored as a user property and it was a big bottleneck when users were coming online or offline.
Move as many db operations (INSERT, UPDATE) to run as batch processes. This will improve overall performance and reduce number of locks.
Use MINA to handle Connection Managers from Wildfire. MINA is a SEDA solution so overall performance will be greatly improved. MINA uses thread pools, queues and a Thread for sending packets.
MINA also supports a throttling filter that could be used to reduce incoming traffic when Wildfire has reached 100% of its capacity.
Optimized Cache#calculateSize to consider Strings and avoid serializing Strings to estimate their size.
Optimization: do not load User when not needed. In particular, do not validate if a user exists after it was successfully authenticated.
Optimization: avoid creating of DOM objects in CMs. Just pass the received Strings to Wildfire. This was a huge optimization.
Optimization: reduce number of parser instances in CMs. One per working thread instead of one per connection. This was a huge optimization.
Avoid reflection if possible. Reflection in Java is not that optimized. In particular, we replaced usage of reflection while executing a DB command with an IF statement.
* Out of the box DB connection pool has a min size of 5. This was the first bottleneck we found. The min and max size was set to 100. reminds me once again of the connection pool I did send you which is much faster than the one you are currently using.
Setting min and max to 100 is a bad idea as this may cause a high memory usage on the database server depending on the used database engine and settings.
* Offline timestamp is now stored in its own table (and uses a new Cache). Before it was being stored as a user property and it was a big bottleneck when users were coming online or offline. - This is just a small step to improve the database design.
You would assume that increasing the number of connections will increase memory utilisation on the database server.
Would it therefore be sensible to establish a ‘‘rule of thumb’’ for setting the value?
i.e. increase the concurrent connection count by 1 for every x number of users?
Many applications have a configuration value within the application to indicate the expected maximum number of concurrent users. Perhaps that could be added to wildfire (and other settings such as Java memory, max number of db connections etc could be automaticallky set based on that)?
the max. needed number of connections depends on a lot of factors like
authentication using Wildfires database vs. LDAP,
Enterprise Plugin and
MUC logging into the database,
the database caches Wildfire uses, …
One should set the min. number of connections to 1 and the max. number to a value which can be handled by the database without problems, usually a value below 500 and monitor how many connections are used, add 10% and set this to the max. number.
When using Connection Managers the max number of connections to the DB is determined by the number of processing threads. Wildfire 3.2 will be using NIO for client-to-server connections so the same logic should be applied. The bottom line is that processing threads will be the ones using the db connections so max number of db connections is set by the number of processing threads.