This is an excerpt from the Openfire Installation Guide found in your documentation folder of the Openfire server install folder:

Advanced users may wish to pass in parameters to the Java virtual machine (VM) to customize the runtime environment of Openfire. You can do this by creating vmoptions files in the bin/ directory of your Openfire installation. For the Windows service, you'd create a new text file called openfire-service.vmoptions. Each parameter to the VM should be on a new line of the file. For example, to set the minimum heap size to 512 MB and max VM heap size to 1024 MB, you'd use:

-Xms512m
-Xmx1024m

To create parameters for the normal launcher, create a file called openfired.vmoptions (since the openfire.exe launcher invokes the openfired.exe executable to actually start the server).

as a work around, I ususally uncomment the line in bin/openfire:

INSTALL4J_ADD_VM_PARAMS="-Xmx20g"

note: 20g is what I have.

I have not figure out why *.vmoptions file is sometimes not read.

Add a line like this in /etc/sysconfig/openfire

OPENFIRE_OPTS="-Xms256m -Xmx512m

******************************************************************

This way worked to me in a Centos 5. tnx

You dont have to add this to your System Properties. You should read that doc carefully. There is a link to Openfire Settings. Here, a quote:

Custom Parameters

Advanced users may wish to pass in parameters to the Java virtual machine (VM) to customize     the runtime environment of Openfire. You can do this by creating vmoptions files in the bin/     directory of your Openfire installation. For the Windows service, you'd create a new text     file called openfire-service.vmoptions. Each parameter to the VM should be on a new line of     the file. For example, to set the minimum heap size to 512 MB and max VM heap size to 1024 MB, you'd use:

-Xms512m
-Xmx1024m

I'm not sure about the deb installation. Maybe you should try to ask LG (the author of these docs) directly.

One more doc http://www.igniterealtime.org/community/docs/DOC-1033

At the bottom there are instructions about openfired.vmoptions file.