I read something interesting recently when looking at BASE databases implemented on the JVM. The linux kernel will quite aggressively swap out pages from processes when they are not being used. This can conflict with the way that generational garbage collection works in the JVM: garbage collection in particular can cause a lot of page faulting. Here the heap will be scanned for objects to be freed, however linux may have already paged out that memory making collection more expensive. If you have enough ram, the pages of your java application can be locked into memory by arranging for the
mlockall system call to be run via one of the java native interface technologies (JNI or JNA).
The number of major page faults are collected per process and are available from
ps (often via
/proc). By fixing the size of your application's heap and watching page faults through garbage collection it ought to be easy to assess how much this is happening.