Full GC is an important phase in the garbage collection process. During this full GC phase, garbage is collected from all the regions in the JVM heap (Young, Old, Perm, Metaspace). During this phase, JVM is paused. No customer transactions will be processed, and JVM will use all the CPU cycles to perform garbage collection. Due to that CPU consumption will be quite high. Thus in general full GCs aren’t desired. Needless to ask the desirability of consecutive full GCs. Consecutive full GCs will cause following problems:
- CPU consumption will spike up.
- Because JVM is paused, the response time of your application transactions will degrade. Thus it will impact your SLAs and cause poor customer experience.
In this article, let’s learn about this consecutive full GCs – what is it? What causes it? How to fix it?
What is Consecutive full GCs?
It’s always easier to explain through an example. So let’s examine the GC log of a real world application, which suffered from this consecutive full GC problem. Below are the graphs generated by GCeasy garbage collection log analysis tool. Notice the highlighted portion of the first graph. You can see the full GCs to be consecutively running (red triangles in the graph indicates full GC). In any application’s life cycle, young GC and full GC runs. In healthy applications, most of the time young GC runs and very few times full GC runs. If full GC runs consecutively, then it’s indicative of a problem.
Even though full GCs were consecutively running, it wasn’t able to reclaim enough memory. You can observe it in the second graph (which shows the reclaimed bytes). In this graph you can see the memory reclaimed from these full GCs to be very less. It is because most of the objects in memory are in active use, thus JVM isn’t able to reclaim enough memory.
What causes Consecutive full GCs?
Consecutive Full GCs are caused because of one single reason: Under allocation of JVM heap size. It’s indicative of the fact that application needs more memory than what you have allocated. In other words, you are trying to fit in a truck load of objects in a small compact car. So JVM has to do more work to clean-up the garbage objects in the small compact car, to make room for actively used objects.
Now you might have a question, my application was running fine all along, why all of a sudden I see this consecutive Full GC problem? That’s a valid question. Answer to this question could be one of the following:
- Your application’s traffic volume has started to grow since the last time you have tuned the JVM heap size. May be your business is improving, more users have started to use your application.
- During your peak volume time period, more objects would get created than normal time. May be you didn’t tune your JVM for peak traffic volume or your peak traffic volume has surged since the last time you have tuned the JVM heap size.
How to solve consecutive Full GCs?
Consecutive Full GCs can be solved through couple of approaches:
1. Increase JVM Heap Size
Since consecutive Full GCs runs due to lack of memory, increasing JVM heap size should solve the problem. Say suppose you have allocated JVM heap size to be 2.5GB, then try increasing it to 3 GB and see whether it resolves the problem. JVM heap size can be increased by passing the argument: “-Xmx”. Example:
This argument will set the JVM heap size to be 3 GB. If it still doesn’t resolve the problem then try increasing the JVM heap size step by step. Over-allocation of JVM heap is also not good either, it might increase the GC pause time as well.
2. Add more JVM instances
Adding more JVM instances is an another solution to this problem.When you add more JVM instance, then traffic volume will get distributed. The amount of traffic volume handled by one single JVM instance will go down. If less traffic volume is sent to a JVM instance, then less objects will be created. If less objects are created, then you will not run into the consecutive Full GC problems.
Validating the fix
Irrespective of the approach you take to resolve the problem, validate the fix in the test environment before rolling out the change to production. Because any changes to JVM heap settings should be thoroughly tested & validated. To validate that problem doesn’t resurface with new settings, study your GC log with GCeasy tool. It has the intelligence to spot and report whether the application is suffering from consecutive full GC problem or not.