...
...
...
...
...
...
...
...
...
...
...
...
...
...
Requirements:
| Jira Legacy | ||
|---|---|---|
|
...
|
| Jira Legacy | ||
|---|---|---|
|
...
|
| Jira Legacy | ||
|---|---|---|
|
...
|
Memory issues detection approach
...
In scope of this investigation cyclic CRUD operations in 25 5 threads during 4 hours was used as base load. Monitoring of service operating was carried out with using the VisualVM profiler.
Deep analysis of the memory utilization was performed with using Eclipse Memory Analyzer on the basis of memory dumps prepared in advance during the operation of the service (after 1 hour of service operating).
...
Issues Found
JMX reporting based on Dropwizard metrics
Description
TBD
Detailed investigation
The heap memory size is constantly increasing under load, reaches the maximum value for the container and a service failure occurs. The service does not recover after a load disconnect.
Detailed investigation
Emulation of the load on local environment showed the following results:
mod-invoice-storage:
mod-invoice:
It should be noted that the memory growth trend is observed for both of mod-invoice and mod-invoice-storage modules. However, for the mod-invoice-storage the process is more intensive. Therefore, invoices flow failure occurs at the mod-invoice-storage layer.
Heap dumps: mod-invoices.hprof, mod-invoices-storage.hprof.
Summary
TBDHeap dumps analysis showed that problem relates to Dropwizard metrics collecting.
After manual disabling of metrics collecting load test passes successfully with the following CPU/Heap utilization during 4 hours:
mod-invoice:
mod-invoice-storage:
Heap dumps analysis showed that issue was fixed.
Summary
Disable Dropwizard metrics on production environment. For this purpose, it makes sense to refactor the existing code to exclude the hardcoded enabling of the metrics RestLauncher.java#L30. Moreover, Vert.X supports the enabling of metrics collecting as a key of the Java command line, so the possibility of its enabling can be saved if command line activation is used.