Collecting metrics on a Scala Play service client for monitoring. This library covers the metrics collection via Prometheus client types such as Counter, Summary. A metrics server can scrape the collected metrics via a specified HTTP endpoint.
- http_requests_total
- http_requests_percentile_seconds
From io.prometheus simpleclient_hotspot library
- jvm_* metrics
- process_* metrics
These 2 metrics can be configured to have more detailed and focused request count and timing percentiles via the routes-to-track configuration. See example below.
- api_requests_total
- api_duration_seconds_summary
"com.salesforce.mce" %% "kineticpulse-metric" % "0.0.1" Specify a configuration at application.conf to override settings in the reference.conf of the kineticpulse library.
Example
com.salesforce.mce.kineticpulse.metrics { enabled = true # some http request paths are not interesting to capture metrics such as http duration # Disable metric collection for these request paths bypass.paths = ["/__status", "/", "/__metrics"] # routes segments to track in api_duration_seconds_summary, matched values go to the metric label "path" # example: ["api/v1", "model/predict", "model/accuracy"] routes-to-track = ["spotme/latest"] # delimiter is used within metric label "path" to replace the "/" in the request.path # See example where spotme/latest is changed to spotme-latest delimiter = "-" # label value to be used when a path is not bypass, but did not match routes-to-track, unmatched-path = "" } Suppose routes-to-track = ["spotme", "spotme/latest"] is configured. Consider an example end-point GET http://my.example.com/api/v1/spotme/latest/app, both spotme and spotme-latestare matched for the resulting metrics having a row each of labels with the path key.
With another GET http://my.example.com/api/v1/spotme/green/app, which matches spotme though not spotme/latest, the following would be in the __metrics payload
api_requests_total{status="200",method="GET",path="spotme",arguments="",} 2.0 api_requests_total{status="200",method="GET",path="spotme-latest",arguments="",} 1.0 Similarly, for api_duration_seconds_summary.
The current implementation does not track label values for arguments to optimize the number of time series to be created since higher cardinality of the labels results in more time series. The library wants to control the memory and compute load for collecting the metrics. Please customize for different requirements as needed.
In Play app/Module.scala, add
bind(classOf[Metric]).to(classOf[PrometheusMetric]) If not already existing, add an app/Filters.scala class.
import com.salesforce.mce.kineticpulse.MetricFilter class Filters @Inject() ( metricFilter: MetricFilter ) extends DefaultHttpFilters(metricFilter) In play app/conf/routes, add endpoint such as __metrics to invoke com.salesforce.mce.kineticpulse.MetricController.collect
If needed, extend the Metric or PrometheusMetric class to override the existing behavior.
For other custom metrics, we suggest to introducing a new Scala object to define and register additional Prometheus client types under the default registry, and to add collect logic where needed. The MetricController will include such custom metrics from the default registry to serialize in Prometheus client format for metrics scrape.
# HELP http_requests_total Total HTTP Requests Count # TYPE http_requests_total counter http_requests_total{status="200",method="GET",} 3.0 # HELP http_requests_percentile_seconds Total HTTP Requests Duration Percentile in seconds # TYPE http_requests_percentile_seconds summary http_requests_percentile_seconds{quantile="0.5",} 1.4882145 http_requests_percentile_seconds{quantile="0.95",} 3.5854054 http_requests_percentile_seconds{quantile="0.99",} 3.5854054 http_requests_percentile_seconds_count 3.0 http_requests_percentile_seconds_sum 6.4634309 # HELP api_requests_total Total API HTTP Requests Count # TYPE api_requests_total counter api_requests_total{status="200",method="GET",path="spotme-latest",arguments="",} 3.0 # HELP api_duration_seconds_summary Profile API response time in seconds summary # TYPE api_duration_seconds_summary summary api_duration_seconds_summary{method="GET",path="spotme-latest",arguments="",quantile="0.5",} 1.4882051 api_duration_seconds_summary{method="GET",path="spotme-latest",arguments="",quantile="0.95",} 3.5853817 api_duration_seconds_summary{method="GET",path="spotme-latest",arguments="",quantile="0.99",} 3.5853817 api_duration_seconds_summary_count{method="GET",path="spotme-latest",arguments="",} 3.0 api_duration_seconds_summary_sum{method="GET",path="spotme-latest",arguments="",} 6.463391 # HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool. # TYPE jvm_buffer_pool_used_bytes gauge jvm_buffer_pool_used_bytes{pool="direct",} 1.409438E7 jvm_buffer_pool_used_bytes{pool="mapped",} 0.0 # HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool. # TYPE jvm_buffer_pool_capacity_bytes gauge jvm_buffer_pool_capacity_bytes{pool="direct",} 1.409438E7 jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0 # HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool. # TYPE jvm_buffer_pool_used_buffers gauge jvm_buffer_pool_used_buffers{pool="direct",} 16.0 jvm_buffer_pool_used_buffers{pool="mapped",} 0.0 # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds. # TYPE jvm_gc_collection_seconds summary jvm_gc_collection_seconds_count{gc="PS Scavenge",} 2393.0 jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 10.263 jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 6.0 jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.684 # HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue. # TYPE jvm_memory_objects_pending_finalization gauge jvm_memory_objects_pending_finalization 0.0 # HELP jvm_memory_bytes_used Used bytes of a given JVM memory area. # TYPE jvm_memory_bytes_used gauge jvm_memory_bytes_used{area="heap",} 6.77809128E8 jvm_memory_bytes_used{area="nonheap",} 3.79781192E8 # HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area. # TYPE jvm_memory_bytes_committed gauge jvm_memory_bytes_committed{area="heap",} 1.668284416E9 jvm_memory_bytes_committed{area="nonheap",} 4.30465024E8 # HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area. # TYPE jvm_memory_bytes_max gauge jvm_memory_bytes_max{area="heap",} 1.854930944E9 jvm_memory_bytes_max{area="nonheap",} -1.0 # HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area. # TYPE jvm_memory_bytes_init gauge jvm_memory_bytes_init{area="heap",} 5.36870912E8 jvm_memory_bytes_init{area="nonheap",} 2555904.0 # HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool. # TYPE jvm_memory_pool_bytes_used gauge jvm_memory_pool_bytes_used{pool="Code Cache",} 9.9252928E7 jvm_memory_pool_bytes_used{pool="Metaspace",} 2.48641888E8 jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 3.1886376E7 jvm_memory_pool_bytes_used{pool="PS Eden Space",} 5.28267192E8 jvm_memory_pool_bytes_used{pool="PS Survivor Space",} 65536.0 jvm_memory_pool_bytes_used{pool="PS Old Gen",} 1.49485176E8 # HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool. # TYPE jvm_memory_pool_bytes_committed gauge jvm_memory_pool_bytes_committed{pool="Code Cache",} 1.14098176E8 jvm_memory_pool_bytes_committed{pool="Metaspace",} 2.824192E8 jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 3.3947648E7 jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 6.94157312E8 jvm_memory_pool_bytes_committed{pool="PS Survivor Space",} 524288.0 jvm_memory_pool_bytes_committed{pool="PS Old Gen",} 9.73602816E8 # HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool. # TYPE jvm_memory_pool_bytes_max gauge jvm_memory_pool_bytes_max{pool="Code Cache",} 2.5165824E8 jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0 jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9 jvm_memory_pool_bytes_max{pool="PS Eden Space",} 6.94157312E8 jvm_memory_pool_bytes_max{pool="PS Survivor Space",} 524288.0 jvm_memory_pool_bytes_max{pool="PS Old Gen",} 1.391460352E9 # HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool. # TYPE jvm_memory_pool_bytes_init gauge jvm_memory_pool_bytes_init{pool="Code Cache",} 2555904.0 jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0 jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0 jvm_memory_pool_bytes_init{pool="PS Eden Space",} 1.34742016E8 jvm_memory_pool_bytes_init{pool="PS Survivor Space",} 2.2020096E7 jvm_memory_pool_bytes_init{pool="PS Old Gen",} 3.58088704E8 # HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool. # TYPE jvm_memory_pool_collection_used_bytes gauge jvm_memory_pool_collection_used_bytes{pool="PS Eden Space",} 0.0 jvm_memory_pool_collection_used_bytes{pool="PS Survivor Space",} 65536.0 jvm_memory_pool_collection_used_bytes{pool="PS Old Gen",} 1.4569344E8 # HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool. # TYPE jvm_memory_pool_collection_committed_bytes gauge jvm_memory_pool_collection_committed_bytes{pool="PS Eden Space",} 6.94157312E8 jvm_memory_pool_collection_committed_bytes{pool="PS Survivor Space",} 524288.0 jvm_memory_pool_collection_committed_bytes{pool="PS Old Gen",} 9.73602816E8 # HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool. # TYPE jvm_memory_pool_collection_max_bytes gauge jvm_memory_pool_collection_max_bytes{pool="PS Eden Space",} 6.94157312E8 jvm_memory_pool_collection_max_bytes{pool="PS Survivor Space",} 524288.0 jvm_memory_pool_collection_max_bytes{pool="PS Old Gen",} 1.391460352E9 # HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool. # TYPE jvm_memory_pool_collection_init_bytes gauge jvm_memory_pool_collection_init_bytes{pool="PS Eden Space",} 1.34742016E8 jvm_memory_pool_collection_init_bytes{pool="PS Survivor Space",} 2.2020096E7 jvm_memory_pool_collection_init_bytes{pool="PS Old Gen",} 3.58088704E8 # HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. # TYPE jvm_memory_pool_allocated_bytes_total counter jvm_memory_pool_allocated_bytes_total{pool="Code Cache",} 1.0100448E8 jvm_memory_pool_allocated_bytes_total{pool="PS Eden Space",} 2.95168309768E11 jvm_memory_pool_allocated_bytes_total{pool="PS Old Gen",} 2.83409672E8 jvm_memory_pool_allocated_bytes_total{pool="PS Survivor Space",} 1.3836392E7 jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 3.19352E7 jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 2.48784576E8 # HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM # TYPE jvm_classes_currently_loaded gauge jvm_classes_currently_loaded 34380.0 # HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution # TYPE jvm_classes_loaded_total counter jvm_classes_loaded_total 35311.0 # HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution # TYPE jvm_classes_unloaded_total counter jvm_classes_unloaded_total 931.0 # HELP jvm_threads_current Current thread count of a JVM # TYPE jvm_threads_current gauge jvm_threads_current 59.0 # HELP jvm_threads_daemon Daemon thread count of a JVM # TYPE jvm_threads_daemon gauge jvm_threads_daemon 32.0 # HELP jvm_threads_peak Peak thread count of a JVM # TYPE jvm_threads_peak gauge jvm_threads_peak 71.0 # HELP jvm_threads_started_total Started thread count of a JVM # TYPE jvm_threads_started_total counter jvm_threads_started_total 402001.0 # HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers # TYPE jvm_threads_deadlocked gauge jvm_threads_deadlocked 0.0 # HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors # TYPE jvm_threads_deadlocked_monitor gauge jvm_threads_deadlocked_monitor 0.0 # HELP jvm_threads_state Current count of threads by state # TYPE jvm_threads_state gauge jvm_threads_state{state="NEW",} 0.0 jvm_threads_state{state="TERMINATED",} 0.0 jvm_threads_state{state="RUNNABLE",} 10.0 jvm_threads_state{state="BLOCKED",} 0.0 jvm_threads_state{state="WAITING",} 34.0 jvm_threads_state{state="TIMED_WAITING",} 16.0 jvm_threads_state{state="UNKNOWN",} 0.0 # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds. # TYPE process_cpu_seconds_total counter process_cpu_seconds_total 348.06 # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1.659474199947E9 # HELP process_open_fds Number of open file descriptors. # TYPE process_open_fds gauge process_open_fds 350.0 # HELP process_max_fds Maximum number of open file descriptors. # TYPE process_max_fds gauge process_max_fds 1048576.0 # HELP process_virtual_memory_bytes Virtual memory size in bytes. # TYPE process_virtual_memory_bytes gauge process_virtual_memory_bytes 4.152029184E9 # HELP process_resident_memory_bytes Resident memory size in bytes. # TYPE process_resident_memory_bytes gauge process_resident_memory_bytes 1.599746048E9 # HELP jvm_info VM version info # TYPE jvm_info gauge jvm_info{runtime="OpenJDK Runtime Environment",vendor="Azul Systems, Inc.",version="1.8.0_302-b08",} 1.0 # HELP api_duration_seconds_summary_created Profile API response time in seconds summary # TYPE api_duration_seconds_summary_created gauge api_duration_seconds_summary_created{method="GET",path="spotme-latest",arguments="",} 1.659474412873E9 # HELP api_requests_created Total API HTTP Requests Count # TYPE api_requests_created gauge api_requests_created{status="500",method="GET",path="spotme-latest",arguments="",} 1.659474416467E9 # HELP http_request_percentile_seconds_created Total HTTP Requests Duration Percentile in seconds # TYPE http_request_percentile_seconds_created gauge http_request_percentile_seconds_created 1.65947441287E9 # HELP http_requests_created Total HTTP Requests Count # TYPE http_requests_created gauge http_requests_created{status="500",method="GET",} 1.659474416467E9 # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. # TYPE jvm_memory_pool_allocated_bytes_created gauge jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.659474399643E9 jvm_memory_pool_allocated_bytes_created{pool="PS Eden Space",} 1.659474399635E9 jvm_memory_pool_allocated_bytes_created{pool="PS Old Gen",} 1.659474399643E9 jvm_memory_pool_allocated_bytes_created{pool="PS Survivor Space",} 1.659474399643E9 jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.659474399643E9 jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.659474399643E9