diff options
4 files changed, 107 insertions, 0 deletions
diff --git a/posts/prometheus_in_early_2016.mdwn b/posts/prometheus_in_early_2016.mdwn new file mode 100644 index 0000000..36aa184 --- /dev/null +++ b/posts/prometheus_in_early_2016.mdwn @@ -0,0 +1,104 @@ +[[!meta title="Prometheus in early 2016"]] + +[[!taglink Prometheus]] ([[website|http://prometheus.io/]]) has been used on +and off by [[!taglink Thread]] since May 2015 (before I [[joined in +June|posts/first_week_at_thread]]). Its a [[!taglink free_software]] time +series database which is very useful for monitoring systems and services. + +There was a brief gap in use, but then I set it up again in October, driven by +the need to monitor machine resources, and monitor the length of the queues +(Thread use [[django lightweight +queue|https://github.com/thread/django-lightweight-queue]]). This proved very +useful, as suddenly when a problem arose, you could look at the queues and +machine stats which helped greatly in determining the correct course of action. + +When using Prometheus, the server scrapes metrics from services that expose +them (e.g. the prometheus-node-exporter). This is a common pattern, and I had +already thrown together a exporter for django lightweight queue (that just +simply got the data out of redis), so as new and interesting problems occured, +I began looking at how Prometheus could be used to provide better visiblity. + +## PGBouncer + +The first issue that I addressed was a fun issue with exausting the pgbouncer +connection pool. The first time this happened, it was only noticed as emails +were being sent really slowly, and eventually it was determined that this was +due to workers waiting for a database connection. [[!taglink PGBouncer]] does +expose metrics, but having them in Prometheus makes them accessible, so I wrote +a [[Prometheus exporter for +PGBouncer|http://git.cbaines.net/prometheus-pgbouncer-exporter/about/]]. + +The data from this is displayed on relevant dashboards in Grafana, and has +helped more than once to quickly solve issues. Recently, the +prometheus-pgbouncer-exporter was [[accepted in to +Debian|https://tracker.debian.org/news/753455]], which will hopefully make it +easy for others to install and use. + +## HAProxy logs + +With the success of the PGBouncer exporter, I recently started working on +another exporter, this time for [[!taglink HAProxy]]. Now, there is already a +HAProxy exporter, but the metrics which I wanted (per HTTP request path rates, +per HTTP request path response duration histograms, ...) are not something that +it offers (as it just exposes metrics on the status page). These are something +that you can get from the HAProxy logs, and there are [[existing libraries to +parse this|https://github.com/gforcada/haproxy_log_analysis]], which made it +easier to put together an exporter. + +It was using the data from the HAProxy log exporter that I began to get a +better grasp on the power of aggregating metrics. The HAProxy log exporter, +exports a metric haproxy_log_requests_total and this can have a number of +labels (status_code, frontend_name, backend_name, server_name, +http_request_path, http_request_method, client_ip, client_port). Say you enable +the status_code, server_name and http_request_method labels, then, if you want +to get a rate of requests per status code (e.g. to check the rate of HTTP 500 +responses), you just run: + + sum(rate(haproxy_log_requests_total[1m])) by (status_code) + +[[!img request_rate_by_status_code.png + caption="Per second request rates, split by status code (key shown at the bottom)" +]] + +Perhaps you want to compare the performance of two servers for the different +request paths, you would run: + + sum( + rate(haproxy_log_requests_total[1m]) + ) by ( + http_request_path, server_name + ) + +[[!img request_rate_by_request_path.png + caption="Each metric represents the request rate for a single request path (e.g. /foo) for a single server" +]] + +And everything you can do with a simple counter, you can also do with +histograms for response duration. So say you want to know how a particular +request path is being handled over a set of servers, you can run: + + histogram_quantile( + 0.95, + sum( + rate(haproxy_log_response_processing_milliseconds_bucket[20m]) + ) by ( + http_request_path, le, server_name + ) + ) + +[[!img response_processing_duration_by_server_and_request_path.png + caption="Each metric represents the 95 percentile request rate for a single request path, for a single server" +]] + +This last query is aggregating the culamative histograms exported for each set +of label values, allowing very flexible views on the response processing +duration. + +## Next steps + +At the moment, both exporters are running fine. The PGBouncer exporter is in +Debian, and I am planning to do the same with the HAProxy log exporter +(however, this will take a little longer as there are missing dependencies). + +The next things I am interested in exploring in Prometheus is its capability to +make metrics avaialble for automated alerting. diff --git a/posts/prometheus_in_early_2016/request_rate_by_request_path.png b/posts/prometheus_in_early_2016/request_rate_by_request_path.png new file mode 120000 index 0000000..199f196 --- /dev/null +++ b/posts/prometheus_in_early_2016/request_rate_by_request_path.png @@ -0,0 +1 @@ +../../.git/annex/objects/Qx/K8/SHA256E-s55153--f145907842a5a097b753eb8977673eff3fdda9bcbc0186560e1d892b9bc68546.png/SHA256E-s55153--f145907842a5a097b753eb8977673eff3fdda9bcbc0186560e1d892b9bc68546.png
\ No newline at end of file diff --git a/posts/prometheus_in_early_2016/request_rate_by_status_code.png b/posts/prometheus_in_early_2016/request_rate_by_status_code.png new file mode 120000 index 0000000..4aacc64 --- /dev/null +++ b/posts/prometheus_in_early_2016/request_rate_by_status_code.png @@ -0,0 +1 @@ +../../.git/annex/objects/4F/p0/SHA256E-s54790--d24ba401f9c89a676b86911e559a90c78ad69984a53ad4270f7e4f86d9cab88d.png/SHA256E-s54790--d24ba401f9c89a676b86911e559a90c78ad69984a53ad4270f7e4f86d9cab88d.png
\ No newline at end of file diff --git a/posts/prometheus_in_early_2016/response_processing_duration_by_server_and_request_path.png b/posts/prometheus_in_early_2016/response_processing_duration_by_server_and_request_path.png new file mode 120000 index 0000000..1f7dc46 --- /dev/null +++ b/posts/prometheus_in_early_2016/response_processing_duration_by_server_and_request_path.png @@ -0,0 +1 @@ +../../.git/annex/objects/58/70/SHA256E-s56196--20a9b1b55fffa6b5c7e5177341291519f23ab17aed71eb434b1d18604f974152.png/SHA256E-s56196--20a9b1b55fffa6b5c7e5177341291519f23ab17aed71eb434b1d18604f974152.png
\ No newline at end of file |
