After being quiet for a a decade I love to start sharing some experiences again!
My new posts will be focused on Kong API Gateway Enterprise Edition. As we started years ago with Oracle API Gateway we switched two years ago and migrated all our API's to Kong 2.3.
This year I've had many interesting journeys with Kong, I successfully
migrated the datastore from Cassandra to PostgreSQL, upgraded Kong to 3.4,
performed some POC's with Grafana and Dynatrace and updated and improved our
CI/CD pipelines.
This
post is about monitoring, how to monitor Kong API Gateway with Grafana Cloud,
Grafana is the dashboard to visualize all metrics (Mimir), logs (Loki) and traces (Tempo).
Metrics:
Prometheus: Kong offers the
Prometheus plugin which exposes metrics on the Kong /metrics endpoint, to be
scraped by an agent, like the Grafana agent.
- Advantages: out-of-the-box, Kong official dashboard available in Grafana.
- Disadvantages: agent needed, Prometheus plugin may create high
cardinality of metrics and may cause performance issues.
- Links:
https://docs.konghq.com/hub/kong-inc/prometheus/
https://docs.konghq.com/gateway/latest/production/monitoring/prometheus/
https://grafana.com/grafana/dashboards/7424-kong-official/
Statsd: Statsd with Kong 2.8
didn't work smooth, required a lot of field mappings, in 2.8 there existed both
a statsd and statsd advanced plugin, statsd should work better from 3.x
onwards, see https://konghq.com/blog/engineering/how-to-use-prometheus-to-monitor-kong-gateway
Logs:
Http-log: Kong offers with the
http-log the possibility to send the log of each request to Grafana.
- Advantages: all request meta-information is available in Grafana, from
latencies to upstream IP. Minimal performance impact as the HTTP Log plugin
uses internal queues to decouple the production of log entries from their
transmission to the upstream log server.
- Disadvantages: No out-of-the-box dashboards available
- Links:
https://docs.konghq.com/hub/kong-inc/http-log/
Note
that this plugin works nicer with a custom field by lua added: Streams with the
following value:
local cjson = require "cjson" local ts=string.format('%18.0f', os.time()*1000000000) local log_payload = kong.log.serialize() local json_payload = cjson.encode(log_payload) local service = log_payload['service'] local t = { {stream = {kong_http_log='log-payload', service=service['name']}, values={{ts, json_payload}}}} return t
When
using cjson the following should be added to kong.conf:
untrusted_lua_sandbox_requires=cjson
Now we can easier explore the logs in Grafana, see
- all requests as they come in, simply select log-payload: {kong_http_log="log-payload"} |= ``
- parsed as json: {kong_http_log="log-payload"} | json
- filter on service, as Kong sent the service in the custom field by lua Streams: {service="ASW_Europe_Standards_API"} | json
Some
example dashboards:
- Workspace request per minute: sum by(workspace_name) (count_over_time({kong_http_log="log-payload"} | json [1m]))
- Response status per minute: sum by(response_status) (count_over_time({kong_http_log="log-payload"} | json [1m]))
- Service per minute: sum by(service) (count_over_time({kong_http_log="log-payload"} | json [1m]))
- Service status per minute: sum by(service, response_status) (count_over_time({kong_http_log="log-payload"} | json [1m]))
File
logs: the Grafana agent can monitor the access and error logs, especially the
access logs provide useful information about total latency and upstream path, but the whole set of information is less than what the http-log provides. Also the format of the loglines need to be defined in order to get parsed...
Traces:
OpenTelemetry: or olly (o-11-y) is a
framework to send spans and traces from Kong to Grafana, or any other OTLP enabled application.
- Advantages: a lot of information available, won't add much if the other systems (client/upstream) don't use OpenTelemetry
- Disadvantages: not fully ready yet, request to Grafana support was
needed in order to get metrics from traces. Also Kong mentions high tracing
instrumentation sampling rate has impact of the instrumentation on Kong
Gateway’s proxy performance in production.
- Links:
https://docs.konghq.com/hub/kong-inc/opentelemetry/
https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/
Currently I'm exploring this feature to see what kind of extra insights this gives
us when e.g. our upstream microservices also enable OpenTelemetry.