95 lines
4.3 KiB
Org Mode
95 lines
4.3 KiB
Org Mode
#+TITLE: Integrating Equinix Metal API with Equinix Watch
|
|
#+AUTHOR: Adam Mohammed
|
|
|
|
* Problem
|
|
|
|
Equinix Watch has defined the format for which they want to ingest
|
|
auditable events. They chose the OTLP as the protocol for
|
|
ingesting these events from services restricting their ingestion to
|
|
just the logging signal.
|
|
|
|
Normally when sending data to a collector, you would make use of the
|
|
OpenTelemetry libraries to make it easy to grab metadata about the
|
|
request and surrounding environment, without needing to manually
|
|
cobble that data together. Unfortunately, using OTEL logging as the
|
|
only signal that Equinix Watch accepts, makes adoption needlessly
|
|
painful. Ruby does not have a stable client library for OTEL logs, and
|
|
neither does Golang.
|
|
|
|
Most of the spec provided by EquinixWatch does not actually relate to
|
|
the log that we would like to provide to the customer. OTEL Logging
|
|
aims to make this simple by using the Baggage and Context APIs to
|
|
enrich the log records with information about the surrounding
|
|
environment and context. Again, the implementations for these are
|
|
incomplete and not production ready.
|
|
|
|
Until the OTEL libraries provide support for the context and baggage
|
|
propogation in the Logs API/SDK, this will data will need to be
|
|
extracted and formatted specifically for Equinix Watch, meaning the
|
|
burden of integration is higher than it needs to be. If we end up
|
|
doing this, we'll probably just fetch the same data from the span
|
|
attributes anyway, to keep things consistent.
|
|
|
|
There's absolutely no reason to do this work when we can add the logs
|
|
in a structured way to the trace and pass that through to their custom
|
|
collector. By doing this we don't need to wait for the OTEL libraries
|
|
to provide logging implementations that do what traces already
|
|
provide.
|
|
|
|
The only reason I can see not to do this is that it makes Equinix
|
|
Watch have to handle translating trace information to a format that
|
|
can be delivered to their end targets. I'd argue that's going to need
|
|
to happen anyway, so why not make use of all the wonderful tools we
|
|
have to enrich the data you have as input, so you can build complete
|
|
and interesting audit logs for you end user.
|
|
|
|
* Concerns
|
|
|
|
- Alex: Yeahhhh I've gotta say I'm uncomfortable making our existing
|
|
OTEL collector, which is right now part of our internal tooling, and
|
|
making it part of the critical path for customer data with Equinix
|
|
Watch.
|
|
|
|
I don't understand this, of course you're going to be in your
|
|
critical path. I'm not saying to use your collector as the ONLY
|
|
collector, this is why we even have collectors. We are able to
|
|
configure where the data are exported.
|
|
|
|
- Alex: IMO internal traces are implementation details that are
|
|
subject to change and there are too many things that could go
|
|
wrong. What happens if the format of those traces changes due to
|
|
some library upgrade, or if there's memory pressure and we start
|
|
sampling events or something?
|
|
|
|
Traces being implementation details - like audit logs? There's a
|
|
reason we use standard libraries to instrument our traces. These
|
|
libraries follow OTEL Semantic Conventions so we have stable and
|
|
consistent span attributes that track data across services.
|
|
|
|
Memory pressure, this isn't solved by OLTP at all, in fact
|
|
collectors will refuse spans if they're experiencing memory pressure
|
|
to prevent getting OOMKilled. This is not an application concern,
|
|
this is an monitoring concern. You should know if your collector is
|
|
|
|
- Alex: In my experience, devs in general have a higher tolerance for
|
|
gaps and breakage in their internal tooling than what I'm willing to
|
|
have for customer-facing audit logs.
|
|
|
|
This is just poor form. If you don't trust the applications that
|
|
integrate with your application, what do you trust?
|
|
|
|
- Alex: I think customer-facing observability is net-new functionality
|
|
and, for the time being, I'm OK with putting a higher burden on
|
|
applications producing that data than "flip a flag in the collector
|
|
to redirect part of the firehose to Equinix Watch
|
|
|
|
Net-new - sure, I agree
|
|
|
|
Higher burden on applications producing the data - why though? we
|
|
can provide you a higher quality data source already instead of
|
|
hand-rolling an implementation to the the logs signal
|
|
|
|
"flip a flag in the collector" - I think this just shows illiteracy,
|
|
but we are able to control what parts are shipped to your fragile
|
|
collector.
|