Situation

Needed To Wire All Of Akamai’s Data Into An Existing Grafana On GKE. Split Into Two Dashboards: CDN Data Rode On A Plugin That Was Already There, And WAF/Bot Logs Went Through A Small Proxy I Wrote Myself, Because The Only Grafana Plugin That Speaks Akamai’s SIEM API Is A Paid Third-Party One. Didn’t Want To Pay For It. (Cheapskate.)

Both pieces hit the same kind of bug: something that looks fixed, deploys clean, and still doesn’t work — with nothing useful in the logs to explain why.

Result First:

Every one of these failed the same way: no crash, no error, just nothing happening, and the fix looking correct right up until it silently wasn’t.

Trap One: An Unsigned Plugin’s Frontend Was Still Calling A Route That Doesn’t Exist Anymore

Akamai’s own plugin isn’t in Grafana’s signed catalog, so it has to load with GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS set. That part’s expected. What wasn’t expected: the plugin loaded fine, registered fine, and every panel just said “No Data.”

The frontend (module.js) was still building requests against Grafana’s old datasource-ID resource route:

/api/datasources/{id}/resources/...

This Grafana version only serves the uid-based route:

/api/datasources/uid/{uid}/resources/...

Patched the obvious one, the datasource’s getBackendDataSourceUrl helper, and redeployed. Still “No Data.” Turns out that helper wasn’t the only caller. The panel that actually fetches chart data calls the same pattern directly, twice, inline, in its own query() method, completely separate from the helper function. Four more call sites total, found only by grepping the whole bundle for the route pattern instead of trusting that one fix would propagate.

The part that made this worse than a normal bug hunt: fixing it and reloading the page didn’t show the fix either, at first. Grafana caches a plugin’s frontend assets keyed off the unchanged version string in plugin.json. Same version number, same cached JS, no matter how many times the container gets rebuilt underneath it. Had to stamp a fresh version string on every build just to be sure the browser wasn’t lying to me about what was actually running.

Trap Two: A url One Level Too Deep, And A Log That Had Nothing To Say About It

The SIEM side is a small FastAPI proxy that signs requests with Akamai’s EdgeGrid auth and hands Grafana back plain JSON, read by the generic (and actually signed) Infinity datasource plugin. Wired the datasource’s provisioning YAML, deployed, opened the dashboard: empty panels, no error banner, nothing in Grafana’s logs.

The mistake was one level of YAML nesting:

# wrong — url nested inside jsonData
jsonData:
  url: http://akamai-siem-proxy.monitoring.svc/siem

# right — url belongs at the datasource's own top level
url: http://akamai-siem-proxy.monitoring.svc/siem
jsonData:
  ...

With url in the wrong place, the top-level field Grafana actually reads stayed empty, and Infinity built its request as https:///siem?... — a URL with no host at all. Grafana’s own logs had nothing about this because the request never left the pod. Only way to actually see the failure was to bypass the dashboard and call POST /api/ds/query directly, simulating the exact panel query by hand, which finally returned the real error: http: no Host in request URL.

Trap Three: The Fix That Didn’t Take

Moved url to the top level, committed, let the GitOps pipeline sync, restarted Grafana. Opened the dashboard again. Still empty.

Checked the datasource’s actual stored config via the API — url was still blank, exactly as before the fix. Compared the restart’s logs against a Grafana pod that had just been created fresh: the fresh one logs a logger=provisioning.datasources line while it applies the YAML. The restarted one didn’t log that line at all. It wasn’t re-reading the file. It wasn’t touching that datasource.

Grafana’s file-based provisioner only writes a datasource’s config into its database the first time it sees that datasource. After that, the database row is authoritative, and the provisioner leaves it alone on every subsequent restart — even if the underlying YAML changed. There’s no flag for “please re-apply this,” and the docs don’t call this out anywhere obvious. The only way to actually change an already-provisioned datasource is to hit its API directly:

curl -u "admin:$ADMIN_PW" -X PUT "http://localhost:3000/api/datasources/uid/<uid>" \
  -H "Content-Type: application/json" \
  -d '{"name":"...","type":"...","access":"proxy","uid":"...","url":"...","jsonData":{...}}'

That call updates the database row directly, and it survives the next restart. Editing the YAML and redeploying is now step one of a two-step process for this cluster, not the whole fix — and it’ll stay that way for any datasource that already exists.

Notes

• A url field that’s technically valid YAML in the wrong nesting level fails with no error, only silence. Grafana had no reason to complain — from its side, the field it expected was simply absent, and the datasource plugin just built a broken request quietly. Simulating the exact panel query via POST /api/ds/query surfaced the real error instantly; trusting the dashboard UI or the pod logs alone would not have.
• A provisioner that only applies on first-create is a trap disguised as a convenience feature. It’s designed to avoid clobbering changes made through the UI, which is reasonable — but it means “I fixed the config file” and “the running system reflects that fix” are two different claims for anything already provisioned, and nothing tells you they’ve diverged.
• An unsigned plugin’s frontend deserves the same scrutiny as its backend. The obvious fix (one helper function) looked complete. Grepping the whole bundle for every occurrence of the broken pattern found four more places doing the exact same thing, undocumented and un-abstracted.
• Plugin asset caching keyed off a version string you control is easy to forget you control. If a rebuilt container doesn’t seem to have picked up a frontend change, check whether the cache key (plugin.json’s version) actually changed before assuming the build failed.
• Once both pieces were actually working, a follow-on problem showed up that’s really the same theme again: a 12-hour SIEM query window can return 10,000+ events, and neither an unbounded table panel nor a naive “just get the count” query survive that volume without a real answer for how much data crosses the wire. Capping the table panel and adding a count-only mode that never materializes the full event list fixed it, and made the eventual fix look, in hindsight, exactly as inevitable as the query volume problem was invisible beforehand.

How To Prevent It

Reference:

• Grafana — Provisioning Data Sources
• Grafana — Plugin Signature Verification
• Real Incident: Self-hosted Grafana on GKE, Akamai CDN + SIEM integration, 2026-08