feat: report usage metrics for the cli channel - #76
Conversation
## Changed - Fatal Lara API errors now unwind instead of exiting on the spot, so a command always closes cleanly before the process ends - Every command routes through runSafely, which reports failures uniformly; init gains the error handling it never had - init verifies the API key against Lara and warns if it cannot — it never blocks, and still writes the config offline ## New - Usage metrics for the cli channel: auth_success, auth_fail, call_success and call_error, batched to a queue file and sent once per command - Two-step ingestion auth: the channel key buys a short-lived token, which is the only credential the ingestion endpoint accepts - Events carry the Lara account id, the CLI version, what was translated and the language pair — never text, paths, config or credentials - Opt out with LARA_TELEMETRY_DISABLED=1; a build with un-baked keys is silently inert - LARA_SERVER_URL points the SDK at a non-production Lara deployment - verify-metrics-e2e.mjs exercises the built CLI end to end, degradation paths included
60d0153 to
1178a3d
Compare
There was a problem hiding this comment.
Pull request overview
Adds a usage-metrics pipeline for the CLI channel (queue-on-disk + bounded flush + two-step auth), and refactors command error handling to unwind cleanly so metrics can be finalized on exit. This fits into the CLI’s operational telemetry/observability layer while keeping translation behavior “best effort” (telemetry must not break or slow commands).
Changes:
- Introduces
src/modules/metrics/*to queue events locally and flush them via token exchange (/auth/issue-token→/metrics/ingest-events) with retries/cooldowns. - Centralizes command execution through
runSafely()to instrument metrics, classify failures, and replace directprocess.exit(1)call sites with unwindable errors. - Adds unit + E2E verification for metrics, plus a publish-time bake step to embed backend URL/key into the built package.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/laraHeaders.ts | Exposes a defensive accessor for the SDK internal client to support metrics identity resolution. |
| src/utils/error.ts | Adds HandledExitError + reporting helper so callers unwind instead of exiting immediately. |
| src/modules/translation/translation.service.ts | Routes Lara calls through auth-reporting wrapper and records metrics for translated chars/errors. |
| src/modules/metrics/metrics.ts | Implements queueing, token exchange, flush, error classification, and per-command lifecycle. |
| src/modules/metrics/metrics.const.ts | Defines metrics contract constants and baked placeholder values. |
| src/messages/messages.ts | Adds user-facing messages for partial failures and init credential verification warning. |
| src/cli/cmd/translate/translate.ts | Wraps translate command in runSafely, sets metrics context, and unwinds on failures. |
| src/cli/cmd/init/init.utils.ts | Updates process.env after writing .env so the current process can validate credentials. |
| src/cli/cmd/init/init.ts | Wraps init in runSafely and adds non-blocking credential validation/warn behavior. |
| src/cli/cmd/common/prompts.ts | Updates Lara error handling to unwind under runSafely instead of exiting underneath it. |
| src/cli/cmd/common/command.ts | Makes runSafely the metrics lifecycle owner and single error-to-exit conversion point. |
| src/tests/modules/metrics.test.ts | Adds comprehensive unit tests for metrics behavior (queueing, token refresh, cooldown, etc.). |
| scripts/verify-metrics-e2e.mjs | Adds E2E harness validating end-to-end contract with fake Lara + metrics backend. |
| scripts/bake-metrics-key.mjs | Adds publish-time baking of backend URL/key into the compiled output. |
| README.md | Documents usage-metrics behavior and opt-out. |
| package.json | Adds verify:metrics script. |
| eslint.config.js | Adds Node globals to ESLint config (needed by new Node APIs usage). |
| .github/workflows/publish.yml | Adds “Bake metrics key” step using repository secrets. |
Suppressed comments (2)
src/modules/metrics/metrics.ts:253
- There is a stray JSDoc line ("True when events can actually be attributed and sent.") immediately followed by another JSDoc block. As written, the first block attaches to nothing and is likely to trigger lint/doc tooling issues.
/** True when events can actually be attributed and sent. */
/**
* Never blocks, never touches the network, never throws. Every guard telemetry
* has lives here: this is the only door onto the queue, so nothing upstream has
* to re-check whether reporting is on.
README.md:237
- The README states events are sent "when the command finishes, and again at the start of the next one". The code never flushes at command start; queued events are retried on the next command's closing flush (finishAndFlush), so this should be reworded.
Events are appended to a local file and sent in one batch — when the command finishes, and again
at the start of the next one if anything was left over. If the metrics backend is slow or
unreachable the CLI behaves exactly as it does today: the send is bounded to two seconds and
whatever could not be delivered simply waits for the next run.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
## Changed - auth_fail is reported even when no Lara account id is known yet — a key rejected on first use is exactly the funnel step that was being dropped - accountId is omitted when unknown rather than sent empty; call_success and call_error still require it, since the backend rejects the whole batch - queued events older than the backend's 30-day bound are dropped at flush, so one stale line no longer takes every fresh event down with it - a 429 from the ingestion endpoint now honours Retry-After across processes, instead of retrying on the very next command ## New - e2e scenario for a first-ever key Lara refuses: auth_fail delivered with no accountId, no call event, nothing faked
## Changed - a Lara error that comes back past the auth check (400, 402, 429) now reports auth_success: the key was accepted, and the funnel was losing users who got in fine and then ran out of credit - the module docstring and the README no longer claim a flush at the start of a command; there has only ever been one, on the way out - the README says what actually triggers each event: a translation command always closes with call_success or call_error, even when it fails before reaching Lara, while init, memory and glossary only report the key - dropped a stray doc comment left attached to nothing above queueEvent ## New - e2e scenario for a valid key with no credit left: auth_success followed by call_error / payment_402
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/cli/cmd/init/init.ts:118
- The comment says “Never blocks”, but
checkCredentials()awaits a real network call viaTranslationService.validateCredentials()(which is bounded by the SDK request timeout, currently 30s). If Lara is unreachable,initcan still pause noticeably before proceeding. Either tighten the wording or enforce a shorter timeout for the credential check.
* Turns the key the user just entered into a real yes/no answer, and reports it.
* Never blocks: `init` writes the config either way, so the command still works
* offline or behind a proxy that is momentarily down.
## Changed - DO_NOT_TRACK now switches metrics off exactly like LARA_TELEMETRY_DISABLED: no events, no requests, not even an installation id on disk - an opt-out variable set to 0, false or an empty value is read as "leave telemetry alone" — before, any value at all switched it off, so =0 quietly did the opposite of what it says ## New - a one-line notice on the first run that records anything: what is reported, what never is, and how to turn it off. Once per machine, on stderr, so a piped stdout is untouched — the README is not somewhere consent can hide
## Changed - a file the user's own tooling broke now lands as `parse_error` instead of `unknown`: the per-file failure is recorded where it happens, so the funnel sees it rather than the generic "some files failed" the command unwinds with - the first-run notice and the README no longer enumerate the events and fields; the README keeps one line on what is recorded, including the Lara account id, so the "never sent" list stays true ## New - `metadata.fileTypes`: the formats a run touched, as one sorted value such as `json,po` — read from the config's own file types, never a name or a path
## Changed - init writes the config before verifying the key, so a slow or unreachable Lara can no longer hold up the file the command exists to write; with no credentials at all it stops warning twice for the same reason - what a Lara status code means is decided in the metrics module now, not in the translation service: the service reports the answer, metrics reads it - an error the retry loop recovers from is no longer recorded, so a transient failure the user never saw cannot end up as the run's errorType - every branch of the per-file catch in translate reports the real error, not just the one that happened to have the call - dropped a two-entry allowlist, a one-caller error helper and a cast the caller had to repeat; the SDK's internal client now has a type
There was a problem hiding this comment.
🟡 Changes recommended
There are a couple of concrete shutdown/telemetry reliability issues (spinner cleanup on error and malformed queued-event timestamp handling) that should be fixed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/modules/metrics/metrics.ts:460
- flushQueue() currently keeps events whose timestamp is missing (typeof !== 'string'). Those events violate the stated contract (every event has a timestamp) and can cause the backend to reject the whole batch, dropping otherwise valid events when the response becomes 400. Filter out events without a valid timestamp so one malformed line cannot poison a batch.
events = kept
.map((line) => JSON.parse(line) as { timestamp?: unknown })
.filter(
(event) =>
typeof event.timestamp !== 'string' || Date.parse(event.timestamp) >= cutoff
);
- Files reviewed: 18/18 changed files
- Comments generated: 1
- Review effort level: Lite
## Changed - the queue only sends events the backend can actually accept: a line with an unreadable, missing or far-future timestamp is dropped rather than taking the whole batch — and every fresh event in it — down with it - the progress bar stops before the command unwinds, so it no longer animates over the closing flush and leaves the terminal mid-redraw - an error that was already printed now unwinds carrying the words the user actually read: the reporting helper hands back the message it displayed, instead of each caller inventing a second wording
Reports usage metrics for the
clichannel, against the contract inlara-integrations-monitoring@developand the shared event-reporting standard.What is sent
auth_success/auth_failwhen a key is accepted or rejected, andcall_success/call_errorwhen a translation finishes. One terminal eventper
translateinvocation;memory,glossaryandinitreport auth only —a
lara memory listcounted as a successful translation would move the funnelfor work that never happened.
Every event carries the raw Lara account id (
acc_...), the CLI version, aper-run session id, a per-event id, latency, characters translated, and a
metadatablock withfeature(text/document),surfaceand thelanguage pair when a single one is known.
Never sent: source or translated text, file names or paths, configuration,
credentials.
How it authenticates
Two steps, per the standard. The channel key buys a short-lived token from
POST /auth/issue-tokenwith a persisted installation UUID; only that token isaccepted by
POST /metrics/ingest-events. The key never reaches the ingestionendpoint and the token never reaches the disk. On 401 the client asks for one
new token and retries once.
The account id is read from the JWT the Lara SDK already holds after it
authenticates — no extra round trip, no re-implementation of the signed
/v2/authcall. It is cached per access key id so a revoked or expired key canstill report
auth_fail.How it stays out of the way
Events are appended synchronously to a queue file and sent in one batch when
the command finishes. Nothing is flushed at start-up: that would put a token
exchange in front of the user's command before it had even validated its
arguments, to send events the closing flush picks up anyway.
The whole exchange is bounded to 2s. A slow or unreachable backend, a
read-only disk, a missing key, an un-baked build — all degrade to "no
telemetry", never to a failed translation.
LARA_TELEMETRY_DISABLED=1turns itoff entirely, and with it set the state directory is not even created.
The token endpoint rate limits per installation and answers 429 with a
Retry-After; the client records that deadline so a burst of commands does noteach pay a doomed round trip.
Changes to existing behaviour
process.exit(1)on thespot, so a command always closes cleanly.
HandledExitErrorcarries theoriginal error as its
cause, which is what keeps the failure classified.runSafely, which is now the single place thatopens and closes the metrics window and reports failures.
initgains theerror handling it never had — a throw from
saveConfigpreviously escaped toa raw stack trace.
initverifies the API key against Lara. It warns and continues on anyfailure, so the command still works offline.
LARA_SERVER_URLpoints the SDK at a non-production Lara deployment.Testing
pnpm test— 920 unit tests, 70 of them covering this module: tokenexchange, refresh-once-on-401, rejected key, cooldown, account id resolution
and its refusals, queue retention per status code, metadata shape.
pnpm run verify:metrics— 53 checks against the real built CLI with bothLara and the metrics backend faked on localhost, including the degradation
paths (backend down, Lara down, opt-out, un-baked build).
in ClickHouse with raw
acc_ids, unique event ids, no deprecatedfirst_call, and a rejected channel key drops the queue instead ofaccumulating it forever.
Before merging
METRICS_URLandMETRICS_API_KEYneed to exist as repository secrets — thepublish workflow bakes them into the published package, and a build without
them is inert. The backend has no deployment yet, so there is no host to point
at today.