Skip to content

feat: report usage metrics for the cli channel - #76

Draft
Starllordz wants to merge 8 commits into
mainfrom
feat/metrics-integration
Draft

feat: report usage metrics for the cli channel#76
Starllordz wants to merge 8 commits into
mainfrom
feat/metrics-integration

Conversation

@Starllordz

Copy link
Copy Markdown
Collaborator

Reports usage metrics for the cli channel, against the contract in
lara-integrations-monitoring@develop and the shared event-reporting standard.

What is sent

auth_success / auth_fail when a key is accepted or rejected, and
call_success / call_error when a translation finishes. One terminal event
per translate invocation; memory, glossary and init report auth only —
a lara memory list counted as a successful translation would move the funnel
for work that never happened.

Every event carries the raw Lara account id (acc_...), the CLI version, a
per-run session id, a per-event id, latency, characters translated, and a
metadata block with feature (text / document), surface and the
language pair when a single one is known.

Never sent: source or translated text, file names or paths, configuration,
credentials.

How it authenticates

Two steps, per the standard. The channel key buys a short-lived token from
POST /auth/issue-token with a persisted installation UUID; only that token is
accepted by POST /metrics/ingest-events. The key never reaches the ingestion
endpoint and the token never reaches the disk. On 401 the client asks for one
new token and retries once.

The account id is read from the JWT the Lara SDK already holds after it
authenticates — no extra round trip, no re-implementation of the signed
/v2/auth call. It is cached per access key id so a revoked or expired key can
still report auth_fail.

How it stays out of the way

Events are appended synchronously to a queue file and sent in one batch when
the command finishes. Nothing is flushed at start-up: that would put a token
exchange in front of the user's command before it had even validated its
arguments, to send events the closing flush picks up anyway.

The whole exchange is bounded to 2s. A slow or unreachable backend, a
read-only disk, a missing key, an un-baked build — all degrade to "no
telemetry", never to a failed translation. LARA_TELEMETRY_DISABLED=1 turns it
off entirely, and with it set the state directory is not even created.

The token endpoint rate limits per installation and answers 429 with a
Retry-After; the client records that deadline so a burst of commands does not
each pay a doomed round trip.

Changes to existing behaviour

  • Fatal Lara API errors now unwind instead of calling process.exit(1) on the
    spot, so a command always closes cleanly. HandledExitError carries the
    original error as its cause, which is what keeps the failure classified.
  • Every command routes through runSafely, which is now the single place that
    opens and closes the metrics window and reports failures. init gains the
    error handling it never had — a throw from saveConfig previously escaped to
    a raw stack trace.
  • init verifies the API key against Lara. It warns and continues on any
    failure, so the command still works offline.
  • LARA_SERVER_URL points the SDK at a non-production Lara deployment.

Testing

  • pnpm test — 920 unit tests, 70 of them covering this module: token
    exchange, refresh-once-on-401, rejected key, cooldown, account id resolution
    and its refusals, queue retention per status code, metadata shape.
  • pnpm run verify:metrics — 53 checks against the real built CLI with both
    Lara and the metrics backend faked on localhost, including the degradation
    paths (backend down, Lara down, opt-out, un-baked build).
  • Verified end to end against the monitoring backend running locally: rows land
    in ClickHouse with raw acc_ ids, unique event ids, no deprecated
    first_call, and a rejected channel key drops the queue instead of
    accumulating it forever.

Before merging

METRICS_URL and METRICS_API_KEY need to exist as repository secrets — the
publish workflow bakes them into the published package, and a build without
them is inert. The backend has no deployment yet, so there is no host to point
at today.

## Changed
- Fatal Lara API errors now unwind instead of exiting on the spot, so a
  command always closes cleanly before the process ends
- Every command routes through runSafely, which reports failures uniformly;
  init gains the error handling it never had
- init verifies the API key against Lara and warns if it cannot — it never
  blocks, and still writes the config offline

## New
- Usage metrics for the cli channel: auth_success, auth_fail, call_success
  and call_error, batched to a queue file and sent once per command
- Two-step ingestion auth: the channel key buys a short-lived token, which
  is the only credential the ingestion endpoint accepts
- Events carry the Lara account id, the CLI version, what was translated
  and the language pair — never text, paths, config or credentials
- Opt out with LARA_TELEMETRY_DISABLED=1; a build with un-baked keys is
  silently inert
- LARA_SERVER_URL points the SDK at a non-production Lara deployment
- verify-metrics-e2e.mjs exercises the built CLI end to end, degradation
  paths included
@Starllordz
Starllordz force-pushed the feat/metrics-integration branch from 60d0153 to 1178a3d Compare August 31, 2026 09:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a usage-metrics pipeline for the CLI channel (queue-on-disk + bounded flush + two-step auth), and refactors command error handling to unwind cleanly so metrics can be finalized on exit. This fits into the CLI’s operational telemetry/observability layer while keeping translation behavior “best effort” (telemetry must not break or slow commands).

Changes:

  • Introduces src/modules/metrics/* to queue events locally and flush them via token exchange (/auth/issue-token/metrics/ingest-events) with retries/cooldowns.
  • Centralizes command execution through runSafely() to instrument metrics, classify failures, and replace direct process.exit(1) call sites with unwindable errors.
  • Adds unit + E2E verification for metrics, plus a publish-time bake step to embed backend URL/key into the built package.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/utils/laraHeaders.ts Exposes a defensive accessor for the SDK internal client to support metrics identity resolution.
src/utils/error.ts Adds HandledExitError + reporting helper so callers unwind instead of exiting immediately.
src/modules/translation/translation.service.ts Routes Lara calls through auth-reporting wrapper and records metrics for translated chars/errors.
src/modules/metrics/metrics.ts Implements queueing, token exchange, flush, error classification, and per-command lifecycle.
src/modules/metrics/metrics.const.ts Defines metrics contract constants and baked placeholder values.
src/messages/messages.ts Adds user-facing messages for partial failures and init credential verification warning.
src/cli/cmd/translate/translate.ts Wraps translate command in runSafely, sets metrics context, and unwinds on failures.
src/cli/cmd/init/init.utils.ts Updates process.env after writing .env so the current process can validate credentials.
src/cli/cmd/init/init.ts Wraps init in runSafely and adds non-blocking credential validation/warn behavior.
src/cli/cmd/common/prompts.ts Updates Lara error handling to unwind under runSafely instead of exiting underneath it.
src/cli/cmd/common/command.ts Makes runSafely the metrics lifecycle owner and single error-to-exit conversion point.
src/tests/modules/metrics.test.ts Adds comprehensive unit tests for metrics behavior (queueing, token refresh, cooldown, etc.).
scripts/verify-metrics-e2e.mjs Adds E2E harness validating end-to-end contract with fake Lara + metrics backend.
scripts/bake-metrics-key.mjs Adds publish-time baking of backend URL/key into the compiled output.
README.md Documents usage-metrics behavior and opt-out.
package.json Adds verify:metrics script.
eslint.config.js Adds Node globals to ESLint config (needed by new Node APIs usage).
.github/workflows/publish.yml Adds “Bake metrics key” step using repository secrets.
Suppressed comments (2)

src/modules/metrics/metrics.ts:253

  • There is a stray JSDoc line ("True when events can actually be attributed and sent.") immediately followed by another JSDoc block. As written, the first block attaches to nothing and is likely to trigger lint/doc tooling issues.
/** True when events can actually be attributed and sent. */
/**
 * Never blocks, never touches the network, never throws. Every guard telemetry
 * has lives here: this is the only door onto the queue, so nothing upstream has
 * to re-check whether reporting is on.

README.md:237

  • The README states events are sent "when the command finishes, and again at the start of the next one". The code never flushes at command start; queued events are retried on the next command's closing flush (finishAndFlush), so this should be reworded.
Events are appended to a local file and sent in one batch — when the command finishes, and again
at the start of the next one if anything was left over. If the metrics backend is slow or
unreachable the CLI behaves exactly as it does today: the send is bounded to two seconds and
whatever could not be delivered simply waits for the next run.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/modules/translation/translation.service.ts
Comment thread src/modules/metrics/metrics.ts
Comment thread src/cli/cmd/common/prompts.ts Outdated
Comment thread README.md Outdated
## Changed
- auth_fail is reported even when no Lara account id is known yet — a key
  rejected on first use is exactly the funnel step that was being dropped
- accountId is omitted when unknown rather than sent empty; call_success and
  call_error still require it, since the backend rejects the whole batch
- queued events older than the backend's 30-day bound are dropped at flush,
  so one stale line no longer takes every fresh event down with it
- a 429 from the ingestion endpoint now honours Retry-After across processes,
  instead of retrying on the very next command

## New
- e2e scenario for a first-ever key Lara refuses: auth_fail delivered with no
  accountId, no call event, nothing faked
## Changed
- a Lara error that comes back past the auth check (400, 402, 429) now reports
  auth_success: the key was accepted, and the funnel was losing users who got
  in fine and then ran out of credit
- the module docstring and the README no longer claim a flush at the start of
  a command; there has only ever been one, on the way out
- the README says what actually triggers each event: a translation command
  always closes with call_success or call_error, even when it fails before
  reaching Lara, while init, memory and glossary only report the key
- dropped a stray doc comment left attached to nothing above queueEvent

## New
- e2e scenario for a valid key with no credit left: auth_success followed by
  call_error / payment_402

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/cli/cmd/init/init.ts:118

  • The comment says “Never blocks”, but checkCredentials() awaits a real network call via TranslationService.validateCredentials() (which is bounded by the SDK request timeout, currently 30s). If Lara is unreachable, init can still pause noticeably before proceeding. Either tighten the wording or enforce a shorter timeout for the credential check.
 * Turns the key the user just entered into a real yes/no answer, and reports it.
 * Never blocks: `init` writes the config either way, so the command still works
 * offline or behind a proxy that is momentarily down.

Comment thread src/modules/metrics/metrics.ts Outdated
Comment thread README.md Outdated
## Changed
- DO_NOT_TRACK now switches metrics off exactly like LARA_TELEMETRY_DISABLED:
  no events, no requests, not even an installation id on disk
- an opt-out variable set to 0, false or an empty value is read as "leave
  telemetry alone" — before, any value at all switched it off, so =0 quietly
  did the opposite of what it says

## New
- a one-line notice on the first run that records anything: what is reported,
  what never is, and how to turn it off. Once per machine, on stderr, so a
  piped stdout is untouched — the README is not somewhere consent can hide
## Changed
- a file the user's own tooling broke now lands as `parse_error` instead of
  `unknown`: the per-file failure is recorded where it happens, so the funnel
  sees it rather than the generic "some files failed" the command unwinds with
- the first-run notice and the README no longer enumerate the events and
  fields; the README keeps one line on what is recorded, including the Lara
  account id, so the "never sent" list stays true

## New
- `metadata.fileTypes`: the formats a run touched, as one sorted value such as
  `json,po` — read from the config's own file types, never a name or a path
## Changed
- init writes the config before verifying the key, so a slow or unreachable
  Lara can no longer hold up the file the command exists to write; with no
  credentials at all it stops warning twice for the same reason
- what a Lara status code means is decided in the metrics module now, not in
  the translation service: the service reports the answer, metrics reads it
- an error the retry loop recovers from is no longer recorded, so a transient
  failure the user never saw cannot end up as the run's errorType
- every branch of the per-file catch in translate reports the real error, not
  just the one that happened to have the call
- dropped a two-entry allowlist, a one-caller error helper and a cast the
  caller had to repeat; the SDK's internal client now has a type

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a couple of concrete shutdown/telemetry reliability issues (spinner cleanup on error and malformed queued-event timestamp handling) that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/modules/metrics/metrics.ts:460

  • flushQueue() currently keeps events whose timestamp is missing (typeof !== 'string'). Those events violate the stated contract (every event has a timestamp) and can cause the backend to reject the whole batch, dropping otherwise valid events when the response becomes 400. Filter out events without a valid timestamp so one malformed line cannot poison a batch.
      events = kept
        .map((line) => JSON.parse(line) as { timestamp?: unknown })
        .filter(
          (event) =>
            typeof event.timestamp !== 'string' || Date.parse(event.timestamp) >= cutoff
        );
  • Files reviewed: 18/18 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread src/cli/cmd/translate/translate.ts
## Changed
- the queue only sends events the backend can actually accept: a line with an
  unreadable, missing or far-future timestamp is dropped rather than taking the
  whole batch — and every fresh event in it — down with it
- the progress bar stops before the command unwinds, so it no longer animates
  over the closing flush and leaves the terminal mid-redraw
- an error that was already printed now unwinds carrying the words the user
  actually read: the reporting helper hands back the message it displayed,
  instead of each caller inventing a second wording
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants