Skip to content

reweight the combined metric for lukewarm and inapplicable metrics - #1634

Open
toschmidt wants to merge 1 commit into
ClickHouse:mainfrom
toschmidt:schmidt/reweight
Open

reweight the combined metric for lukewarm and inapplicable metrics#1634
toschmidt wants to merge 1 commit into
ClickHouse:mainfrom
toschmidt:schmidt/reweight

Conversation

@toschmidt

Copy link
Copy Markdown
Contributor

The Combined score is a weighted geomean of load (10%), data size (10%), cold (20%) and hot (60%) ratios. That unfairly penalizes systems for metrics that don't apply to them, and lets lukewarm "cold" numbers (really warm queries) distort the cold component.

Unify the per-metric exclusion rules in a single metricExcludes() helper (stateless from load, in-memory from cold/combined/load, lukewarm from cold, missing data size from size) and reuse it everywhere:

  • Cold Run metric: lukewarm systems are excluded from the ranking by default.
  • Combined per-query baseline: the cold-run minimum excludes lukewarm / in-memory systems, so their warm "cold" numbers can't depress the baseline and inflate every true-cold system's cold ratio. min load time / min data size likewise exclude systems that don't qualify.
  • Combined score: a metric that doesn't apply to a system is dropped and the remaining weights are renormalized, instead of feeding a bogus ratio. Lukewarm systems keep a cold component of 0 with its weight folded into hot (load 10% / size 10% / hot 80%); a stateless engine that still reports a load time (e.g. Polars (Parquet)) drops the load component; etc. The cold term is guarded so an all-lukewarm selection (empty cold baseline) can't poison the score with NaN.

@toschmidt

Copy link
Copy Markdown
Contributor Author

@alexey-milovidov can you take a look at this?

@alexey-milovidov

Copy link
Copy Markdown
Member

Depends on #1646

@toschmidt

Copy link
Copy Markdown
Contributor Author

@alexey-milovidov I rebased the PR and changed the flag to no-cold

The Combined score is a weighted geomean of load (10%), data size (10%),
cold (20%) and hot (60%) ratios. That unfairly penalizes systems for
metrics that don't apply to them, and lets lukewarm "cold" numbers
(really warm queries) distort the cold component.

Unify the per-metric exclusion rules in a single metricExcludes()
helper (stateless from load, in-memory from cold/combined/load, lukewarm
from cold, missing data size from size) and reuse it everywhere:

- Cold Run metric: lukewarm systems are excluded from the ranking by
  default.
- Combined per-query baseline: the cold-run minimum excludes lukewarm /
  in-memory systems, so their warm "cold" numbers can't depress the
  baseline and inflate every true-cold system's cold ratio. min load
  time / min data size likewise exclude systems that don't qualify.
- Combined score: a metric that doesn't apply to a system is dropped and
  the remaining weights are renormalized, instead of feeding a bogus
  ratio. Lukewarm systems keep a cold component of 0 with its weight
  folded into hot (load 10% / size 10% / hot 80%); a stateless engine
  that still reports a load time (e.g. Polars (Parquet)) drops the load
  component; etc. The cold term is guarded so an all-lukewarm selection
  (empty cold baseline) can't poison the score with NaN.

The Combined view still shows only the single overall score; the
per-component breakdown is added in a follow-up commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants