Firebolt: keep the three entries in the no-cold bucket - #1724
Conversation
GitHub Actions
Error
|
@alexey-milovidov, can you please take a look? |
Firebolt Core does not handle the start / stop cycle ClickHouse#1666 introduced gracefully as of Sept 2026, so those runs do not lead to meaningful measurements. Restore BENCH_RESTARTABLE=no and the no-cold tag on firebolt, firebolt-parquet and firebolt-parquet-partitioned, note at each start/stop that the cycle is off, and add a re-measured c8g.metal-48xl result taken in that configuration. The firebolt entry now pulls engine:dev_mv1, a build we verify and re-tag by hand, rather than engine:dev, which moves with every commit on our mainline. Caches are still flushed before the first run of each query, as for every other BENCH_RESTARTABLE=no system. The result files ClickHouse#1666 produced were true cold runs and keep no tag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30d6425 to
f181af8
Compare
GitHub Actions
Waiting
How exactly? |
|
@alexey-milovidov, thanks a lot for the fast response!
Essentially, we are developing for an "always up" type of usage. E.g., certain per table metadata takes time to load -- we don't have (and do not plan on) optimizing for super-fast truly cold queries. Some of this noise creeps into the 2 warm runs as well. I think the "no-cold" bucket simply makes more sense for firebolt as of today. If this gives the default view with the combined metric an "unfair picture" of Firebolt (and other no-cold systems), you could consider this PR . I hope I'm not misunderstanding the intention of that PR, but I think this is about not counting the cold runs or the load-times for systems that don't actually measure "true cold runs" or "true load-times". Please correct me if I'm wrong. |
2 participants
Firebolt does not handle the start / stop cycle from #1666 gracefully as of Sept 2026, so those runs do not lead to meaningful/interesting measurements. This restores
BENCH_RESTARTABLE=noand theno-coldtag onfirebolt,firebolt-parquetandfirebolt-parquet-partitioned.Note that we continue to flush caches as all other
BENCH_RESTARTABLE=nosystems, see here.Also adds a
c8g.metal-48xlresult measured in that configuration.The result files #1666 produced were true cold runs and keep no tag; new runs will carry it again.
Additionally, switches the
fireboltentry fromengine:devtoengine:dev_mv1, a build we are verifying and re-tagging by hand, so that any mainline regressions cannot walk into the benchmark unnoticed.