Skip to content

Upgrade Elasticsearch to 9.5.3 - #2416

Open
ejsmith wants to merge 20 commits into
mainfrom
feature/elasticsearch-9-compatibility
Open

Upgrade Elasticsearch to 9.5.3#2416
ejsmith wants to merge 20 commits into
mainfrom
feature/elasticsearch-9-compatibility

Conversation

@ejsmith

@ejsmith ejsmith commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Upgrade the ES9 base from 9.5.0 to 9.5.3 across the custom mapper-size image, all-in-one application image, AppHost, Compose/samples, Helm, and Elastic deployment manifests.
  • Rebase on main (5121e734). Retain the existing application queries, index schemas, and Elasticsearch 8 client compatibility bridge; this base runs independently of JOIN experiments.
  • Keep the supported 8.x image source current at 8.19.21 and cross-link the independent rollout PR Upgrade Elastic Stack 8 to 8.19.21 #2563, which should be deployed and soaked before the major upgrade.
  • Publish content-addressed candidate images for branch checks; keep stable image tags behind default-branch publication. Preserve configurable isolated local resources and readiness validation.
  • Correct the all-in-one sample's pre-existing Kibana dependency/host to use the service containing Elasticsearch.

Why 9.5.3

Elastic documents a bulk-indexing defect in 9.5.0–9.5.2 that can cause replica divergence and excessive translog/recovery work; 9.5.3 fixes it. The patch range also includes query correctness, circuit-breaker accounting, and security fixes, including Elasticsearch request smuggling, CVE-2026-78605, inference authorization, CVE-2026-78607, and Kibana Fleet authorization, CVE-2026-82302. This review is not an exhaustive image vulnerability scan.

Compatibility and migration

The production migration plan separates the server upgrade from index-format maintenance and the future repositories-led query refactor. #2511 is an experiment only, not the production implementation or a planned deployment. The proper stack/event query refactor must be designed in Foundatio.Repositories and integrated into Exceptionless separately.

Supported 8-created indexes do not require a bulk rewrite just to run Elasticsearch 9. Writable pre-8 indexes must be handled on 8.19 before starting 9. A disposable earlier rehearsal demonstrated the experimental expression JOIN against unchanged 8-created event data, but does not establish production-scale readiness.

Run Upgrade Assistant, use a current restore-tested snapshot, approve exact image digests, and validate shard/disk headroom before the ECK major upgrade. The available production identity could inspect topology but not actual index sizes/creation versions or free disk; sizing remains a separate prerequisite. Elasticsearch cannot downgrade an upgraded data directory. There are no application public API or index-schema changes in this base PR.

Both 8.19.21 and 9.5.3 were released in the first week of September and are less than two weeks old at this review. Stage and soak the exact images before production. Review custom ILM/inference configuration and Kibana HTTPS email-proxy certificates; no affected configuration was found in the checked-in settings. ECK itself remains unchanged.

Verification

  • Solution build: zero warnings/errors.
  • Custom Elasticsearch 9.5.3 image built successfully with mapper-size.
  • Complete backend suite on isolated custom ES9.5.3 (port 9219): 2,927 passed, 3 opt-in tests skipped, 0 failed. The all-in-one Exceptionless application image also built successfully.
  • All five Compose configurations validate. Existing obsolete Compose version warnings remain.
  • Candidate publisher/resolver scenarios pass for feature/default branches and same-repository/fork PRs, with Docker mocked to prevent local publication.
  • Hosted CI must be checked at the new published head; older 9.5.0/9.4.x results are not current-head validation.

Sources: 9.5.3 release announcement, Elasticsearch release notes, upgrade preparation.

@ejsmith ejsmith changed the title Add Elasticsearch 9 compatibility testing Upgrade Elasticsearch to 9.4.2 Jul 28, 2026
@ejsmith
ejsmith marked this pull request as ready for review July 28, 2026 23:54

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44efc54401

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/Exceptionless.Tests/AppWebHostFactory.cs Outdated
Comment thread .github/workflows/elasticsearch-docker-9.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa3c59c52f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/elasticsearch-docker-9.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d75931504e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docker/docker-compose.dev.yml Outdated
@ejsmith ejsmith changed the title Upgrade Elasticsearch to 9.4.2 Upgrade Elasticsearch to 9.4.4 Jul 29, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74ab19f674

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/build.yaml

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1cfba76585

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/build.yaml Outdated
Comment thread src/Exceptionless.AppHost/Program.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1726e1d340

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/build.yaml
@ejsmith ejsmith changed the title Upgrade Elasticsearch to 9.4.4 Upgrade Elasticsearch to 9.5.0 Aug 4, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c63bd7c7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/build.yaml

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52cc62c213

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Dockerfile Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 135ed0b399

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/build.yaml

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c767edc324

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docker/docker-compose.apm.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3039b22c49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

push:
paths:
- "build/docker/elasticsearch/9.x/**"
- ".github/workflows/elasticsearch-docker-9.yml"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep Docker Hub secrets out of branch-controlled workflows

When a same-repository contributor pushes a feature-branch edit to this workflow, this self-referential path trigger executes that unreviewed workflow definition with DOCKER_USERNAME and DOCKER_PASSWORD at lines 33–37. The default-branch tag guard is therefore not a security boundary: the branch can modify or remove the guard, publish arbitrary mutable images, or replace the login/build steps to disclose the credentials before review. Run branch candidates through a workflow whose privileged portion comes from the default branch, or protect the Docker Hub secrets with an approval-gated environment.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: the default-branch tag guard prevents accidental stable-tag publication but is not a security boundary against a writer modifying a branch-controlled workflow. Leaving this finding open pending the owner decision on a trusted publisher or approval-protected credentials. I have not changed repository secret protections or represented the tag guard as solving this trust issue.

@ejsmith
ejsmith force-pushed the feature/elasticsearch-9-compatibility branch from 3039b22 to ce201fd Compare September 6, 2026 23:47
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-07T00:01:57.389858Z d6a093e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ejsmith ejsmith changed the title Upgrade Elasticsearch to 9.5.0 Upgrade Elasticsearch to 9.5.3 Sep 6, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce201fddc0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/docs/self-hosting/upgrading-self-hosted-instance.md Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6a093e24a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


For `samples/docker-compose.all-in-one.yml`, run these commands from the existing deployment directory with its existing Compose project name and environment. Do not create a new project or change the volume mapping: that can silently select an empty data volume. Take and restore-test the snapshot first. Stop external Exceptionless jobs, ingestion consumers, and other writers too.

1. Pin `exceptionless` to an approved **8.x all-in-one application image containing Elasticsearch 8.19.21**, and `kibana` to `docker.elastic.co/kibana/kibana:8.19.21`. Do not use `latest` or an Elasticsearch 9 image during preparation. Keep the original volume, security settings, and resource limits.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Publish the documented 8.19.21 all-in-one bridge

For existing samples/docker-compose.all-in-one.yml installations, the new quiesced procedure requires an 8.x exceptionless/exceptionless image containing Elasticsearch 8.19.21, but this commit changes the only all-in-one stage directly to 9.5.3; the 8.19.21 Dockerfile and workflow publish only exceptionless/elasticsearch, and a repository-wide search finds no all-in-one 8.19.21 build. Thus users cannot perform the documented preflight without constructing an undocumented image themselves. This is fresh evidence beyond the earlier quiescing concern: publish and name the required bridge image before directing users to pin it.

AGENTS.md reference: AGENTS.md:L72-L74

Useful? React with 👍 / 👎.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.Insulation 37% 35% 286
Exceptionless.Web 85% 70% 8081
Exceptionless.Core 76% 68% 10418
Exceptionless.AppHost 66% 59% 178
Summary 79% (26175 / 33050) 68% (12212 / 17951) 18963

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants