Skip to content

fix: Remove trust in compromised emSign ECC Root CA - G3 - #1715

Open
dervoeti wants to merge 6 commits into
mainfrom
fix/emsign-ca
Open

dervoeti wants to merge 6 commits into
mainfrom
fix/emsign-ca

Conversation

@dervoeti

@dervoeti dervoeti commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Description

Mitigates GHSA-6q48-6c7w-9gw2 (certifi) and GHSA-hq85-5cpj-mr62 (uv).

The private keys of eight eMudhra intermediate CAs chaining to emSign ECC Root CA - G3 were compromised. Mozilla removed the root from NSS on 2026-10-20. The UBI ca-certificates package dropped it in 2026.2.92, which #1716 picked up for the system trust store. This also covers Java, because the Temurin cacerts is a symlink to the extracted system store. Several bundled trust stores in our images still contain the root though:

  • airflow: Bump certifi to 2026.10.19 and uv to 0.13.4. Bump the pip bundled with the uv-managed Python to 26.2.2. On Airflow 2.9.3 (Python 3.9) pip 26.2.2 is not available, so PIP_CERT points pip to the system store instead. Point grpcio to the system store via GRPC_DEFAULT_SSL_ROOTS_FILE_PATH.
  • superset: Bump certifi to 2026.10.19 and uv to 0.13.4. pip gets 26.2.2 on rebuild, as its upgrade is unpinned.
  • opensearch-dashboards: Set NODE_OPTIONS=--use-openssl-ca, so Node.js uses the system store instead of its bundled roots. Node 20 had no fixed release when I checked.
  • spark-connect-client: Point grpcio to the system store.

testing-tools gets the fixed certifi on rebuild, as it is not pinned there.

Tested with locally built images against https://dv-g3b-emudhra.we-have-a-problem.com, which serves a certificate issued by one of the compromised intermediates. The patched images reject it with Python, uv and Node.js, while the current images accept it. A java-base rebuilt on top of #1716 rejects it with curl and Java, while java-base:21-stackable26.7.0 accepts it.

Definition of Done Checklist

Note

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.

Please make sure all these things are done and tick the boxes

  • Changes are OpenShift compatible
  • All added packages (via microdnf or otherwise) have a comment on why they are added
  • Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
  • All packages should have (if available) signatures/hashes verified
  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

boil build <IMAGE> --image-version <RELEASE_VERSION> --strip-architecture --load
kind load docker-image <MANIFEST_URI> --name=<name-of-your-test-cluster>

See the output of boil to retrieve the image manifest URI for <MANIFEST_URI>.

@dervoeti
dervoeti force-pushed the fix/emsign-ca branch 2 times, most recently from cc48318 to f760b56 Compare October 26, 2026 08:59
@dervoeti dervoeti moved this to Development: Waiting for Review in Stackable Engineering Oct 26, 2026
@dervoeti dervoeti self-assigned this Oct 26, 2026
@Techassi Techassi moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Oct 26, 2026

@Techassi Techassi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a few nits.

cattrs==23.2.3
celery==5.4.0
certifi==2024.7.4
certifi==2026.10.19

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is 2026.10.23. Any reason why we don't use that version?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I built this end of last week and there the version wasn't released yet :) Updated.

Comment thread airflow/Dockerfile
Comment on lines +344 to +346
ENV PIP_CERT=/etc/pki/tls/certs/ca-bundle.crt
# Same for grpcio, which bundles its own roots.pem
ENV GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/pki/tls/certs/ca-bundle.crt

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some links to where there env vars are documented/explained?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cd2b3e5

# Node.js uses its own bundled copy of the Mozilla root certificates instead of the system trust store.
# Use the system trust store instead, so Node.js does not trust the compromised
# "emSign ECC Root CA - G3" root certificate (GHSA-6q48-6c7w-9gw2), which is removed in stackable-base.
ENV NODE_OPTIONS="--use-openssl-ca"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a link to the documentation for the --use-openssl-ca option?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cd2b3e5

@dervoeti
dervoeti requested a review from Techassi October 26, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

2 participants