feat(obs): wire sgp-obs from the SDK for traces, metrics and logs by stephen-wang24 · Pull Request #523 · scaleapi/scale-agentex-python

stephen-wang24 · 2026-09-18T16:25:42Z

Wires sgp-obs from the SDK so an agent adopts traces, metrics and logs by installing a
package and setting environment, instead of carrying the wiring itself across ~147 agent
repos.

This is #518 re-landed on next (it was reverted in #522 while the behaviour was being
measured on sgp-dev), plus three follow-up fixes found during that measurement. First
commit is a revert-of-the-revert so the add → revert → re-add history stays legible.

No version change here. The 0.27.0b2 bump that exists on the beta branch (#521) is
deliberately left behind — .release-please-manifest.json, both pyproject.tomls and
_version.py are untouched at 0.26.0, so release-please keeps ownership. #520 already
proposes 0.27.0, and these commits just add feat(obs)/fix(obs) entries to it.

Safety for agents that will never have `sgp-obs`

sgp-obs is served from Scale's curated CodeArtifact mirror, not public PyPI, so most
agents will not have it. This change is inert for them, and that is measured rather
than argued.

It is not a dependency, and not an extra

There is deliberately no obs extra — adk/pyproject.toml documents why (declaring it
makes this repo's own uv workspace unresolvable: locking must resolve every declared
optional dependency and neither --no-extra nor override-dependencies exempts one).
The dependency is the agent's to declare; the SDK wires it only if it is importable.

Comparing the published 0.26.0 and 0.27.0b2 wheels:

	result
Declared dependencies (`Requires-Dist`)	32 entries each, zero differences
`sgp-obs` anywhere in package metadata	absent
Fresh install closure, clean py3.12 venv	133 packages each, `diff` empty
`pip install agentex-sdk` with no CodeArtifact access	resolves normally

No new transitive dependency either. (scale-gp-beta>=0.5.0 is sometimes attributed to
this work — it was raised in 0.23.0 and is already in current stable 0.26.0.)

Behaviour is unchanged

A 28-check A/B probe was run against both published wheels in clean venvs with no
sgp_obs, covering: module imports; logger handler count, formatter, propagation and
one-line-per-info(); BaseACPServer() construction, lifespan enter/exit, /healthz,
JSON-RPC dispatch and x-request-id handling; and the litellm gateway across happy path,
positional model (args[0]), exception propagation, CancelledError, streaming, and
mid-stream failure.

The two reports differ by one line: the version string.

Every import of sgp_obs sits inside a try. The logging rewrite is latched behind
route_loggers_to_root(), which only runs once the logs signal actually wires, so
make_logger is byte-for-byte equivalent otherwise.

The half-configured case is loud, not silent

SGP_OBS_ENABLED=true with the package absent still serves 200, reports
not_installed, and says so:

SGP_OBS_ENABLED is set but sgp-obs is not installed, so no telemetry will be produced.
Add sgp-obs[genai-auto,http,otlp] to this agent's dependencies (it resolves from Scale's
curated mirror, not public PyPI).

Independently confirmed by building and running a real agent image with no sgp-obs and
no obs environment (ilana_digital_twin on sgp-dev): image builds, sgp_obs absent,
init_sgp_obs() returns 'not_installed' and never raises, agent imports and serves, CI
green.

Builds are unaffected

All 38 scaffold Dockerfiles mount the CodeArtifact secret with required=false and carry
# syntax=docker/dockerfile:1.3, so a build with no secret proceeds on the plain install
path. These templates only affect newly scaffolded agents; an existing agent keeps its own
Dockerfile.

Two deltas that DO apply to everyone

Being straight about what is not a no-op. Neither is gated on sgp-obs.

Delta	Who	Detail
Shutdown up to +5s	agents with SGP tracing configured	`shutdown_sync_tracing_processors()` drains a queue nothing drained before
litellm per-call	agents using `LiteLLMGateway`	+0.2µs — within noise

Shutdown is a bug fix: the ACP lifespan only ever drained the async span queue, so a
sync agent dropped whatever business spans were still queued when the pod stopped —
including the ones an obs span's agentex.business_trace_id resolves to. The drain is
bounded (5s), runs the processors concurrently on daemon threads, and is fail-open.
Verified: a 30s stalled flush under the 5s budget returns at 5.00s and the process exits
at 5.53s, not 30s. asyncio.wait_for can stop awaiting a thread but cannot stop the
thread, and asyncio.run joins the default executor — hence daemon threads rather than
to_thread. Comfortably inside a default 30s terminationGracePeriodSeconds, but worth
knowing for SGP-tracing agents.

litellm: every completion now flows through a recorder. With sgp-obs absent that is
a stateless shared null object whose observe() is a pass-through and whose __aexit__
returns False, so it can never swallow a caller's exception. Measured safe under 50
concurrent calls.

The last commit is what makes this free. inference_call originally imported sgp_obs
per call, and Python does not cache a failed import, so it re-walked sys.path on
every model call — 62µs each with only five path entries, and a container has more. Now
resolved once, the same way base_acp_server already handles sgp_obs.context:

	per call
`0.26.0`	11.6µs
`0.27.0b2` as published	84.0µs
this branch	11.8µs

Timing note

uv excludes pre-releases from a bare agentex-sdk specifier, so 0.27.0b2 is invisible
to a fresh resolve today. Agents pinned loosely first pick this up when 0.27.0 ships
stable — which is what merging this and then #520 does. That is the intended rollout, but
it is the moment the above stops being theoretical, which is why it was measured first.

Testing

Full suite on this branch: 1714 passed, 0 failed (1622 skipped)
Obs-specific: 112 passed with no sgp-obs installed
ruff check src tests clean
Import-cache fix carries 3 new tests that count import attempts under a counting
__import__ hook: 50 calls give 50 attempts on the old body and 1 on this one

Confidence Score: 5/5

The PR appears safe to merge. The remaining earlier issue only affects processes that create more than one app.

Findings

The process-wide status returns before a later app reaches sgp obs.init app=... . If AgentexWorker initializes first in a combined process, or code creates a second BaseACPServer , that app never gets the middleware which this function says provides HTTP spans and incoming trace context. Keep process-wide provider setup idempotent, but track and instrument each app separately. ▶

Fix with agent prompt

### Issue 1
src/agentex/lib/core/observability/sgp_obs_setup.py:103-120
The process-wide `_status` returns before a later app reaches `sgp_obs.init(app=...)`. If `AgentexWorker` initializes first in a combined process, or code creates a second `BaseACPServer`, that app never gets the middleware which this function says provides HTTP spans and incoming trace context. Keep process-wide provider setup idempotent, but track and instrument each app separately.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The SDK now wires optional sgp-obs telemetry for ACP servers, Temporal workers, model calls, and generated agents. It also routes request IDs and logs into the telemetry pipeline and drains tracing data during shutdown, while agents without sgp-obs keep working normally.

Adds trace, metric, and log setup behind package and environment checks.
Records GenAI metrics for native and streamed LiteLLMGateway calls.
Updates generated Dockerfiles to use an optional private package index.
Flushes async spans, sync spans, and sgp_obs providers when processes stop.

Diagram

sequenceDiagram
    participant Agent
    participant SDK
    participant Obs as sgp-obs
    participant Backend
    Agent->>SDK: Construct ACP server or start worker
    SDK->>SDK: Check optional sgp-obs import
    alt Package installed and signals enabled
        SDK->>Obs: init(app, service_name, source)
        Obs-->>SDK: Wired signal handles
        SDK->>SDK: Route logs and install trace bridge
        Agent->>SDK: Handle request or model call
        SDK->>Obs: Bind request ID and record telemetry
        Obs->>Backend: Export traces, metrics, and logs
        Agent->>SDK: Shut down
        SDK->>SDK: Drain async and sync business spans
        SDK->>Obs: Flush within five seconds
    else Package absent or disabled
        SDK-->>Agent: Continue without telemetry
    end

_{Reviews (10) · Last reviewed commit: "docs(templates): the opt-in snippet prom..."}

This reverts commit 687ebfb.

…agentex's The hand-over added in #518 had two halves, and only one of them covered an agent's own modules. The latch in `make_logger` never looked at the logger's name, so anything created after init was fine. The sweep matched the `agentex` prefix, so a logger created BEFORE init under any other name kept its handler and went on printing a second, ungoverned copy of every record. That is not a corner case: agents call `make_logger(__name__)` from their own modules, and `project.acp` -- the module that builds the ACP server, in every scaffold -- logs at import, which is necessarily before `init_sgp_obs` runs. Measured on dbt-assistant running 0.27.0b1: 123 of 3361 log lines were the second copy, each 80 microseconds after its governed twin, carrying `name`/`request_id` but no `trace_id`, `span_id`, `source` or `agent_id`. Since it is emitted before the pipeline's filters, it also escapes the allowlist and the truncation. The SDK cannot know an agent's package name, so `make_logger` now marks each handler it attaches and the sweep takes back exactly those, on a logger of any name. Prefix matching stays for `agentex.*` itself, where every handler is ours by definition. A handler this module did not attach is still left alone -- litellm's three loggers and anything else keep what their owner set up, which is why sgp-obs warns about them rather than stripping them. Renamed `route_agentex_loggers_to_root` to `route_loggers_to_root`, since "agentex loggers" is what the bug was. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

`request_id` had exactly one writer: `CustomJSONFormatter`, on the handler `make_logger` attaches to each module's own logger. Removing that handler is what stops the duplicate line -- and it takes the field's only writer with it, so without this the id would not move to the governed copy, it would disappear. Measured on dbt-assistant: `request_id` was on 5.2% of lines, which were exactly the ungoverned copies, against `trace_id` on 90.6%. The ACP middleware now binds its id into sgp-obs' shared correlation context, which the logs pipeline enriches every record from. sgp-obs can also fill that context from its own `RequestIdMiddleware`; binding the SDK's id instead keeps ONE generator for the value, so the id in the logs is the id `ctx_var_request_id` gives application code and the id `x-request-id` carried in. Deliberately not written onto the record here. The pipeline's enrich stage runs on a copy of the record at handler time and treats a hand-set value as authoritative, which cannot collide with a caller's own field. Setting the attribute up front does collide: with `request_id` already on the record, the stdlib raises `KeyError: Attempt to overwrite 'request_id' in LogRecord` from `logger.info(..., extra={"request_id": ...})` -- measured, not theoretical, and an unacceptable way for telemetry to reach an agent. Fail-open throughout, and the optional import is resolved once: Python does not cache a failed import, so attempting one per request would re-walk sys.path for the majority of agents that never install sgp-obs. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Python does not cache a FAILED import, so `from sgp_obs.metrics import genai` inside inference_call re-walked sys.path on every single litellm call — for the majority of agents, which are the ones that do not have sgp-obs installed. Measured on the published 0.27.0b2 wheel in a clean venv with no sgp-obs, best of 5x500 calls against a mocked transport: 0.26.0 11.6 us/call 0.27.0b2 as published 84.0 us/call 0.27.0b2 + this fix 11.8 us/call The bare failed import is 62us of that, with only five sys.path entries; a container image has more. Negligible beside a real model call, but it is pure waste on the hot path, and the module docstring claimed the fallback "costs nothing", which was the one part of it that was not true. base_acp_server.py already solved exactly this for sgp_obs.context with an _OBS_CONTEXT_UNRESOLVED sentinel, and said why in a comment. This applies the same shape so the two files agree. The one-time debug line moves into the resolver, whose body now runs exactly once — which retires the separate _warned latch rather than leaving two latches for one fact. Adds _reset_for_tests(), matching sgp_obs_setup.py and utils/logging.py. The handle is process-wide state, so without it the first test to run with sgp-obs absent would cache None for the rest of the session and every later test that injects a fake sgp_obs.metrics would silently exercise the null path instead of the one it means to. Tests: 22 pass, up from 19. The new ones count import attempts under a counting __import__ hook — 50 calls give 50 attempts on the old body and 1 on this one — and pin that a present sgp-obs is still used on calls two and three, so the cache cannot degrade a working install to the null path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

stephen-wang24 · 2026-09-18T16:25:51Z

@greptileai

greptile-apps · 2026-09-18T16:34:28Z

+    if _status is not None:
+        # init() is not meant to run twice, and a Temporal worker plus an ACP
+        # server can both reach this in one process.
+        return _status


The process-wide _status returns before a later app reaches sgp_obs.init(app=...). If AgentexWorker initializes first in a combined process, or code creates a second BaseACPServer, that app never gets the middleware which this function says provides HTTP spans and incoming trace context. Keep process-wide provider setup idempotent, but track and instrument each app separately.

Knowledge Base Used: Observability

Prompt To Fix With AI

This is a comment left during a code review. Path: src/agentex/lib/core/observability/sgp_obs_setup.py Line: 89-92 Comment: The process-wide `_status` returns before a later app reaches `sgp_obs.init(app=...)`. If `AgentexWorker` initializes first in a combined process, or code creates a second `BaseACPServer`, that app never gets the middleware which this function says provides HTTP spans and incoming trace context. Keep process-wide provider setup idempotent, but track and instrument each app separately. **Knowledge Base Used:** [Observability](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/observability.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

…obs flush Three review findings from #523, all confirmed against the code before fixing. **The worker dropped its own business spans (P1).** Its `finally` called the sync and sgp-obs drains but not `shutdown_default_span_queue()`. Standard Temporal activities trace through `AsyncTracer` (core/temporal/activities/__init__.py:66), and `AsyncTrace.__init__` takes `get_default_span_queue()` when no queue is passed (core/tracing/trace.py:406) — so a worker's spans sit in exactly the queue nothing drained. The async drain now runs first, matching the ACP lifespan's order, and a test asserts both paths carry all three drains so they cannot drift apart again. **The sgp-obs flush had no deadline (P1).** `asyncio.to_thread(shutdown)` was awaited unbounded, so a hung exporter held both callers until the pod was killed. Worse, `asyncio.run` joins the default executor on the way out, so even adding a `wait_for` would not have helped — the process still blocks on the export the deadline was meant to escape. Measured, 20s stalled flush under a 0.25s budget: asyncio.to_thread + wait_for process exits at 20.04s daemon thread + wait_for process exits at 0.31s Now a daemon thread under a 5s budget, the same shape and the same reasoning as `shutdown_sync_tracing_processors`, whose docstring already spelled this out. The overrun is warned about rather than silent. Covered by a subprocess test, because interpreter shutdown cannot be observed from inside the test process. **A second app was silently uninstrumented (P2).** `init()` is process-wide and must not run twice, but the ASGI instrumentation it installs is per-app, so a later `BaseACPServer` — or an ACP server built after `AgentexWorker.run()`, which inits with no app — got no `http.server.*` and no ingress trace continuation, with nothing saying so. This warns instead, naming the ordering that causes it. That last one is deliberately a diagnostic rather than a repair: instrumenting the second app would mean calling into sgp-obs for a per-app entry point I cannot verify from here (it is not a dependency of this package and not on public PyPI), and guessing at an API is worse than making the silent case audible. Flagged for follow-up. Tests: 63 in the two obs suites, 1714 in the repo suite, ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

stephen-wang24 · 2026-09-18T16:55:42Z

@greptileai

harvhan · 2026-09-18T17:48:35Z

+    rest = model[len(_PROXY_PREFIX):] if proxied else model
+    vendor = rest.split("/", 1)[0] if "/" in rest else _DEFAULT_VENDOR
+    # Proxy mode always leaves over the OpenAI client. So does a native openai/* call.
+    return (vendor or _DEFAULT_VENDOR), proxied or vendor == _DEFAULT_VENDOR


issue (blocking): Azure calls already get metrics from the OpenAI client, so this code counts them twice. Claude calls without an anthropic/ prefix have the opposite problem: this code assumes OpenAI will count them, but they go directly to Anthropic and get no metrics.

suggestion: Use LiteLLM’s provider lookup to identify where each call goes, then skip recording only when that client already records metrics.

@harvhan

…efix Review finding from @harvhan on #523, confirmed against litellm 1.87.0 before fixing. Reading the vendor off the model prefix was wrong in two opposite directions, and both were silent. **Azure was counted twice.** `azure/gpt-4o` looks like a native vendor to a prefix reader, so this recorded it — but litellm serves azure with `openai.AzureOpenAI` (litellm/main.py, `if custom_llm_provider == "azure"`), so the OpenAI client instrumentor recorded it as well. The same held for every openai-compatible provider litellm supports: groq, deepseek, xai, fireworks_ai and ~50 others all carry a vendor prefix and all leave over the openai client. **Bare Anthropic models were not counted at all.** `claude-sonnet-4-20250514` is a legal litellm model string that resolves to provider `anthropic` and routes natively, but an unprefixed name was assumed to be OpenAI, so this stood down for an instrumentor that never saw the call. The provider now comes from `litellm.get_llm_provider` — the same resolution litellm uses to route — and the openai-client question from litellm's own `openai_compatible_providers` rather than a list of ours, since that list grows every release. Azure and four siblings are added explicitly; litellm dispatches them over the same client without listing them there. model before after azure/gpt-4o (azure, False) (azure, True) groq/llama3-8b-8192 (groq, False) (groq, True) deepseek/deepseek-chat (deepseek, False) (deepseek, True) claude-sonnet-4-20250514 (openai, True) (anthropic, False) anthropic/claude-sonnet-4 unchanged bedrock/anthropic.claude-v2 unchanged litellm_proxy/anthropic/claude-sonnet-4 unchanged The proxy prefix is still stripped before resolving, so a proxied call keeps reporting the vendor underneath — the one thing the OpenAI client instrumentor cannot report, and the reason this module exists. Resolution is cached behind a bounded dict. Beyond speed (1.9us -> 0.09us), it is what stops litellm's red "Provider List" banner — printed to STDOUT, not through logging, so it cannot be filtered — appearing on every call for a model litellm cannot place. Measured: 50 calls print it once, not 50 times. Redirecting stdout around the lookup was the alternative and is worse; it swaps a process-global, so under concurrency it would swallow other coroutines' output. A plain dict rather than lru_cache because `functools.lru_cache` is banned here (TID251) and the sanctioned one lives in `agentex._utils`, which `agentex/lib` does not otherwise import from. None of this touches agents without sgp-obs: `inference_call` returns the null recorder before `_split_model` is ever reached (verified). Gateway cost on that path, same harness back to back: 13.1us on 0.26.0, 11.9us here. Tests: 33 in this file (was 22), 166 across the obs suites, 1714 repo-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…goes CI's lint job runs pyright, which `ruff check` alone does not: two errors, both mine — `"openai_compatible_providers" is not exported from module "litellm"` (reportPrivateImportUsage). Pyright is right. The attribute is real at runtime but is not in litellm's `__all__`, so it is not a promised export and a future release may rename or drop it. Read via `getattr` now, which fixes the lint and, more to the point, degrades instead of raising if it does disappear. But degrading quietly would silently re-introduce the exact double counting the previous commit fixed: without litellm's list only five providers are known to reach the model over the OpenAI client, so groq, deepseek, xai and ~50 others would look native again and be recorded twice — once here and once by the OpenAI client instrumentor. So the fallback warns and names what it can no longer recognise. Covered by a test that deletes the attribute and asserts both halves: azure still resolves, groq no longer does, and the warning fires. Verified with the real CI entry points rather than a subset: `./scripts/lint` (ruff over the whole tree, pyright, import check) and `./scripts/check-slim-deps` both clean. Tests 34 in this file, 1714 repo-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Greptile P2 on #523, and it is right: the previous commit warned and then kept recording with a five-provider fallback, which marks groq, deepseek, xai and ~50 others as native. litellm still sends those over the OpenAI client, so both that client and inference_call record them. A warning does not prevent the bad metric data — it just annotates it. Both halves of the suggestion are taken. **A stable source.** The list is read from `litellm.constants`, where it is defined (constants.py:789), instead of the `litellm` top level, which is an incidental re-export. litellm declares no `__all__` at all, so the top-level name carries no stability promise even informally, which is what pyright was objecting to in the first place. Verified the two are the same object. **Stand down when it is unknown.** `_over_openai_client` now returns None for "cannot tell", and `_split_model`'s second element is `bool | None`. None is deliberately not collapsed into False, because treating an unknown provider as native is exactly what double-counts it. `inference_call` returns the null recorder on None, so no record is started at all. That choice is deliberate in one direction: standing down also drops the native providers (anthropic, bedrock, vertex_ai) that nothing else records, so the fallback loses real data. It is still the right trade. A doubled token or cost figure is silent and gets believed; a gap is visible, and this one is warned about by name. The path is also effectively unreachable — litellm is a hard dependency and the constant has been there for many releases — so the conservative branch costs nothing in practice. Verified end to end: with `litellm.constants` blocked and a working sgp-obs stub present, every model resolves to None and zero records are started. Tests: 35 in this file, covering both the None contract and the property that actually protects the data — no sgp-obs record is created when routing is unknown. `./scripts/lint` (ruff, pyright 0 errors, import) and `./scripts/check-slim-deps` clean; 1714 repo-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

harvhan · 2026-09-18T19:13:49Z

+    global _openai_client_providers
+    if _openai_client_providers is _OPENAI_CLIENT_PROVIDERS_UNRESOLVED:
+        try:
+            from litellm.constants import openai_compatible_providers


issue (blocking): In LiteLLM 1.87.0, Groq and DeepSeek use LiteLLM’s HTTP client even though they appear in openai_compatible_providers. Treating them as OpenAI client calls makes the gateway skip their duration and token metrics whenever OpenAI instrumentation is enabled.

suggestion: Check which Python client each provider actually uses and test Groq and DeepSeek with instrumentation enabled.

harvhan · 2026-09-18T19:13:49Z

 RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=codeartifact-pip-conf,required=false \
+    if [ -s /run/secrets/codeartifact-pip-conf ]; then \
+      export UV_INDEX="scale-pypi=$(sed -n 's#.*index-url = https://aws:[^@]*@\(.*\)#https://\1#p' /run/secrets/codeartifact-pip-conf | head -1)"; \


issue (blocking): This override drops default = true from the documented private index and restores public PyPI as a fallback. A dependency missing from the mirror can then be downloaded publicly, despite the project selecting the mirror as its only default.

suggestion: Use UV_DEFAULT_INDEX for the named broker URL in both install steps.

harvhan · 2026-09-18T19:13:49Z

+        ).start()
+
+        try:
+            await asyncio.wait_for(finished.wait(), budget_s)


issue (non-blocking): If metrics flushing reaches this timeout, traces have not started shutting down. Python then runs the trace provider’s automatic exit hook, which can keep flushing past this deadline.

suggestion: Make the shutdown budget cover the providers’ exit hooks and test actual process exit.

harvhan · 2026-09-18T19:13:49Z

+        _hand_logging_to_the_pipeline()
+
+    if "traces" in handles:
+        _install_openai_agents_bridge()


issue (non-blocking): SGP_OBS_GENAI_ENABLED=false still allows this bridge to export OpenAI Agents model spans. Deployments therefore cannot turn off GenAI tracing while keeping other traces enabled.

suggestion: Check the GenAI tracing flag before installing the bridge.

@harvhan

… in templates Four review findings from @harvhan on #523. Both blocking ones confirmed by measurement first; the two non-blocking ones handled differently, see below. **groq and deepseek were being dropped (blocking).** They appear in litellm's `openai_compatible_providers`, so the previous commit marked them as reaching the model over the openai client and stood down — but litellm catches both in an EARLIER branch of its dispatch chain that calls `base_llm_http_handler`, its own HTTP client, which the OpenAI instrumentor never sees. Nothing recorded them. That list describes API-FORMAT compatibility, not which Python client is used; membership is necessary but not sufficient. Eight of the 54 are intercepted this way: azure_ai, cometapi, deepseek, fireworks_ai, groq, hosted_vllm, ragflow, xai. They are now subtracted, and a drift test re-derives the set from the installed litellm's dispatch chain so an upgrade that moves a provider between branches fails CI instead of silently losing or doubling its metrics. Verified the guard bites: removing groq from the set fails with the provider named. **Scaffold builds could fall back to public PyPI (blocking).** `Dockerfile-uv.j2` exported `UV_INDEX="scale-pypi=<broker url>"`, which re-binds a NAMED index but does not make it the default — and the project's own `default = true` does not survive the re-bind, so PyPI stayed in the list. A package missing from the mirror was then fetched publicly. Measured against a local index that 404s every package, resolving `idna`: pyproject default = true, no override no solution correct UV_INDEX="scale-pypi=<broker url>" RESOLVED FROM PYPI the hole UV_DEFAULT_INDEX=<broker url> no solution correct All 19 uv templates now export `UV_DEFAULT_INDEX`, matching what `Dockerfile.j2` already did. This also keeps the property the named index was chosen for in the first place — not letting a project's own pyproject redirect where the token is sent. Verified with two local servers, the project pointing `scale-pypi` at a rogue host: rogue received zero requests, only the broker URL was contacted. The token stays percent-encoded, which is correct inside URL userinfo, so the decode step and the name-bound credential vars are gone. PRIVATE_INDEX.md rewritten — its "the two variants work differently, deliberately" section is no longer true. **GenAI tracing could not be switched off on its own (non-blocking).** The openai-agents bridge is installed by this package, not by `sgp_obs.init`, so there was no way to stop model spans short of disabling traces entirely. Gated on SGP_OBS_GENAI_ENABLED, defaulting ON so an unset variable changes nothing. Caveat: that variable name could not be verified against sgp-obs from this repo (it is not a dependency and not on public PyPI), and it appears nowhere in this codebase. If the real name differs, this gate simply never fires and behaviour is unchanged — but it is worth confirming. **The shutdown budget does not cover provider atexit hooks (non-blocking).** Confirmed: TracerProvider and MeterProvider both default `shutdown_on_exit=True` and do `atexit.register(self.shutdown)`, so if the budget expires those hooks still flush at interpreter exit. NOT fixed here, deliberately — unregistering them would also strip runtime-owned providers that sgp-obs leaves alone on purpose. Documented as a known limit with the fix that belongs in sgp-obs: build its own providers with `shutdown_on_exit=False`, or give `shutdown()` a deadline. Tests: 97 across the two obs suites, 1714 repo-wide. `./scripts/lint` (ruff, pyright 0 errors, import) and `./scripts/check-slim-deps` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ever got PRIVATE_INDEX.md told agent authors to put `default = true` on the mirror in their pyproject. That was wrong in two ways at once, and Greptile flagged the consequence on the Dockerfile side. It never took effect. The Dockerfiles export `UV_INDEX`, which binds the mirror as a NAMED index ahead of public PyPI rather than replacing it as the default, and a name rebound that way does not carry the project entry's default flag. Measured: with a mirror that 404s a package, `UV_INDEX` resolves it from public PyPI while `UV_DEFAULT_INDEX` fails the build. And it is not what we want anyway. The mirror is there to supply the Scale-internal packages that are not on public PyPI — sgp-obs, scale-memory — not to become the sole source for every dependency of every scaffolded agent. Making it the default would give ~147 agent repos a hard dependency on the mirror being healthy and complete for builds that have nothing to do with observability. So the snippet drops the flag and the doc now states the real resolution order: mirror first, public PyPI as fallback. sgp-obs can only come from the mirror because it exists nowhere else; an ordinary dependency the mirror lacks still resolves. The note explains why the flag is absent so it does not get helpfully re-added. No code change — the templates already behave this way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

stephen-wang24 and others added 4 commits September 18, 2026 09:21

Revert "revert(obs): remove sgp-obs beta changes (#522)"

ce84e31

This reverts commit 687ebfb.

greptile-apps Bot reviewed Sep 18, 2026

View reviewed changes

stephen-wang24 requested review from NiteshDhanpal and harvhan September 18, 2026 17:16

stephen-wang24 marked this pull request as ready for review September 18, 2026 17:17

harvhan requested changes Sep 18, 2026

View reviewed changes

stephen-wang24 and others added 2 commits September 18, 2026 11:02

greptile-apps Bot reviewed Sep 18, 2026

View reviewed changes

Comment thread src/agentex/lib/core/adapters/llm/_genai_metrics.py Outdated

stephen-wang24 requested a review from harvhan September 18, 2026 18:46

harvhan requested changes Sep 18, 2026

View reviewed changes

harvhan approved these changes Sep 18, 2026

View reviewed changes

stephen-wang24 force-pushed the stephen/agentex-sdk-sgp-obs-wiring branch from c08e70b to 4619b30 Compare September 18, 2026 20:39

greptile-apps Bot reviewed Sep 18, 2026

View reviewed changes

Comment thread src/agentex/lib/cli/templates/default/Dockerfile-uv.j2

stephen-wang24 merged commit 57e1146 into next Sep 18, 2026
48 checks passed

stephen-wang24 deleted the stephen/agentex-sdk-sgp-obs-wiring branch September 18, 2026 21:16

stainless-app Bot pushed a commit that referenced this pull request Sep 18, 2026

feat(obs): wire sgp-obs from the SDK for traces, metrics and logs (#523)

2bd031e

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

stainless-app Bot mentioned this pull request Sep 18, 2026

chore: release main #525

Merged

Navigation Menu

feat(obs): wire sgp-obs from the SDK for traces, metrics and logs - #523

feat(obs): wire sgp-obs from the SDK for traces, metrics and logs#523
stephen-wang24 merged 9 commits into
nextfrom
stephen/agentex-sdk-sgp-obs-wiring

stephen-wang24 commented Sep 18, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

stephen-wang24 commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Sep 18, 2026

Uh oh!

stephen-wang24 commented Sep 18, 2026

Uh oh!

harvhan Sep 18, 2026

Uh oh!

Uh oh!

harvhan Sep 18, 2026

Uh oh!

harvhan Sep 18, 2026

Uh oh!

harvhan Sep 18, 2026

Uh oh!

harvhan Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stephen-wang24 commented Sep 18, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Safety for agents that will never have sgp-obs

It is not a dependency, and not an extra

Behaviour is unchanged

The half-configured case is loud, not silent

Builds are unaffected

Two deltas that DO apply to everyone

Timing note

Testing

Confidence Score: 5/5

Findings

Summary

Uh oh!

stephen-wang24 commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

stephen-wang24 commented Sep 18, 2026

Uh oh!

harvhan Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

harvhan Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

harvhan Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

harvhan Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

harvhan Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stephen-wang24 commented Sep 18, 2026 •

edited by greptile-apps Bot

Loading

Safety for agents that will never have `sgp-obs`