Repository navigation
bug: OpenTelemetry Spans Not Displaying in Self-Hosted Instance Timeline #2821
Description
Provide environment information
Self-hosted Instance: │ │
- Trigger.dev Version:
main(Docker image from 2024-12-26) │ │ - Deployment: Docker Compose (official hosting/docker/webapp setup)
- Instance URL: trigger.devneurox.tech │ │
- Docker Images: │ │
- Webapp:
ghcr.io/triggerdotdev/trigger.dev:main(c3af7f05c1da) │ │ - Supervisor:
ghcr.io/triggerdotdev/supervisor:main(d3d1d21ad176)
Describe the bug
Issue: OpenTelemetry Spans Not Displaying in Self-Hosted Instance Timeline
Environment:
- Trigger.dev SDK: v4.3.1
- Self-hosted instance: trigger.devneurox.tech
- Cloud instance: cloud.trigger.dev
Problem Description:
OpenTelemetry spans created with logger.trace() are not visible in the timeline UI on our self-hosted Trigger.dev instance, but they work correctly on the official cloud instance (cloud.trigger.dev).
Configuration:
trigger.config.ts:
import { defineConfig } from "@trigger.dev/sdk";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { FetchInstrumentation } from "@opentelemetry/instrumentation-fetch";
export default defineConfig({
project: "proj_iahascmunqsxfplmrepp",
runtime: "node",
logLevel: "log",
telemetry: {
instrumentations: [
new HttpInstrumentation(),
new FetchInstrumentation(),
],
},
// ...
});
Code Example:
const marketData = await logger.trace("bybit-api-calls", async (span) => {
span.setAttribute("symbol", input.symbol);
span.setAttribute("endpoints", "getTicker, getOpenInterest, getLongShortRatio, getFundingRate");
return await bybitAPI.getMarketData(input.symbol);
});
Expected Behavior:
Timeline should show individual spans with hierarchy and timing.
Actual Behavior:
Self-Hosted: Timeline shows only one solid green bar - no spans visible
Cloud (cloud.trigger.dev): Timeline correctly shows all spans ✅
Steps to Reproduce:
Deploy same code with [logger.trace()](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html) spans to both self-hosted and cloud
Trigger task execution
Compare timeline view in dashboard
Additional Info:
Tasks execute successfully on both instances. Console logs show correct timing. Issue appears UI-specific to self-hosted.
Question: What self-hosted dashboard version is required for span visualization?
### Reproduction repo
N/A - Private repository. Can provide minimal reproduction example if needed.
### To reproduce
**Steps to reproduce:**
1. Configure Trigger.dev with OpenTelemetry instrumentations:
```typescript
telemetry: {
instrumentations: [
new HttpInstrumentation(),
new FetchInstrumentation(),
],
}
2. Add [logger.trace() spans in your task code:
const result = await logger.trace("operation-name", async (span) => {
span.setAttribute("key", "value");
return await someOperation();
});
3 Deploy to self-hosted instance (trigger.devneurox.tech)
4 Deploy same code to cloud instance (cloud.trigger.dev)
5 Execute task on both instances
6 Compare timeline view in dashboard
Expected: Timeline shows individual spans with hierarchy and timing
Actual on self-hosted: Timeline shows only one solid green bar - no span breakdown visible
Actual on cloud: Timeline correctly displays all spans with proper hierarchy ✅
Note: Tasks execute successfully on both - issue is UI-only on self-hosted.
### Additional information
<img width="751" height="345" alt="Image" src="https://github.com/user-attachments/assets/6eb7b2a4-6265-4c92-a2bd-d6d72e3eaa02" />
<img width="647" height="418" alt="Image" src="https://github.com/user-attachments/assets/d506c4ab-7812-4d50-ba20-73e56df4a86c" />
Activity
coderabbitai commentedon Dec 29, 2025
📝 CodeRabbit Plan Mode
Generate an implementation plan and prompts that you can use with your favorite coding agent.
Examples
🔗 Similar Issues
Possible Duplicates
Related Issues
Schema URLconflict in Trigger.dev deployment #2056👤 Suggested Assignees
🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
caiofeuser commentedon Jan 13, 2026
Same issue here
lecondor-dev commentedon Feb 2, 2026
+1
feat(webapp): add RUNTIME_API_ORIGIN to decouple runner traffic from …
feat(webapp): add RUNTIME_API_ORIGIN to decouple runner traffic from …
fix(webapp): add overridesBySpanId and environmentId filter for self-…
matt-aitken commentedon Aug 29, 2026
Thank you for the report. We investigated this.
The single green bar is the root span of the run. The webapp writes this span when the run is triggered. All child spans (
logger.trace, HTTP instrumentation) and alllogger.*output are sent from the run container to the webapp's/otelendpoint. The run container gets this address from theOTEL_EXPORTER_OTLP_ENDPOINTvariable on the supervisor. The Docker Compose default ishttp://webapp:3000/otel. If the run container cannot reach this URL, the run still completes, because the run lifecycle traffic goes through the supervisor, but no spans or logs arrive. The export failure is not logged by default.Please check the following:
OTEL_EXPORTER_OTLP_ENDPOINTpoints to a URL that resolves from inside a run container. In a split webapp/worker setup this must be the public webapp URL followed by/otel, not the internalwebapphostname.POSTrequests withapplication/x-protobufbodies and does not set a small body-size limit.POST /otel/v1/tracesrequests when a run executes.logger.logandlogger.infolines also missing from the dashboard? They use the same exporter. If they are missing too, the export path is the cause.To see exporter errors, set
TRIGGER_OTEL_LOG_LEVEL=erroras an environment variable on your project. The errors then appear in the run container output.If you store task events in ClickHouse (
EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2), also confirm that the clocks on the worker host and the webapp host are synchronised to within one minute. The trace query filters spans by start time relative to the run creation time.If none of this applies, please share your supervisor environment (with secrets removed) and the webapp log lines from one run, and we will look further.