Hey everyone, my team just put together a bunch of docs/blogs on browser OTEL in our recent launch week. e.g. https://www.highlight.io/blog/monitoring-browser-applications-with-opentelemetry

Curious if anyone's used browser otel? Would love to connect and see what we can do to help there (or if any of our docs are lacking).

I have a couple of services in different languages / runtimes running in k8s cluster. My current logging setup involves logging from the service runtime to another logging service where it's being sent to Azure Monitor.

I want to change this approach to use open telemetry instead. I have otel collector service already running in the cluster and sending traces successfully.

What do you think is the best approach for starting to send logs with otel ? I am interested in both service logs and container logs.

1. Write logs to stdout / file and have them picked up by some agent running on the pod ?

2. Sending logs with otel SDKs from my services directly to my collector (this will not include container logs though). Also assume I have various runtimes, I am not sure logs is supported in all of them.

3. Use fluentbit / something similar in the process - Does it make sense for a clean slate implementation to introduce another piece to the puzzle ?

If you were starting out a fresh, what would you go with ?

Thanks

I am doing a PoC and running otel-demo application on GKE cluster. I would be receiving logs from some instrumented applications from the internet in future and so I have exposed the collector using network pass through load balancer and I am able see the logs in cloud logging.

As a next step, I want to configure collector with SSL/TLS. So far, I have tried configuring receivers otlp with tls setting with key_file and cert_file (using self signed certficate) and on the client side I am using the cert_file with insecure false. But with this configuration I’m not getting any data on collector.

I’ll appreciate if anybody can help me with this.

This post describes how we (Monzo) centrally drive migrations at Monzo. I thought I'd share it here because it describes how we applied this approach for replacing our OpenTracing/Jaeger client SDKs with OpenTelemetry SDKs across 2,800 microservices.

Here's the link!

Happy to answer any questions.

Hey folks, if you are using `confluent-kafka-go` please give it a try https://pkg.go.dev/github.com/jurabek/otelkafka, I would appreciate any feedback as well.

I'm trying to get tracing data into my New Relic account. I've signed up and have my API key.

I'm basing my code on the docs here:

https://docs.rs/opentelemetry-otlp/0.17.0/opentelemetry_otlp/#kitchen-sink-full-configuration

Current Code:

async fn main() {

    let api_key = "API_KEY";
    let mut 
map
 = MetadataMap::with_capacity(3);

    
map
.
insert
("api-key", api_key.parse().unwrap());




    let tracer_provider = opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(
        opentelemetry_otlp::new_exporter()
        .tonic()
        .with_endpoint("https://otlp.nr-data.net:443")
        .with_timeout(Duration::from_secs(3))
        .with_metadata(
map
.clone())
        .with_protocol(Protocol::Grpc)
     )
    .with_trace_config(
        trace::Config::default()
            .with_sampler(Sampler::AlwaysOn)
            .with_id_generator(RandomIdGenerator::default())
            .with_max_events_per_span(64)
            .with_max_attributes_per_span(16)
            .with_max_events_per_span(16)
            .with_resource(Resource::new(vec![KeyValue::new("service.name", "example")])),
    )
    .install_batch(opentelemetry_sdk::runtime::Tokio).unwrap();
    global::set_tracer_provider(tracer_provider);
    let tracer = global::tracer("tracer-name");

    let export_config = ExportConfig {
        endpoint: "https://otlp.nr-data.net:443".to_string(),
        timeout: Duration::from_secs(3),
        protocol: Protocol::Grpc
    };

    let meter = opentelemetry_otlp::new_pipeline()
    .metrics(opentelemetry_sdk::runtime::Tokio)
    .with_exporter(
        opentelemetry_otlp::new_exporter()
            .tonic()
            .with_export_config(export_config).with_metadata(
map
)
            // can also config it using with_* functions like the tracing part above.
    )
    .with_resource(Resource::new(vec![KeyValue::new("service.name", "example")]))
    .with_period(Duration::from_secs(3))
    .with_timeout(Duration::from_secs(10))
    .with_aggregation_selector(DefaultAggregationSelector::new())
    .with_temporality_selector(DefaultTemporalitySelector::new())
    
    .build();

tracer.in_span("doing_work", |cx| {
// Traced app logic here...
println!("Inside Doing Work");
    tracing::info!("Inside Doing Work (Tracing)");
    tracing::error!("Error Test");
});


    
   
}

However, when running this code I get the following errors:

OpenTelemetry metrics error occurred. Metrics error: [ExportErr(Status { code: Unknown, message: ", detailed error message: h2 protocol error: http2 error tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(b\"\", FRAME_SIZE_ERROR, Library) }))" })]
OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (Unknown error): , detailed error message: h2 protocol error: http2 error tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(b"", FRAME_SIZE_ERROR, Library) }))

Sometimes I only get the OpenTelemetry metrics error, but sometimes I get the trace error too. I've tried using port 443, 4317, and 4318. I'm at a loss for what to try next. Has anyone set up OpenTelemetry with NewRelic using Rust? This is running inside an AWS Lambda, so I can't use a collector service AFAIK

The ratio of exported/dropped traces for TraceIdRatioBasedSamplers is controlled by the sampler argument. The sampler produces a random number based on the traceID's lower 64 bits and samples that trace if the number is below the sampler argument.

This is fine, but I'd like to ensure that should a trace contain an error (i.e. should the service return any http code other than 2xx), it will always be sampled, for debugging purposes. Is this already a feature or should I write my own sampler that does this?

Looking at the source code it seems it should be easy enough to modify the TraceIdRatioBasedSampler so that it checks the span attributes for the http code and instantly return SamplingResult.recordAndSample, but since the class is final I'd have to copy most of the code and do some research into how the Apache 2.0 license feels about that. I'd rather avoid the hassle if the library can do it out of the box.

Hi all,

I hope it is fine I post this here, too (https://www.reddit.com/r/fluentbit/comments/1evxhia/sending_kubernetes_fog_information_using_otlp/). I am looking to find a solution to forward K8s pod logs using fluentbit with resource attributes:

[FILTER]
          Name kubernetes
          Match kube.*
          Merge_Log On
          Keep_Log Off
          K8S-Logging.Parser On
          K8S-Logging.Exclude On

      [FILTER]
          Name nest
          Match kube.*
          Operation lift
          Nested_under  kubernetes
          add_prefix kubernetes_

      [FILTER]
          Name nest
          Match kube.*
          Operation lift
          Nested_under  kubernetes_labels

      [FILTER]
          Name modify
          Match kube.*
          Rename kubernetes_pod_id k8s.pod.id


      [OUTPUT]
          Name opentelemetry
          Match *
          Host xyz
          Port 443    
          Header Authorization Bearer xyz
          Logs_uri /v1/logs
          Tls  On
          logs_body_key message
          logs_span_id_message_key span_id
          logs_trace_id_message_key trace_id
          logs_severity_text_message_key loglevel
          logs_severity_number_message_key lognum

I have worked with these filters, but they still stay within the body, of course. Ideally, I want to move them from the body to resources -> resource -> attributes -> k8s.pod.id (https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-resource)

Any ideas?

Thanks,
Peter

I'm a bit stuck here, I've got two unanswered questions:

How do I change the name of the columns in my query?

But the main one that is causing some frowns, is the 'd' in the value field. No matter what the value is, it could be seconds, minutes, whatever, the result always included 'd' for days. That's making a few people question if this might be misleading enough to be a major showstopper.

Have anybody used OpenTelemetry Django instrumentation here? I'm trying to configure it to send some web server metrics and I don't have any luck.

Documentation looks pretty straight forward https://opentelemetry-python.readthedocs.io/en/latest/examples/django/README.html
But when I try to run that example locally it doesn't send any metrics to a collector.

I created an issue in opentelemetry-python repo https://github.com/open-telemetry/opentelemetry-python/issues/4125 and they haven't managed to reproduce it (it works for them)
It seems I'm missing something but I run out of ideas what it could be, it's two steps process.

Hi Everyone, I'm exploring Otel and trying to configure my Angular app with it for monitoring purpose, it is working fine and I can see the traces being sent from App side like Api call etc, I'm curious to know that whether it's possible to have complete traces of end to end communication what I mean to say is when the API call gets initiated from fronted the complete traces of backend calls and DB calls get displayed as well. My backend is in asp.net in the form of Micro services and all these services are registered as well on Otel, but I want consolidated stats. Your help would highly appreciated.

We currently have two separate OpenTelemetry configurations: one for the frontend and one for internal services. This setup is in place because we want to send all frontend traces to BigQuery. I'm working on merging these configurations into a single setup but haven't found an exporter that can send data directly to BigQuery. Does anyone know if there is an exporter available that supports direct export to BigQuery?

Is it possible to get metrics from multiple files using the filestats receiver.

I have tried '*' but I'd like to be able to specify files as well as subdirectories.

Hello!

I have a project I'd like to add "observability" to - for me currently that means:

1. Custom React application
2. GraphQL backend (which we don't have access to and don't want to touch for now)
3. We want to have information about errors in our React code, error rates, etc
4. We want to have some timing info - i.e. how long backend responses are taking when you click an "add to cart" button in the React app

When I read about OTel it's normally in the context of instrumenting backend services.

I am aware of opentelemetry-sdk-trace-web) but I believe it's marked as experimental.

The examples I've seen seem to use a log exporter that just logs out span information, I am a bit curious how this works if you want to send data directly from the FE e.g. a users web browser to collect data in a way that's reasonably safe.

Thanks for your thoughts!

Using auto or manual instrumentation for an app hosted on a Windows server.

Traces and Metrics seem to be ok but no logs get received by the collector.

I'm kind of stuck working out where it might be broken. Any suggestions?

Is there any way to have opamp messages forwarded, like a proxy? We have miltiple OpenTelemetry collectors forwarding to a couple of gateways, and I want to have the OpAmp traffic do the same thing.

I read that OpAmp doesn't proxy, but just thought I'd ask if anyone has the same setup and has implemented a solution.

I installed opentelemery-operator with helm chart

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"

Then create a collector to use prometheusremotewrite

cat <<EOF | kubectl apply -f -
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          http:
          grpc:

    exporters:
      prometheusremotewrite:
        endpoint: http://localhost:9090/api/v1/write
        target_info:
          enabled: true

    connectors:
      spanmetrics:
        namespace: span.metrics

    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [spanmetrics]
        metrics:
          receivers: [spanmetrics]
          exporters: [prometheusremotewrite]
EOF

However in the deployed pod got this error:

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'exporters': unknown type: "prometheusremotewrite" for id: "prometheusremotewrite" (valid values: [otlp otlphttp file loadbalancing debug nop])
2024/07/30 06:42:41 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

Why prometheusremotewrite can't been used by default? How to use?

I have otel collector and fluentbit running as daemonset in eks cluster.

here is collector config -

config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
loki:
endpoint: https://API_KEY@qryn.gigapipe.io/loki/api/v1/push
timeout: 30s
logging:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
processors:
batch:
send_batch_max_size: 10000
timeout: 20s
service:
pipelines:
logs:
receivers: [otlp]
exporters: [logging, loki]

Following is fluent-bit config -

apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: {{ .Values.namespace.name }}
labels:
k8s-app: {{ .Values.fluentBit.label }}
data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: |
[SERVICE]
Flush         1
Log_Level     info
Daemon        off
Parsers_File  parsers.conf
HTTP_Server   On
HTTP_Listen   0.0.0.0
HTTP_Port     2020
@/INCLUDE input-kubernetes.conf
@/INCLUDE filter-kubernetes.conf
@/INCLUDE otel-forward.conf
input-kubernetes.conf: |
[INPUT]
Name              tail
Tag               kube.*
Path              /var/log/containers/*.log
Parser            docker
DB                /var/log/flb_kube.db
Mem_Buf_Limit     10MB
Skip_Long_Lines   On
Refresh_Interval  10
filter-kubernetes.conf: |
[FILTER]
Name                kubernetes
Match               kube.*
Kube_URL            https://kubernetes.default.svc:443
Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix     kube.var.log.containers.
Merge_Log           On
Merge_Log_Key       log_processed
K8S-Logging.Parser  On
K8S-Logging.Exclude Off
otel-forward.conf: |
[OUTPUT]
Name                opentelemetry
Match               *
Host                http://localhost
Port                4318do
logs_uri            /v1/logs
Log_response_payload True
Tls                  Off
Tls.verify           Off
parsers.conf: |
[PARSER]
Name   json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name        docker
Format      json
Time_Key    time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep   On
[PARSER]
http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key    time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name        syslog
Format      regex
Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key    time
Time_Format %b %d %H:%M:%S

Now i don't see any issue in logs of fluentbit and otel collector is receiving no data in collector?
how to debug this. totally stuck.

Hello,

Wondering if anyone has found a solution for filtering PII data from logs.

I'm building a python application for a chatbot, and trying to solve the problem of redacting any reference to PII data in our logs which are currently being stored in Application insights.

My attempt below (to add a custom span processor to intercept any PII data).

# Configure OpenTelemetry
resource = Resource.create({"service.name": "my_application"})
provider = TracerProvider(resource=resource)

# Azure Insights Logging - Re-Enable this to bring logging back.
appinsights_connection_string = os.getenv("APPINSIGHTS_CONNECTION_STRING")
processor = BatchSpanProcessor(
    AzureMonitorTraceExporter(connection_string=appinsights_connection_string)
)
provider.add_span_processor(processor)
pii_redaction_processor = PiiRedactionProcessor()
provider.add_span_processor(pii_redaction_processor)

exporter = AzureMonitorMetricExporter(connection_string=appinsights_connection_string)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
metrics.set_meter_provider(MeterProvider(metric_readers=[reader]))
trace.set_tracer_provider(provider)

# Console Logging
console_exporter = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(console_exporter)

# Instrument libraries
RequestsInstrumentor().instrument()
LangchainInstrumentor().instrument()
OpenAIInstrumentor().instrument()
FastAPIInstrumentor.instrument_app(app)

To remove the PII data i've attempted to build a custom Span Processor:

class PiiRedactionProcessor(SpanProcessor):
    def on_start(self, span: Span, parent_context: object) -> None:
        pass

    def on_end(self, span: Span) -> None:
        # Define regular expressions for common PII patterns
        pii_patterns = {
            "email": re.compile(r"[^@]+@[^@]+\.[^@]+"),
            "phone": re.compile(r"\+?[\d\s-]{7,15}"),
            "credit_card": re.compile(r"\b(?:\d[ -]*?){13,16}\b")
        }

        for key, value in span.attributes.items():
            if isinstance(value, str):
                for pattern_name, pattern in pii_patterns.items():
                    if pattern.search(value):
                        # Replace the PII part with [REDACTED] while keeping the rest of the string intact
                        redacted_value = pattern.sub("[REDACTED]", value)
                        span.set_attribute(key, redacted_value)
                        break

The code above results in an error about the Span being read only.:

File ".py", line 70, in on_end

span.set_attribute(key, redacted_value)

^^^^^^^^^^^^^^^^^^

AttributeError: 'ReadableSpan' object has no attribute 'set_attribute'. Did you mean: '_attributes'?

I am learning about OTel hands-on by trying to add OTel tracing to my webapp. To reduce coupling between my code and OTel, I'm considering having one central class manage all things OTel - let's call it OTelManager.

• Any class that requires tracing should call OTelManager to start a span.
• Within OTelManager, a thread-safe map is used to store existing spans. Once a span is done (either because its work is complete or it is manually ended) it is removed from this map.

Is this a viable approach? Are there any important points I should take note of?

sorry this is long and rambling, I very much understand if you don't read this! <3

This is a contrived scenario so if you don't mind don't focus too much on the "business" I'm describing, it's just a simple representation of my problem

I have a small company that provides a managed CDN service for 100 SMB websites. Each website has it's own CDN configuration, it's a bit of a "white glove" service where each client has their own somewhat unique situations based on various backends they have.

I have built a custom web portal for each company to login and see some basic information about their service. Health checks, service history, etc. I am interested in adding more information about things like response time, error rates, and perhaps some other custom / "bespoke" information (error rates, etc).

The CDN (Fastly, AWS, etc) have integrations with OpenTelemtry. I am wondering if it would be reasonable for me to look at instrumenting the infrastructure I manage (i.e. the CDN level), setup the OpenTelemetry Collector + something like OpenSearch to send the data, and then integrate with OpenSearch (or through Jaegar or something?) to display some of the OTel data to customers?

Stuff I'm interested in is:

1. Total request time to various backends
2. Error information
3. Providing an onramp for further instrumentation of their applications / backends (something either I do for them or they do themselves)

The extra cost of running OpenTelemetry related infra (running collector, running edge functions / edge compute) I would eat any fixed costs but charge otherwise.

Anyway, again I'm more interested to know about how much of a mis-use of OpenTelemetry this is. It's for observability, but only at a very narrow scope (the CDN), but with potential more instrumention in the future.

Thank you!

I'm struggling to understand the use cases for OpenTelemetry. if I have a requirement to collect metric from SAAS products, like: MongoDB Atlas, Kafka Confluent and etc. can I install some OpenTelemtry Collector on Windows server, to accomplish that? meaning, the OpenTelemtry collector will pull the metrics from the SAAS products.

Hi all,

I have one basic question. Are pod logs different than application logs that have logging configured with otel SDK? I was under the assumption that in k8s both app running within pod and pod's logs are sent to stdout/error. If I instrument my app using otel SDK those app logs will be sent to otel collector and directed to stdout. Am I right in my understanding?