...
TLDR: Tracing has been added for A1 Policy Management Service. By default tracing is disabled. To enable it change the flag there are two ways:
A) System Property
Change the flag otel.sdk.disabled to false in the application.yaml (New Delhi)
Code Block | ||||
---|---|---|---|---|
| ||||
otel: sdk: disabled: false |
...
|
...
|
...
|
...
Code Block | ||||
---|---|---|---|---|
| ||||
disabled: ${ONAP_SDK_DISABLED=:false} south: ${ONAP_TRACING_SOUTHBOUND=true |
Tracing Test
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
:true}
instrumentation:
spring-webflux:
enabled: true |
- otel.sdk.disabled: enable/disable tracing all toghether
- otel.sdk.south: if ONAP_SDK_DISABLED is false then we can enable/disable southbound tracing
- otel.instrumentation.spring-webflux.enabled: if ONAP_SDK_DISABLED is false we can enable/disable northbound tracing
B) Enviroment Variable
Have the environment variables, this way you don't need to change the application.yaml and rebuild the docker imagea) A docker compose with a1pms, a1-osc-simulator, and jaeger that acts as a collector and exporter
Code Block | ||||
---|---|---|---|---|
| version: '3.7'
services:
a1_policy_management:
container_name: a1-pms
image: onap/ccsdk-oran-a1policymanagementservice:1.7.0-SNAPSHOT
ports:
- "8433:8433"
- "8081:8081"
volumes:
- ./application_configuration.json.nosdnc:/opt/app/policy-agent/data/application_configuration.json:ro
networks:
- jaeger-example
depends_on:
- jaeger
environment:
- ||||
ONAP_SDK_DISABLED=false
ONAP_TRACING_SOUTHBOUND=true
OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=true |
- ONAP_SDK_DISABLED: enable/disable tracing all toghether
- ONAP_TRACING_SOUTHBOUND: if ONAP_SDK_DISABLED is false then we can enable/disable southbound tracing
- OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED: if ONAP_SDK_DISABLED is false we can enable/disable northbound tracing
Possible Combinations
So we can have the following combinations:
Tracing | Northbound | Southbound | Flags |
---|---|---|---|
ONAP_SDK_DISABLED=true | |||
ONAP_SDK_DISABLED=false |
...
; ONAP_TRACING_SOUTHBOUND=true |
...
; OTEL_ |
...
INSTRUMENTATION_ |
...
SPRING_ |
...
WEBFLUX_ENABLED=true | |||
ONAP_SDK_DISABLED=false; ONAP_TRACING_SOUTHBOUND=false; OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=true | |||
ONAP_SDK_DISABLED=false; ONAP_TRACING_SOUTHBOUND=true; OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=false |
Tracing Test
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
a) A docker compose with a1pms, a1-osc-simulator, and jaeger that acts as a collector and exporter. Note: onap/ccsdk-oran-a1policymanagementservice:1.7.0-SNAPSHOT is built locally by doing "mvn clean install", you can use the nexus hosted image changing the prefix.
Code Block | ||||
---|---|---|---|---|
| ||||
version: '3.7' services: a1_policy_management: ENDPOINT=http://jaeger:14250 - ONAP_OTEL_EXPORTER_ENDPOINT=http://jaeger:4317 - ONAP_OTEL_EXPORTER_PROTOCOL=grpc - ONAP_OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc a1-sim-OSC: image: "nexus3.o-ran-sc.org:10002/o-ran-sc/a1-simulator:2.1.0" container_name: a1-sim-OSCpms ports:image: onap/ccsdk-oran-a1policymanagementservice:1.7.0-SNAPSHOT ports: - "300018433:80858433" - "300028081:81858081" environmentvolumes: - A1_VERSION=OSC_2.1.0./application_configuration.json.nosdnc:/opt/app/policy-agent/data/application_configuration.json:ro - REMOTE_HOSTS_LOGGING=1networks: - ALLOW_HTTP=truejaeger-example networksdepends_on: - jaeger-example jaeger: image: jaegertracing/all-in-one:latestenvironment: container_name: jaeger ports: - "16686:16686"- ONAP_SDK_DISABLED=false - "14250:14250"ONAP_TRACING_SOUTHBOUND=true - "14268:14268"OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=true - "4317:4317"ONAP_OTEL_SAMPLER_JAEGER_REMOTE_ENDPOINT=http://jaeger:14250 - "4318:4318" environment:ONAP_OTEL_EXPORTER_ENDPOINT=http://jaeger:4317 - JAEGER_DISABLED=true - LOG_LEVEL=debugONAP_OTEL_EXPORTER_PROTOCOL=grpc - COLLECTORONAP_OTEL_EXPORTER_OTLP_ENABLED=trueTRACES_PROTOCOL=grpc a1-sim-OSC: networks:image: "nexus3.o-ran-sc.org:10002/o-ran-sc/a1-simulator:2.1.0" container_name: a1- jaegersim-example networksOSC ports: jaeger -example "30001:8085" driver: bridge |
b) The application_configuration.json.nosdnc in the same folder
Code Block | ||||
---|---|---|---|---|
| ||||
{ "description":"Application configuration", - "30002:8185" "config"environment:{ "ric":[- A1_VERSION=OSC_2.1.0 {- REMOTE_HOSTS_LOGGING=1 - ALLOW_HTTP=true "name":"ric1",networks: - jaeger-example jaeger: "baseUrl":"https://a1-sim-OSC:8185/",image: jaegertracing/all-in-one:latest container_name: jaeger ports: - "managedElementIds16686:16686":[ - "14250:14250" - "kista_1","14268:14268" - "4317:4317" - "kista_24318:4318" environment: - JAEGER_DISABLED=true ] - LOG_LEVEL=debug - COLLECTOR_OTLP_ENABLED=true } networks: ]- jaeger-example networks: jaeger-example: } } driver: bridge |
...
...
b) The application_configuration.json.nosdnc in the same folderc) Creating a PolicyType in the simulator
Code Block | ||||
---|---|---|---|---|
| ||||
curl{ -v -X 'PUT' \ "description":"Application configuration", 'http://localhost:30001/a1-p/policytypes/1' \ "config":{ -H 'accept: application/json' \ "ric":[ -H 'Content-Type: application/json' \ -d '{ "name":"pt1", "description":"pt1 policy type", "policy_type_idname":1"ric1", "create_schema":{ "$schemabaseUrl":"httphttps://json-schema.org/draft-07/schema#a1-sim-OSC:8185/", "titlemanagedElementIds":"OSC_Type1_1.0.0",[ "description":"Type 1 policy type", "type":"object""kista_1", "properties":{ "scopekista_2":{ "type":"object",] } "properties":{ ] } } |
...
c) Creating a PolicyType in the simulator
Code Block | ||||
---|---|---|---|---|
| ||||
curl -v -X 'PUT' \ "ueId":{'http://localhost:30001/a1-p/policytypes/1' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "typename":"stringpt1", "description":"pt1 policy type", }"policy_type_id":1, "qosId"create_schema":{ "type"$schema":"string"http://json-schema.org/draft-07/schema#", "title":"OSC_Type1_1.0.0", "description":"Type 1 }policy type", "type":"object", }, "properties":{ "additionalPropertiesscope":false,{ "requiredtype":["object", "ueId","properties":{ "qosIdueId":{ ] }, "qosObjectives":{"type":"string" "type":"object", "properties":{}, "priorityLevelqosId":{ "type":"numberstring" } }, "additionalProperties":false, "required":[ "priorityLevelueId", ] "qosId" } ] }, "additionalPropertiesqosObjectives":false,{ "required "type":["object", "scopeproperties", :{ "qosObjectivespriorityLevel":{ ] "type":"number" } }' |
d) Creating a policy in a1-pms, after the policy type is successfully registered (curl http://localhost:8081/a1-policy/v2/policy-types)
Code Block | ||||
---|---|---|---|---|
| ||||
curl -v -X 'PUT' \ 'http://localhost:8081/a1-policy/v2/policies' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "ric_id": "ric1", "policy_id": "aa8feaa88d944d919ef0e83f2172a51002", "transient": false, "service_id": "controlpanel", "policy_data": { } }, "additionalProperties":false, "required":[ "scopepriorityLevel": { "ueId": "ue5100", ] "qosId": "qos5100"} }, "qosObjectivesadditionalProperties": { false, "priorityLevelrequired": 5100.0 [ }"scope", }, "status_notification_uri": "http://callback-receiver:8090/callbacks/test", "policytype_id": "1" "qosObjectives" ] } }' |
...
e) d) Creating a policy in a1-pms, after the policy type is successfully registered (curl http://localhost:166868081/ Load Jaeger UI, a1-pms traces, and a sample of the last call would be:
...
Opentelemetry documentation provides a bean to mutate the default WebClient builder and to add tracing filters.
In our case the AsyncRestClient manually builds a WebClient for every asynchronous request.
The challenge was to add the tracing filters to this non-Spring class.
1.Adding OpenTelemetry Bean
...
Code Block | ||||
---|---|---|---|---|
| ||||
curl -v -X 'PUT' \ 'http://localhost:8081/a1-policy/v2/policies' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "ric_id": "ric1", "policy_id": "aa8feaa88d944d919ef0e83f2172a51002", "transient": false, "service_id": "controlpanel", "policy_data": { |
...
Introduced circular dependency openTelemetryConfig defined in URL [jar:file:/opt/app/policy-agent/a1-policy-management-service.jar!/BOOT-INF/classes!/org/onap/ccsdk/oran/a1policymanagementservice/configuration/OpenTelemetryConfig.class
2. Adding filters into AsyncRestClient directly and not into builder bean, but the AutoConfiguredOpenTelemetrySdk uses by default parameters such as localhost:4317 to export grpc, so we opted for using the application.yaml parameters to build the exporters beans.
...
"scope": { "ueId": "ue5100", "qosId": "qos5100" }, "qosObjectives": { |
...
"priorityLevel": 5100.0 |
...
} }, |
...
"status_notification_uri": "http://callback-receiver:8090/callbacks/test",
"policytype_id": "1"
}' |
...
e) http://localhost:16686/ Load Jaeger UI, a1-pms traces, and a sample of the last call would be:
Steps Taken and Challenges:
Adding Telemetry to a1policymanagementservice: The application uses the WebClient from SpringWebflux to contact from the northbound interface a southbound interface (for the latter a A1-OSC simulator has been used).
Opentelemetry documentation provides a bean to mutate the default WebClient builder and to add tracing filters.
In our case the AsyncRestClient manually builds a WebClient for every asynchronous request.
The challenge was to add the tracing filters to this non-Spring class.
1.Adding OpenTelemetry Bean
@Bean
public OpenTelemetry openTelemetry() {
return AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();
}
Introduced circular dependency openTelemetryConfig defined in URL [jar:file:/opt/app/policy-agent/a1-policy-management-service.jar!/BOOT-INF/classes!/org/onap/ccsdk/oran/a1policymanagementservice/configuration/OpenTelemetryConfig.class
2. Adding filters into AsyncRestClient directly and not into builder bean, but the AutoConfiguredOpenTelemetrySdk uses by default parameters such as localhost:4317 to export grpc, so we opted for using the application.yaml parameters to build the exporters beans.
AsyncRestClient.java
...
OpenTelemetry openTelemetry = AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();
var webfluxTelemetry = SpringWebfluxTelemetry.builder(openTelemetry).build();
return WebClient.builder() //
...
.filters(webfluxTelemetry::addClientTracingFilter)
.build();
3. Context Provider class to use get the ApplicationContext into Non-Spring Components
import org.springframework.
...
3. Context Provider class to use get the ApplicationContext into Non-Spring Components
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.stereotype.Component;
@Component
public class ApplicationContextProvider implements ApplicationContextAware {
private static ApplicationContext context;
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
context = applicationContext;
}
public static ApplicationContext getApplicationContext() {
return context;
}
}
...
Code Block | ||||
---|---|---|---|---|
| ||||
2024-06-16 18:55:19 2024-06-16 17:55:19.060 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JBaggageEventListener - Got scope attached event [ScopeAttached{context: [span: null] [baggage: null]}] 2024-06-16 18:55:19 2024-06-16 17:55:19.060 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JEventListener - Got scope changed event [ScopeAttached{context: [span: null] [baggage: null]}] 2024-06-16 18:55:19 2024-06-16 17:55:19.060 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JBaggageEventListener - Got scope closed event [io.micrometer.tracing.otel.bridge.EventPublishingContextWrapper$ScopeClosedEvent@56db4345] 2024-06-16 18:55:19 2024-06-16 17:55:19.060 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JEventListener - Got scope closed event [io.micrometer.tracing.otel.bridge.EventPublishingContextWrapper$ScopeClosedEvent@56db4345] 2024-06-16 18:55:19 2024-06-16 17:55:19.061 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JBaggageEventListener - Got scope restored event [ScopeRestored{context: [span: null] [baggage: null]}] 2024-06-16 18:55:19 2024-06-16 17:55:19.061 [TRACE] [BatchSpanProcessor_WorkerThread-1] i.m.t.o.b.Slf4JEventListener - Got scope restored event [ScopeRestored{context: [span: null] [baggage: null]}] 2024-06-16 18:55:19 2024-06-16 17:55:19.062 [ERROR] [OkHttp http://localhost:4318/...] i.o.e.i.h.HttpExporter - Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4318 2024-06-16 18:55:19 java.net.ConnectException: Failed to connect to localhost/127.0.0.1:4318 2024-06-16 18:55:19 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297) 2024-06-16 18:55:19 at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) 2024-06-16 18:55:19 at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) 2024-06-16 18:55:19 at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) 2024-06-16 18:55:19 at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) 2024-06-16 18:55:19 at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) 2024-06-16 18:55:19 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) 2024-06-16 18:55:19 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2024-06-16 18:55:19 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) 2024-06-16 18:55:19 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2024-06-16 18:55:19 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) 2024-06-16 18:55:19 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2024-06-16 18:55:19 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) 2024-06-16 18:55:19 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2024-06-16 18:55:19 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) 2024-06-16 18:55:19 at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) 2024-06-16 18:55:19 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2024-06-16 18:55:19 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2024-06-16 18:55:19 at java.base/java.lang.Thread.run(Thread.java:833) 2024-06-16 18:55:19 Caused by: java.net.ConnectException: Connection refused 2024-06-16 18:55:19 at java.base/sun.nio.ch.Net.pollConnect(Native Method) 2024-06-16 18:55:19 at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) 2024-06-16 18:55:19 at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) 2024-06-16 18:55:19 at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) 2024-06-16 18:55:19 at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327java:327) 2024-06-16 18:55:19 at java.base/java.net.Socket.connect(Socket.java:633) 2024-06-16 18:55:19 at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) 2024-06-16 18:55:19 at javaokhttp3.base/javainternal.netconnection.SocketRealConnection.connectconnectSocket(SocketRealConnection.javakt:633295) 2024-06-16 18:55:19 at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) 2024-06-16 18:55:19 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) 2024-06-16 18:55:19 ... 18 common frames omitted 2024-06-16 18:55:19 2024-06-16 17:55:19.066 [DEBUG] [BatchSpanProcessor_WorkerThread-1] i.o.s.t.e.BatchSpanProcessor - Exporter failed. 18 common frames omitted 2024-06-16 18:55:19 2024-06-16 17:55:19.066 [DEBUG] [BatchSpanProcessor_WorkerThread-1] i.o.s.t.e.BatchSpanProcessor - Exporter failed |
4. Flags to enable/disable northbound or southbound interfaces
Since we used Java Springboot starter library from OpenTelemetry we can use their flags to enable or disableinstrumentation libraries.
https://opentelemetry.io/docs/zero-code/java/agent/configuration/#suppressing-specific-agent-instrumentation
OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED=true from the documentation we can use this flag to disable the automatic spring instrumentation and we keep a separate manual flag ONAP_TRACING_SOUTHBOUND for the AsyncRestClient requests made to the southbound.
System property: otel.instrumentation.[name].enabled
Environment variable: OTEL_INSTRUMENTATION_[NAME]_ENABLED
Note: When using OPENTELEMETRY (Evrything starting with otel) environment variables, dashes (-) should be converted to underscores (_). For example, to suppress traces from spring-webflux library, set OTEL_INSTRUMENTATION_SPRING_WEBFLUX_ENABLED to false
Full Tracing:
Only Southbound Tracing Output:
Only Northbound Tracing Output: