...
ONAP Log Streaming Deployment View
Gliffy Diagram |
---|
name | ONAP Log Streaming Architecture |
---|
pagePin | 3 |
---|
|
Analysis
Existing ONAP Logging State - Infrastructure and Schema
Image Added
Analysis
using
http://openlab13.onap.info:30253/app/kibana
Compliance
Streaming
https://12factor.net/logs
Format
Library
Fields
Code Block |
---|
|
timestampadd
t versionadd
t ACTIONadd
t API_KEYadd
t ARTIFACT_DATAadd
t AUTH_STATUSadd
t AlertSeverityadd
AuditBeginTimestampadd
t AuditMessageadd
BeginTimestampadd
t CONSUMER_IDadd
t CURR_ARTIFACT_UUIDadd
t CURR_STATEadd
t CURR_VERSIONadd
t ClassNameadd
t CustomField1add
t CustomField2add
t DESCadd
t DNOTIF_TOPICadd
t DSTATUS_TOPICadd
t D_ENVadd
t ECOMP_USERadd
t ElapsedTimeadd
EndTimestampadd
t ErrorCategoryadd
t ErrorCodeadd
t INVARIANT_UUIDadd
t InstanceUUIDadd
t MODIFIERadd
t PREV_ARTIFACT_UUIDadd
t PREV_STATEadd
t PREV_VERSIONadd
t PartnerNameadd
t REALMadd
t RESOURCE_NAMEadd
t RESOURCE_TYPEadd
t RemoteHostadd
t RequestIdadd
t ResponseCodeadd
t ResponseDescriptionadd
t SERVICE_INSTANCE_IDadd
t STATUSadd
t ServerFQDNadd
t ServerIPAddressadd
t ServiceInstanceIdadd
t ServiceNameadd
t StatusCodeadd
t TargetEntityadd
t TargetServiceNameadd
t Threadadd
t Timeradd
Timestampadd
t URLadd
t USERadd
t USER_AFTERadd
t USER_BEFOREadd
t _idadd
t _indexadd
# _scoreadd
t _typeadd
t auditOnadd
t beat.hostnameadd
t beat.nameadd
t beat.versionadd
t componentLogFileadd
t componentNameadd
t hostadd
t input_typeadd
t logleveladd
t messageadd
# offsetadd
t serviceInstanceIDadd
t sourceadd
t tagsadd
t timeradd
t typeadd |
Existing ONAP Logging State - Infrastructure and Schema
Deployment Dependency Tree
Note: these dependencies - introduced using the readiness container by Mandeep Khinda are at the lowest deployment level and represent a partial view of the REST/API dependency tree - they do not reflect any compile time or runtime/injection code dependencies (pom.xml)
Gliffy Diagram |
---|
name | Kubernetes Deployment Dependencies |
---|
pagePin | 3 |
---|
|
Jira |
---|
server | ONAP JIRA |
---|
serverId | 425b2b0a-557c-3c0c-b515-579789cceedb |
---|
key | LOG-924 |
---|
|
tested sdnc via sudo ./cd.sh -b master -e onap -p false -a false -t robot -x 1 -z false -f false -s 600 -c false -d true -w false -r false
where dmaap, consul and sdc are raised first via dev0.yaml and sdnc is in dev1.yaml
Deployment Dependency Tree - Container level
Gliffy Diagram |
---|
name | Deployment Dependencies - containers |
---|
pagePin | 3 |
---|
|
Code Block |
---|
|
Use for any containers stuck at the 0/1 Init:0/1 stage - these are likely waiting on dependent containers
check the --container-name kv pair in StatefulSet/Deployment.yaml:spec:intiContainers:args
or the corresponding defined chart/container names in values.yaml:config:
106 sets in 87 files
overall order
aaf<-aai
aaf<-oof
music<-oof
dmaap<-aai
dmaap<-pomba
dmaap<-sdnc
consul<-sdnc
sdc<-sdnc
consul<-dcaegen2
msb<-dcaegen2
order
dmaap, aaf, msb, consul, dcaegen2, aai, oof, sdc, sdnc, pomba
valid sets of pods
aaf
aai, dmaap, aaf
dcaegen2, consul, msb
oof, aaf, music
pomba, dmaap
sdnc, consul, sdc
..the rest can be singular
aaf
aaf-cm
aaf-locate
aaf-fs
aaf-locate
aaf-gui
aaf-cm
aaf-hello
aaf-locate
aaf-locate
aaf-service
aaf-oauth
aaf-locate
aaf-service
aaf-cs
aaf-sms
aaf-sms-quorumclient (via job)
aaf-sms-vault
aaf-sms-vault-backend
aai
aai
aai-resources
aai-traversal
aai-graphadmin
aai-champ
aai-cassandra
aai-graphadmin
aai-cassandra
aai-resources
aai-cassandra
aai-sparky-be
aai-elasticsearch
aai-search-data
aai
aai-spike
message-router-kafka
aai-traversal
aai
aai-cassandra
aaf-locate (conditional)
appc
appc
mariadb-galera
appc-ansible-server
appc
clamp
clamp
mariadb
clamp-dash-kibana
clamp-dash-es
clamp-dash-logstash
clamp-dash-es
common
controller-blueprints
mariadb-galera
mongo
*-nfs-provisioner
mysql
*-nfs-provisioner
dgbuilder
network-name-gen
mariadb-galera
dcaegen2
dcae-bootstrap
dcae-cloudify-manager
consul-server
msb-discovery
kube2msb
dep-health-init
hbase
dmaap
dmaap-bus-controller
postgres
dmaap-dr-node
dmaap-dr-prov
dmaap-dr-prov
mariadb
message-router
kafka
zookeeper
message-router-kafka
zookeeper
log
log-kibana
log-elasticsearch
log-logstash
log-elasticsearch
msb
kube2msb
msb-discovery
msb-discovery
msb-consul
msb-eag
msb-discovery
msb-iag
msb-discovery
music
music-cassandra
music-tomcat
zookeeper
oof
oof-has-api
oof-has-controller
aaf-service
oof-has-controller
music-tomcat
aaf-sms
oof-has-data
music-tomcat
oof-has-reservation
music-tomcat
oof-has-service
music-tomcat
policy
policy
mariadb
brmsgw
pap
drools
mariadb
nexus
pdb
pap
pomba
pomba-contextaggregator
message-router
pomba-kibana
pomba-elasticsearch
pomba-data-router
pomba-search-data
pomba-search-data
pomba-elasticsearch
portal
portal-widget
portal-db
portal-sdk
portal-db
sdc
sdc-dcae-be
common.name
sdc-be
sdc-dcae-dt
sdc-dcae-be
sdc-dcae-fe
sdc-dcae-be
sdc-dcae-tosca-lab
sdc-dcae-be
sdc-fe
sdc-kb
sdc-wfd-fe
sdc-wfd-be
sdnc
sdnc
mysql
sdnc-ansible-server
sdnc
dmaap-listener
mysql
sdnc
message-router
sdnc-portal
mysql / sdnc-db
sdnc
sdnc-prom
sdnc
consul
ueb-listener
mysql
sdnc
sdc-be
message-router
so
so
mariadb
so-bpmn-infra
mariadb
so-catalog-db-adapter
mariadb
so-openstack-adapter
mariadb
so-request-db-adapter
mariadb
so-sdc-adapter
mariadb
so-sdc-controller
mariadb
so-vfc-adapter
mariadb
vfc
vfc-catalog
vfc-db
vfc-ems-driver
mariadb // commented
vfc-generic-vnfm-driver
mariadb // commented
vfc-hauwei-vnfm-driver
mariadb // commented
vfc-juju-vnfm-driver
mariadb // commented
vfc-multivim-proxy
mariadb // commented
vfc-nokia-vnfm-driver
mariadb // commented
vfc-nokia-v2vnfm-driver
mariadb // commented
vfc-nslcm
vfc-db
vfc-vnfmgr
vfc-db
vfc-resmgr
mariadb // commented
vfc-workflow
mariadb // commented
vfc-workflow-engine
mariadb // commented
vfc-vnflcm
vfc-db
vfc-vnfres
vfc-db
vfc-zte-sdnc-driver
mariadb // commented
vfc-zte-vnfm-driver
mariadb // commented
vid
vid
mariadb-galera
vnfsdk
postgres |
Deployment Profiles
For use case specific deployments
Code Block |
---|
|
# deployment profiles to cd.sh
# each of the 28 components in alpha sequence --set pod.enabled=true|false status
aaaccccd delmmnop pprssssu vvvv
11000100 10010101 01111011 0100 - vFW - no monitoring
11000100 10110101 11111011 0100 - vFW - with monitoring
11110111 10110111 11111111 0100 - vFW - cl |
# | Team | pod | container | Required y/n/fw/fwCl | Ram | Cpu | nodeport | Type | logback | Dependencies |
---|
| aaf |
|
|
|
|
|
|
|
|
|
| aai |
|
| y |
|
|
|
|
|
|
| appc |
|
| fwCL |
|
|
|
|
|
|
| clamp |
|
| fwCL |
|
|
|
|
|
|
| cli |
|
|
|
|
|
|
|
|
|
| consul |
|
|
|
|
|
|
|
|
|
|
|
|
| fwCL |
|
|
|
|
|
|
| dmaap |
|
| y |
|
|
|
|
|
|
| esr |
|
|
|
|
|
|
|
|
|
| log |
|
| n |
|
|
|
|
|
|
| sniro- emulator |
|
| n |
|
|
|
|
|
|
| oof |
|
| n |
|
|
|
|
|
|
| msb |
|
| n |
|
|
|
|
|
|
| multicloud |
|
| n |
|
|
|
|
|
|
| nbi |
|
|
|
|
|
|
|
|
|
| policy |
|
| y |
|
|
|
|
|
|
| pomba |
|
|
|
|
|
|
|
|
|
| portal |
|
| y |
|
|
|
|
|
|
| robot |
|
| y |
|
|
|
|
|
|
| sdc |
|
| y |
|
|
|
|
|
|
| sdnc |
|
| y |
|
|
|
|
|
|
| so |
|
| y |
|
|
|
|
|
|
| uui |
|
| n |
|
|
|
|
|
|
| vfc |
|
| n |
|
|
|
|
|
|
| vid |
|
| y |
|
|
|
|
|
|
| vnfsdk |
|
| n |
|
|
|
|
|
|
Pod | Container/service | Tech | URLs | Filebeat Streaming | logback.xml | library |
| dependencies | Log level adjust |
---|
aaf |
Code Block |
---|
| aaf-cm
aaf-fs
aaf-gui
aaf-hello
aaf-locate
aaf-oauth
aaf-service
aaf-sms-vault-0
|
|
|
|
|
|
|
|
|
|
aai |
Code Block |
---|
| aai-babel
aai-champ
aai-data-router
aai-gizmo
aai-graphadmin
aai-modelloader
aai-resources
aai-search-data
aai-sparky-be
aai-spike
aai-traversal
|
|
|
|
|
|
|
|
|
|
appc | |
|
|
|
|
|
|
|
|
clamp | |
|
|
|
|
|
|
|
|
cli | |
|
|
|
|
|
|
|
|
consul | |
|
|
|
|
|
|
|
|
contrib-netbox | |
|
|
|
|
|
|
|
|
dcaegen2 | |
|
|
|
|
|
|
|
|
dmaap | |
|
|
|
|
|
|
|
|
esr | |
|
|
|
|
|
|
|
|
log | |
|
|
|
|
|
|
|
|
logdemonode | |
|
|
|
|
|
|
|
|
msb | |
|
|
|
|
|
|
|
|
multicloud | |
|
|
|
|
|
|
|
|
nbi | |
|
|
|
|
|
|
|
|
oof | |
|
|
|
|
|
|
|
|
policy | |
|
|
|
|
|
|
|
|
pomba | |
|
|
|
|
|
|
|
|
portal | |
|
|
|
|
|
|
|
|
robot | |
|
|
|
|
|
|
|
|
sdc | |
|
|
|
|
|
|
|
|
sdnc | |
|
|
|
|
|
|
|
|
sniro | |
|
|
|
|
|
|
|
|
so | |
|
|
|
|
|
|
|
|
uui | |
|
|
|
|
|
|
|
|
vfc | |
|
|
|
|
|
|
|
|
vid | |
|
|
|
|
|
|
|
|
vnfsdk | |
|
|
|
|
|
|
|
|
vvp | |
Pod | Container/service | Tech | URLs | logback | library | dependencies |
---|
aaf | Code Block |
---|
| aaf-cm
aaf-fs
aaf-gui
aaf-hello
aaf-locate
aaf-oauth
aaf-service
aaf-sms-vault-0
|
| aai | Code Block |
---|
| aai-babel
aai-champ
aai-data-router
aai-gizmo
aai-graphadmin
aai-modelloader
aai-resources
aai-search-data
aai-sparky-be
aai-spike
aai-traversal
|
| appc | Code Block |
---|
|
appc-0
Deployment Profile
Type | VMs | Total RAM vCores HD | VM Flavor | K8S/Rancher Idle RAM | Deployed Total RAM | Deployed ONAP RAM | Pods | Max vCores | Idle vCores | HD/VM | HD NFS only | IOPS | Date | Notes deployment post 75min |
---|
Full Cluster (14 + 1) - recommended | 15 | 224G 112 vC 100G/VM | 16G, 8 vCores C5.2xLarge |
| 187Gb | 102Gb | 248 total 241 onap 217 up 0 error 24 config |
| 18 | 6+G master 14 to 50 +G slave(s) | 8.1G |
| 20181106 |
|
Single VM (possible - not recommended) | 1 | 432G 64 vC 120G | 256G+ 32+ vCores | Rancher: 13G Kubernetes: 8G Top: 10G | 151Gb | 141Gb | 240 total 233 onap 200 up 6 error 38 config | 55 | 22 | 108 of 128G | n/a | Max: 550/sec Idle: 220/sec | 20181105 | Tested on 432G/64vCore azure VM - R 1.6.22 K8S 1.11 |
Developer 1-n pods | 1 | 16G 4 vC 100G | 16/32G 4-16 vCores |
| 14Gb | 10Gb |
|
|
| 120+G | n/a |
|
| AAI+robot only |
Example 15 node (1 master + 14 nodes) OOM Deployment
...
Library Selected for Reference Implementation
Java
Python
Specification
Developed in Casablanca - implemented in Dublin - ONAP Application Logging Specification v1.2 (Casablanca)
Use Cases
Design Issues
Developer Guide
...