Page History

...

The problem of decentralized logger output is addressed by analytics pipelines such as Elastic Stack (ELK). Elastic Stack consumes logs, indexes their contents in Elasticsearch, and makes them accessible, queryable and navigable via a sophisticated UI, Kibana Discover. This elevates the importance of standardization and machine-readability. Logfiles can remain browseablebrowsable, but output can be simplified.

Logger configurations in ONAP are diverse and idiosyncratic. Addressing these issues will prevent costs from being externalized to consumers such as analytics. It also affords the opportunity to remedy any issues with the handling and propagation of contextual information such as transaction identifiers (presently passed as X-ECOMP-RequestID - to be X-ONAP-RequestID). This propagation is critical to tracing requests as they traverse ONAP and related systems, and is the basis for many analytics functions. 

...

1. ONAP-wide changes to regularize logger providers, logger configuration and logger output. The changes are largely cosmetic, but greater coherence will simplify deployment orchestration and customization, and improve extensibility and support for analytics. 
2. A reference analytics pipeline configuration consisting of:
   • Filebeat shipping
   • Logstash indexing
   • Elasticsearch datastore
   • Kibana Discover UI
3. Documentation:
   • Configuration
   • Operations
   • Updates to ONAP application logging guidelines.
4. Other example configurations, including: 
   • JSON output. 
   • Non-file transports, including SYSLOG and TCP.  
   • Durable shipping transports, such as Logstash persistent queues, or Kafka or Redis or similar if there's interest.
   • Kibana Dashboards.

...

Standardization

All changes augment ONAP application logging guidelines on the ONAP wiki. 

Default logfile locations

...

• Currently: varies (though a default is loosely prescribed in ONAP application logging guidelines).
• Affects: most ONAP components, superficially. 
• Priority: HIGH.
• Why: 
  • Reduced complexity. 
  • Fewer special cases:
    • Easier configuration.
    • Easier indexing.
    • Easier orchestration.
• Notes:
  • These paths may be on a mounted persistent volume so that:
    • Logs are accessible to shippers such as Filebeat or Fluentd.
    • Logs are persistent, and independent of the lifecycle of individual containers.
  • Any indirection for multiple instances of a container writing to a shared volume can be managed in one of two ways:
    • By including a container identifier in the above spec. 
    • By deployment automation; in volume mappings, so that tenant applications don't need to know. 

...

• Currently: varies (no default prescribed by ONAP application logging guidelines).
• Affects: most ONAP components, superficially. 
• Priority: HIGH.
• Why:
  
  • Reduced complexity. 
  • Fewer special cases, simplified automation. 
  • Certain components are presently not configurable, e.g. Portal, for which even trivial reconfiguration of logging requires reassembly of its WAR.
  • See Support for analytics pipelines. 
• Notes:
  • This greatly simplifies deployment automation, reconfiguration, etc. 
  • These paths will normally be local to a container, not on a mounted volume, but either is fine.
  • The exact path can be fine-tuned, but:
    • The benefit of an ONAP-wide convention should be evident.
    • This corresponds to the convention implemented (updated for ONAP) by MSO and others.
    • It should be aligned with conventions for other component- and subcomponent-specific configuration documents. 

...

On the basis of regularized provider configuration and output, and the work of the ONAP Operation Operations Manager project, a complete Elastic Stack pipeline can be deployed automatically. 

...

• Filebeat shipping:
  • Line format and locations as discussed. 
  • Other shipping transports are simplified by regularized logger provider configuration. 
• Logstash indexing.
• Elasticsearch in a minimal three-node cluster. 
• Kibana Discover UI.
• Fullly automated Kubernetes deployment based on ONAP Operation Operations Manager.

• Currently: N/A
• Affects: 
  • Adds analytics capability, but no impact on individual ONAP components. 
  • Reconfiguration, including replacement with OTHER analytics pipelines will be similarly transparent.
• Priority: HIGH.
• Why: 
  • Demonstrates a bolt-on analytics capability.
  • Provides a working analytics OOB.
  • Elastic Stack is available as FOSS. 
• Notes:
  
  • Files + Filebeat are the lowest common denominator. They provide a simple mechanism for resilience, and yet require no disruptive (for example classpath) changes. 
  • Other sources of logs can be indexed, with appropriate configuration. Generally these will not be able to be correlated with an ONAP transaction, however. 

...

• Elastic Stack - https://www.elastic.co/products
• Filebeat - https://www.elastic.co/products/beats/filebeat
• Logstash - https://www.elastic.co/products/logstash
• Kibana Discover - https://www.elastic.co/guide/en/kibana/current/discover.html

ONAP Operation Operations Manager: https://wiki.onap.org/display/DW/ONAP+Operation+Manager+%28OOM%29+DescriptionONAP Operations Manager Project