Project Name:
- Proposed name for the project:
Holmes - Proposed name for the repository: h
olmes
Project description:
- Holmes project provides alarm correlation and analysis for Telecom cloud infrastructure and services, including hosts, vims, VNFs and NSs. Holmes aims to find the real reason which causes the failure or degradation of services by digging into the ocean of events collected from different levels of the Telecom cloud.
- Holmes provides docker(s) based fault correlation analysis system with APIs which could be called by external systems.
DCAE supports Holmes to be deployed as an analytic application in the form of docker(s). Actual deployment options are flexible, which should be decided by the user/use cases, Holmes can be either deployed as a standalone alarm correlation application or be integrated into DCAE as an analytic application.
Differences between Policy and Holmes
- The business scope of Holmes is different from that of Policy
Both Holmes and Policy adopt Drools as the rules engine. The main difference between these two projects is that Holmes is mainly targeted at correlation analysis between different alarms while Policy is aimed to implement control loops by triggering a series of actions. Briefly speaking, Holmes is targeted at root cause analysis but policy is aimed for auto-healing/auto-scaling.
- Holmes is necessary for reducing the pressure caused by the large alarm quantity for Policy
Policy does not need to face the original alarms directly with the help of Holmes. The root cause is picked out from all the original alarms by Holmes and then, the most suitable policy ID is selected and published accordingly. In this way, Policy is liberated from triggering similar or duplicated actions which are caused by the alarms with internal relations.
For example, if there are 3 events A, B and C which could lead to a power down fault, and B and C are caused by A. Without Holmes, all of these 3 events will be sent to Policy and 3 corresponding actions are going to be triggered. After we add Holmes to the close loop controller and make it the upstream system of Policy, only Event A will be sent to Policy and thus only one action will be triggered, which makes the close loop control more precise and efficient.
Scope:
Alarm Correlation Rule Management
- Holmes provides basic rule management functionalities which allow users to design, create or modify rules via a rule designer.
Collect Alarms from Different Alarm Sources
- Holmes supports different kinds of alarm sources, including NFV, SDN and any other legacy systems (as long as the corresponding interfaces of the source system are exposed).
- Alarm Analysis
- Holmes can pick out the root cause from the ocean of alarms with the assistance of the topology information provided by other related systems.
- Persistence of the Results of Data Analyses
- All analytic results are written to DB for persistence.
- Holmes provides the functionality for users to view the statistical result of data analysis.
- Publish the Analytic Results to Subscribers
- Besides result persistence, Holmes publishes the analytic results to a specific topic. Any potential users can subscribe to the topic to get the results in real time.
Architecture Alignment:
Holmes is an application that processes events published by managed resources or other applications that detect specific conditions. Based on defined root cause analysis rules about the network, an application of this kind would determine root cause for various conditions and notify other interested applications.
Holmes is designed in compliance with the VES standard. It can take event data from DMaaP, consume them and then send the correlation result back to DMaaP in the form of VES structure. Any other projects subscribe to the corresponding topic could fetch and use the result.
Holmes consists of a rule designer and a correlation engine.
Real-time Analytic application get the analysis result, which is the root cause can be used to drive the policy run-time for automation operation.
Correlation Engine receives original alarms from monitor and outputs the result back to monitor after analysis based on rules and the resources relationships from A&AI.
The Rule Designer provides a user-friendly GUI to design the Correlation rules. The GUI can be either run in standalone mode or integrated into other projects if needed.
Resources:
| Role | Name | Gerrit ID | Company | Email | TimeZone |
|---|---|---|---|---|---|
| Primary Contact | Guangrong Fu | ZTE | Beijing, China. UTC +8 | ||
| Commiters | Guangrong Fu | ||||
| Peng Tang | ZTE | tang.peng5@zte.com.cn | Beijing, China. UTC +8 | ||
| Contributors | Jiaqiang Du | ZTE | du.jiaqiang@zte.com.cn | Beijing, China. UTC +8 | |
| Yi Li | ZTE | li.yi101@zte.com.cn | Beijing, China. UTC +8 | ||
| Youbo Wu | ZTE | Beijing, China. UTC +8 | |||
| Liang Feng | ZTE | feng.liang1@zte.com.cn | Beijing, China. UTC +8 | ||
| Yuan Liu | China Mobile | liuyuanyjy@chinamobile.com | Beijing, China. UTC +8 | ||
| Chengli Wang | China Moblile | wangchengli@chinamobile.com | Beijing, China. UTC +8 | ||
| Xin(Saw) Jin | Huawei | saw.jin@huawei.com | Beijing, China. UTC +8 |
Other Information:
- link to seed code (if applicable)
git clone https://gerrit.open-o.org/r/holmes-actions
git clone https://gerrit.open-o.org/r/holmes-engine-management
git clone https://gerrit.open-o.org/r/holmes-gui
git clone https://gerrit.open-o.org/r/holmes-rule-management - Vendor Neutral
- if the proposal is coming from an existing proprietary codebase, have you ensured that all proprietary trademarks, logos, product names, etc., have been removed?
- Meets Board policy (including IPR)
TSC Comment Clarification
(Roberto Kung)
Holmes should be looked with Clamp or/and Policy, mainly policy (with introduction of engines and so on). May be a split is needed (analytics – alarm aggregation, filtering, correlation in DCAE analytics microservices / policy design RCA in policy). May not be high priority for R1 (not needed for our use cases). But it is useful to show intents for following releases
(Lingli Deng)
Just to clarify, cross-layer fault correlation is in scope for VoLTE usecase for auto-healing.
(Mazin Gilbert)
This project should be split and combine with DCAE (for the correlation engine), Policy engine (for Drools), and CLAMP (for designing the open loop).
(Lingli Deng)
What about the portal demonstrating the alarms gathered, and correleation made? Would DCAE be providing a portal for that?
(Unknown)
What’s the relationship between CLAMP and Holmes?
(Guangrong Fu)
Holmes is essential for control loops so it should be somewhat provisioned by CLAMP. For instance, if possible, rules of Holmes can be deployed/un-deployed via CLAMP. But how to implement this is still a mystery because so far we haven't got any seed code or API docs about CLAMP, which prevents us from further analysis.
Key Project Facts
Project Name:
- JIRA project name: holmes
- JIRA project prefix: holmes
Repo name: holmes
Lifecycle State:
Primary Contact: Guangrong Fu (fu.guangrong@zte.com.cn)
Project Lead: Guangrong Fu (fu.guangrong@zte.com.cn)
mailing list tag [Should match Jira Project Prefix]
Committers:
Please refer to the table above.
*Link to TSC approval:
Link to approval of additional submitters:
46 Comments
Pamela Dragosh
It seems like there is overlap here also with CLAMP and Policy. When it comes to designing control loops, CLAMP is the place where that work is done. CLAMP integrates with the Policy Platform for building configuration policies (i.e. Rules Designer) for DCAE components. Holmes seems to fit within DCAE and could be integrated within there.
Guangrong Fu
Hi Pamela,
Actually, the rule designer is an optional module for Holmes. If there's any overlapping, the GUI module can be integrated into DCAE.
Mary-Elise Haug
Could you describe a little bit more about how the alarms get to monitor ? DCAE is built around the concept of a shared message bus.
Guangrong Fu
Hi,
Sorry for the late response.
Holmes is not intended for alarm monitoring. If you're talking about how Holmes gets alarms from DCAE, as I know, there are many topics in the data movement module (DMaaP?) of DCAE. Holmes gets alarm data in almost real time by subscribing to the specific topic. Given the difference on the data format between DMaaP and Holmes, we are going to develop an adapter to convert the data into the format we need.
Correct me if I an wrong because I haven't thoroughly got to know the details of DCAE yet.
Regards.
Mary-Elise Haug
Thanks-- so Holmes is not compatible with the VES standard ?
Guangrong Fu
I have read the materials on VES. It seems that Holmes is almost compatible with the standard. The reason I said "almost" is that some key information (e.g. alarmCondition, specificProblem or any other information alike) is required to be globally unique in the fault fields for Holmes to identify an alarm. But it's not explicitly declared to be unique in the documents on https://github.com/att/evel-test-collector.
Mary-Elise Haug
It is not globally unique, it is unique in combination with other fields, such as device name or timestamp.
Guangrong Fu
Thanks for the information.
Since we could make an alarm identifiable by combining the attributes together, everything seems okay now.
Yuan Liu
I saw there is a GUI to design the correlation rules. Does it show the results of alarm correlation? Or it belongs to other projects?
Guangrong Fu
The GUI of Holmes provides interfaces for both rule designing and result viewing. But where to deposit our codes or where the GUI should be integrated is still under discussion.
Yuan Liu
OK, Thanks.
Tao Shen
Does Holmes support for importing/exporting/querying alarm correlation rules?
Guangrong Fu
Yes. All the operations above are supported.
Tao Shen
About publish analytic results to subscribers, is there any dispatch rules using for posting alarms to subscribers automatically?
If provide posting alarms automatically, will Holmes consider supporting management of alarm dispatch rules?
Guangrong Fu
We do publish the analytic results to a specific topic, but there's no dispatching rules. Anyone who wants to get the results have to subscribe to the topic.
Carol Nelson
Thanks for proposing this Event Correlation application. Event Correlation is a crucial capability in network management, for correlating events coming from various sources to identify the root cause event, reducing the number of events that need to be further processed. Based on the root cause event, further steps can be taken to rectify the problem.
The role of DCAE is to collect and analyze events & data from various sources e.g., VNFs, virtual/physical infrastructure (servers and network) and applications (including ONAP components themselves.) DCAE Collectors can support multiple formats, with VES and SNMP Trap formats supported in the first ONAP release.
A correlation analytic application will need to be registered in the SDC catalog in order to be used by CLAMP in designing closed or open loops. CLAMP designer is used to create a flow
The correlation analytic application needs rules/policies to drive the correlation logic. To do this, the analytic application can provide the ONAP Policy Designer with a (TOSCA) policy model specifying configurable attributes for that application. The Policy Designer uses that model to configure its GUI to collect values (configuration policies) for these attributes from the user. The Policy Designer distributes the collected values (in JSON format) to the Correlation application.
To leverage ONAP fully, the Holmes Correlation Engine application can be implemented as a DCAE analytic and be integrated with SDC. Aspects of the Holmes Rule Designer that are not supported in the ONAP Policy Designer can be absorbed into the ONAP Policy Designer.
Vijay Venkatesh Kumar
DCAE is real-time streaming platform; data collected are available to components within DCAE (like analytics, eventprocessor etc) and external application via dmaap in real-time fashion. There is no delay or performance overhead to be concerned here. One of DCAE primary function is to analyze raw events and trigger RCA event to downstream (Policy for CL flow). If Holmes function is contained within DCAE, the collected events can be processed consistently by a single engine driven by configurable policy. This would also help operation in defining analytics/correlation rules and associated openloop/closedloop action in a consistent manner.
HuabingZhao
Hi, Vijay, Could you please be more specific about which part of DCAE is handling "Analyze raw events and trigger RCA event to downstream (Policy for CL flow)"? Thanks.
Vijay Venkatesh Kumar
Hi HuabingZhao - RCA is not supported in the ONAP version of DCAE. However if you consider vF5/vDNS usecase - not all the events received into DCAE are sent to Policy. DCAE-Analytics generates a synthetic events based on matching thresholding rules (configured through policy) for Closedloop to be triggered.
John Strassner
"Holmes aims to find the real reason which causes the failure or degradation of services by digging into the ocean of events collected from different levels of the Telecom cloud."
<jcs>
This sounds like CEP, not like Policy. Of course, CEP MAY trigger policy. Hence, my confusion: does Holmes also want to define Policy, or can it use other types of Policy?
</jcs>
"The main difference between these two projects is that Holmes is mainly targeted at correlation analysis between different alarms while Policy is aimed to implement control loops by triggering a series of actions."
<jcs>
Policy SHOULD NOT be "limited to 'implement control loops by triggering a series of actions'." Policy is a vital element for decision-making and situation-awareness.
For example, the declarative policy:
"No processor should exceed 70% utilization"
is NOT targeted at a control loop (unless you want to have 1000s of control loops), it is stating operational intent.
Furthermore, it is not (in this form) suitable for DROOLS, as it needs translation into production rules.
I can give you other examples from OpenStack Congress (I helped write the compiler).
</jcs>
"The root cause is picked out from all the original alarms by Holmes and then, the most suitable policy ID is selected and published accordingly."
<jcs>
Do you really mean "the **most suitable** policy ID? What defines "most suitable"? I suppose you could do this using metadata and/or ontologies, but your proposal doesn't mention either.
Furthermore, why is this only limited to one policy?
Finally, s/published/invoked.
</jcs>
Jiaqiang Du
1, CEP?Complex Event Processing?is not open source, Drools and CEP have different applicable scene
For Event Correlation we think Drools is better choice
2, Drools is Declarative Programming and Logic and Data Separation
I think The declarative policy like "No processor should exceed 70% utilization" is suitable for drools,
Guangrong Fu
Hi John,
Holmes is sort of like CEP. It does NOT define or execute any policy. It only provides the analytic result to its downstream systems. Policy defining or execution is the responsibility of the Policy project.
For the function of Policy, I apologize for my improper descriptions. What you said is correct. I'm gonna update the wiki page according to your feedback.
I was saying "Policy ID" because I've seen it in some documents that the output of the string matchers somewhere inside DCAE is in forms of policy IDs. Generally, the output of Holmes is the relation between the alarms. Any downstream system who wants to consume the output of Holmes should take the original output from Holmes and convert it into the form that could be used in it.
I hope my answers do help.
John Strassner
Hi Guangrong,
thank you for your description. It helps a lot. There are still a number of unclear areas:
A separate question is, what is meant by "DCAE supports Holmes to be deployed as an analytic application in the form of docker(s)." Does this mean that you want to deploy Holmes as a set of container-based applications, or as a set of services that collectively define an application, or something else? Also, I am assuming that you are happy to use any container technology; if instead you mean just Docker (or have a preference for Docker), please let me know.
Guangrong Fu
Hi John,
Thanks for asking and here's the answers:
Holmes is currently designed to collect data in a pub-sub way. For example, to support DCAE as an analytic application, Holmes will subscribe to specific topics on DMaaP to collect events and publish the results back to DMaaP's topics.
Holmes is able to collect events from other publishers on condition that the data format is appointed in advance. As a stretched goal, we can design interfaces for data sources to push data to Holmes.
Sorry I don't quite get the point of "publish in a Drools format". Do you mean the data format or the system structure? If former, the answer is no. We'll adopt a normalized data format rather than a Drools specific one. If you're talking about the system structure, then yes. For now, we only take Drools as our rule engine.
The rule designer can be considered an configuration interface for the CRUD operations on rules. How Holmes deal with data depends on the logic of rules. A lot of rules can be set up in Holmes so we provide different ways to process data. If you're asking about this, I hope this helps. If I misunderstood your question, pls let me know then I'll reply again.
For R1, Holmes will be released in the form of dockers only. If other containers are required, we will take this into consideration in the following release.
John Strassner
Hi Guangrong,
"Holmes provides basic rule management functionalities which allow users to design, create or modify rules via a rule designer." This was the main paragraph that confused me. I am now assuming that what you meant is that Holmes provides basic management functions that enable users to create, design, edit, or delete the types of events that are ingested. Question: can Holmes support different correlation, filtering, and transformation functions? Or does it rely on the capabilities of an ESB to do this?
What is the purpose of the Rule Designer?
<jcs>
Having a rule designer implies that Holmes will create and manage its own rules. But earlier, you said:
"Holmes is sort of like CEP. It does NOT define or execute any policy. It only provides the analytic result to its downstream systems. Policy defining or execution is the responsibility of the Policy project."
Hence, my question is, IF Holmes does not define or execute policies, then why do you need a Rule Designer?
Perhaps what you mean is a designer to define how data are processed? This could be considered rules, but would be separate and distinct from Policy Rules. Is that what you mean?
</jcs>
Guangrong Fu
Yep.
I'm so glad that you got my point because it feels like we have different understanding on the words "policy" & "rules". I think I may make you more confused if I keep explaining using these two words. Fortunately, you've figured a way out for both of us to make it clear.
John Strassner
Hi Guangrong,
yes, I understand. Thanks for your clarification.
John Strassner
Thanks for the answers, but you didn't really answer my questions above. Following are comments on your 2 replies:
1, CEP?Complex Event Processing?is not open source, Drools and CEP have different applicable scene
For Event Correlation we think Drools is better choice
<jcs>
OF COURSE there are many open source CEP tools (the open source version of Drools being one of them, though you should specify which of the Drools suite of tools you need (e.g., just Drools Fusion, or others?). For example: OpenNMS, Simple Event Correlator, AlienVault OSSIM, and others (not to mention a host of tools available from universities). This doesn't include open source log analyzers (e.g., LogAlyze), which may also be able to be used.
More importantly, you haven't said whether
</jcs>
2, Drools is Declarative Programming and Logic and Data Separation
I think The declarative policy like "No processor should exceed 70% utilization" is suitable for drools,
<jcs>
Regarding the second point, why do you think that this is suitable for Drools? Drools is not a pure declarative language - its implementation translates a set of if-then clauses to an execution tree (meaning that order IS important). Don't get me wrong, I love Drools and have used it for a long time. I just don't think of it (despite the press) as a true declarative language.
</jcs>
Jiaqiang Du
"Whether Holmes is really just a CEP module, or something or more, less, or completely different ?"
A Sample:
Holmes Build parent and child relationship according to alarm raisedtime, topology relationship, and alarm code then find real root source. Same Site 'Power down' lead to 'BTS down' in 10 Minites, 'Power down' will be parent and 'BTS down' will be child after check alarm raisedtime, site information, and alarm code.
So I think Holmes is application/Component of CEP(Complex Event Processing) Kind
"Wheter Holmes is using a specific type of Policy, or Drools, or something else"
Holmes based on Drools/Drools Fusion currently
Zhiyi Luo
Holmes project provides alarm correlation and analysis for Telecom cloud infrastructure and services, including hosts, vims, VNFs and NSs.
Who or which project gives the resource relationship of Bare metal,the VMs or Containers and the services running on it? Cause that's the key to do the root cause analysis.
And if ONAP is the future OSS, how about the PNFs' fault management? Holmes can handle it or you still need the legacy OSS to get rid of it?
Guangrong Fu
In ONAP, A&AI is responsible for the maintenance of resources.
For the legacy systems, I have no idea whether they are going to be supported(for now, mostly no) by ONAP. Maybe you could ask the TSC or board members for help.
Sanjeeb Kumar Padhan
hello,
Can any please guide me how to integrate and deploy HOLMES with DCAE or in standalone mode. Any reference or any document related to deployment of HOLMES.
holmes-gui is also not available on the given link https://gerrit.open-o.org/r/holmes-gui.
thanks and regards,
Sanjeeb Kumar Padhan
Guangrong Fu
Hi,
The GUI of Holmes is integrated into the client project in Open-O. So you'd better use git clone https://gerrit.open-o.org/r/client-gui to fetch the code.
As for how to integrate Holmes with DCAE or deploy it in standalone mode in ONAP, we are still looking for a way to achieve it. Once we figure it out, we'll create a wiki page asap.
Sorry for the inconvenience.
Regards,
Guangrong
Manoj Nair
Can you please let me know where I can find an example alarm correlation rule for Holmes ?
Guangrong Fu
Hi Manoj,
As we haven't got enough information on realistic scenarios and alarms, only a simple sample could be provided as follows:
package gsm.bsc.test import xxx.xxx.xxx; rule "NFVO_SameHost_Relation_Rule" salience 120 no-loop true when $root : Alarm( eventType == Alarm.EVENT_RAISED, probableCause == "RAM error", $hostname:hostname, hostname != null && !hostname.equals(""), $vmname:vmname, vmname == null || vmname.equals(""), $aid: aid) $child : Alarm( eventType == Alarm.EVENT_RAISED, aid != $aid, hostname == $hostname, probableCause in ( "compute is not available","VNF IP NSVC unavailable" ), this after [-60s, 60s] $root) then ServiceAccess.getAplusOutputService().publishRootChildResult("rule1", $root, $child); end rule "NFVO_Notify_Policy_Rule" salience 120 no-loop true when $alarm : Alarm( eventType == Alarm.EVENT_RAISED, probableCause == "RAM error", $hostname:hostname, hostname != null && !hostname.equals(""), $vmname:vmname, vmname == null || vmname.equals(""), $aid: aid) then Event event = new Event($alarm); ServiceAccess.getAplusOutputService().sendEvent("rule2", event); endHope it helps.
Regards
Guangrong
Manoj Nair
Thanks Guangrong. In the above sample if I understand correctly once a root alarm is raised a child alarm is awaited for 60 seconds . And resulting action is triggered. As per my view of Alarm correlation, when an alarm is raised, we check the inventory to map the parameters and enrich and probably wait for related alarms and then do a complex condition check . As per the above rule if I understand we are checking only what type of alarm we received and what related alarm we expect, not more than that right ?
Guangrong Fu
Hi Manoj,
I'm sorry for the confusion caused by this example. Actually, as what you said, there will be a complex condition check based on the topological information or any other criteria. Regarding the sample above, it's just a brief rule intended for demonstration only for the simplest scenario. In reality, the rules would be much more complicated.
Maybe you could take a look at the sample rule provided on this page: Holmes Weekly Meeting (20170720). In this demo rule, the complex analysis is accomplished by the method CorrelationUtil.getInstance().isTopologicallyRelated(sourceId, $sourceId). Anyway, it's still a demo so the logic of the rule seems to be simple.
I hope that it could make the question you raised clearer and you're welcome for further question if there's any.
Regards,
Guangrong
You Li
Hello, if I would like to develop a new analysis application ?like Holmes?for dcae, how can I intergrate it into DCAE? Do I have to add my application code to DCAE-APOD and rebuild the whole DCAE, or there is other way to add plugin to DCAE without rebuilding the whole project ?
Guangrong Fu
Hi,
As for this question, I think you'd better ask the DCAE team for suggestions. Holmes has its own gerrit repo so that it could be built independently and deployed into DCAE in the form of docker containers. I have no idea on your question either. Sorry for that.
Regards,
Guangrong
You Li
OK, Thanks
You Li
Hi?I would like to learn the Holmes Project, however, there are only 4 child project in gerrit repo(which do not have the parent pom file). Is there anyway that I can fetch the parent project? Thanks.
Guangrong Fu
Hi. Sorry for my late response.
Actually, what you find in the gerrit repo is all we have got at the moment. Holmes does not actually have a parent pom. It comprises of 3 micro-services and each of them is supposed to be made into docker containers and be running individually.
For now, they are not runnable yet because some dependencies are missing and we are still trying to figure out a way to deploy the dependencies and change our code accordingly.
Ning Zhu
Hello,
There is such a sentence :
String filePath = "C:\\correlation-engine.yml"
in file EngineDActiveAppTest.java.
where can I get this yml file? it is not a resource file. thanks.
wu hantao
hi?any progress between the DCAE and Holmes project, what is the relationship between these two parts ?
Guangrong Fu
In R1, DCAE and Holmes are integrated. Holmes could be deployed and (sort of) configured by DCAE. But there are still several gaps for us to get over in R2 regarding the integration work and the whole control loop operations.
Govardhan Reddy
Hi Guangrong Fu
In Holmes,
1) Is there any priority assigned for alarms or rules?if yes, on what basis we assign priority?
2) How does holmes pick out the root cause from an ocean of alarms? Can you please share the code of where and how the correlation logic is really implemented?