Requirement:

As an ONAP developer, I need to support policy_type_impl as an entity in Policy Framework.

Currently only APEX-PDP is using policy_type_impl in Policy framework, but other PDPs could use it too.

The policy_type_impl itself is contained within the ToscaPolicy in case of APEX.

The intention of this Jira is to decouple the policy_type_impl from ToscaPolicy definition itself.

policy_type_impl could be stored in a database say with a name and version, and then in the ToscaPolicy, 
only this name and version need to be specified instead of the whole policy_type_impl  content.


Current approach:



Existing Policy Tosca:

{
  "tosca_definitions_version": "tosca_simple_yaml_1_1_0",
  "topology_template": {
    "policies": [
      {
       "operational.apex.decisionMaker": {
          "type": "onap.policies.native.Apex",
          "type_version": "1.0.0",
          "derived_from": "tosca.policies.Root",
          "name": "operational.apex.decisionMaker",
          "version": "1.0.0",
          "properties": {
            "engineServiceParameters": {
              "name": "MyApexEngine",
              "version": "0.0.1",
              "id": 45,
              "instanceCount": 4,
              "deploymentPort": 12561,
              "policy_type_impl": {
                ....
                ....
                
              }
            }
          }
        }
      }
    ]
  }
}


Proposed solution:

The existing structure of Tosca policy template is preserved and works as it is today.  Additionally the following support will also be included.

  • The policy_type_impl entity can be inferred as a metadataSet required for the policy engine.
  • The metadataSets can be represented as node_templates in the Tosca mapped to a specific node type. It can be independently stored and retrieved from the db
  • The metadataSet can also be passed as a referenceId  in policy definition under metadata field.
  • Rest endpoints will be introduced in the policy api to for metadataSet CRUD in to the database.
  • PAP checks for the metadataSet reference in tosca policy.
  • PAP fetches the corresponding metadataSet definition from the db if required before passing on the ToscaPolicy to the Pdps.
      
        


The following cases are proposed to be supported in Jakarta supporting the backward compatibility of Istanbul:

In policy tosca , "metadataSet reference" (or) "metadataSet" or neither can be provided, providing both together is an error.

Case 1


Reference provided only
API→PAP

{
  "tosca_definitions_version": "tosca_simple_yaml_1_1_0",
  "topology_template": {
    "policies": [
      {
        "operational.apex.decisionMaker": {
          "type": "onap.policies.native.Apex",
          "type_version": "1.0.0",
          "derived_from": "tosca.policies.Root",
          "name": "operational.apex.decisionMaker",
          "version": "1.0.0",
          "metadata": {
             "metadataSetName": "apexMetadata_decisionMaker",
             "metadataSetVersion": 1.0.0"             
           },           
           "properties": {            
            "engineServiceParameters": {
              "name": "MyApexEngine",
              "version": "0.0.1",
              "id": 45,
              "instanceCount": 4,
              "deploymentPort": 12561
            }
          }
        }
      }
    ]
  }
}

PAP->PDP

{
  "tosca_definitions_version": "tosca_simple_yaml_1_1_0",
  "topology_template": {
    "policies": [
      {
        "operational.apex.decisionMaker": {
          "type": "onap.policies.native.Apex",
          "type_version": "1.0.0",
          "derived_from": "tosca.policies.Root",
          "name": "operational.apex.decisionMaker",
          "version": "1.0.0",
          "metadata" : {             
             "metadataSetName": "apexMetadata_decisionMaker",
             "metadataSetVersion": 1.0.0",                
             "metadata": "The rule set read from the DB"
           },
           "properties": {
            "engineServiceParameters": {
              "name": "MyApexEngine",
              "version": "0.0.1",
              "id": 45,
              "instanceCount": 4,
              "deploymentPort": 12561
            }
          }
        }
      }
    ]
  }
}


Case 2


Rule set provided in Policy
API-> PAP


{
  "tosca_definitions_version": "tosca_simple_yaml_1_1_0",
  "topology_template": {
    "policies": [
      {
        "operational.apex.decisionMaker": {
          "type": "onap.policies.native.Apex",
          "type_version": "1.0.0",
          "derived_from": "tosca.policies.Root",
          "name": "operational.apex.decisionMaker",
          "version": "1.0.0",
          "metadata": {
             "metadataSet": "My Rule Set for some PDP"
           },
           "properties": {
            "policy_type_impl": "My Rule Set for some PDP",
            "engineServiceParameters": {
              "name": "MyApexEngine",
              "version": "0.0.1",
              "id": 45,
              "instanceCount": 4,
              "deploymentPort": 12561
            }
          }
        }
      }
    ]
  }
}


PAP->PDP just forward same policy, no DB lookup


Case 3: APEX Backward Compatibility

API → PAP

{
  "tosca_definitions_version": "tosca_simple_yaml_1_1_0",
  "topology_template": {
    "policies": [
      {
        "operational.apex.decisionMaker": {
          "type": "onap.policies.native.Apex",
          "type_version": "1.0.0",
          "derived_from": "tosca.policies.Root",
          "name": "operational.apex.decisionMaker",
          "version": "1.0.0",
          "properties": {
            "engineServiceParameters": {
              "name": "MyApexEngine",
              "version": "0.0.1",
              "id": 45,
              "instanceCount": 4,
              "deploymentPort": 12561,
              "policy_type_impl": {
                
              }
            }
          }
        }
      }
    ]
  }
}


PAP->PDP just forward same policy, no DB lookup


For APEX, if the metadataSet is not in the engine service parameters and is in the policy metadata, just copy
it from the metadata and forward to the APEX core.



Yaml representation of sample metadataSets that can be persisted to db:

sampleMetadataSets.yaml

The 'metadataSet' entity is represented as a node template in Tosca mapped to a specific node type under the topology template: The following parameters are specified in tosca under the node templates:

parametersTypeDescription
<metadataSetName>String (key)The name of the metadataSet
versionString (key)The version of the metadataSet
typeStringNode type
type_versionStringNode type version
descriptionStringDescription of the metadataSet
metadataMap

A map of <String object> that contains the policy Metadata

  • No labels

13 Comments

  1. Ram Krishna Verma

    Thanks for putting up the high level design Ramesh Murugan Iyer 

    Few observations/clarifications

    • Does it mean PAP will fetch the ToscaPolicy & the policy_type_impl at deployment time and merge them together, before sending to PDP? If yes, it will add extra database dips and compute cycles, which is okay. But we need to make sure it doesn't become too slow.
    • You are adding a new field in ToscaPolicy named policy_type_impl_ref, what about conflicts? Which one takes preference? Or we allow both?
    • Are you are planning a change in Apex-PDP to read the new field and take policy implementation for deployment? Again, we need to think about conflicts here.
    • The new field will be added in the common operational policy type or apex-pdp specific one's. If common, then we need access impact on other PDP engines (drools & xacml).
    • Any thoughts about representing the format of policy_type_impl as separate entity? Example - Tosca, Json, Yaml etc.
    • There can be complications in apex-pdp, to handle duplicate policy constructs (task, state, event, etc.). Example - if someone tries to deploy multiple ToscaPolicy using the same policy_type_impl_ref. Probably a pre-verification of the scenario with existing flow will be helpful.
    • It will be good to have a possibility to pre-load few policy_type_impl entities in the database during startup. Same as policy type and policy.

    1. Ramesh Murugan Iyer

      Ram Krishna Verma Thanks for the observations.  

      • I will do the performance analysis in PAP after adding the additional database queries for policyTypeImpl.
      • We allow either policyTypeImplRef or policyTypeImpl.  PAP takes the decision to query the db only if policyTypeImplRef is provided. 
      • Apex-PDP will have minor additional changes in the json parsing functionality to consume the policyTypeImpl entity from the metadata if provided. Supports backward compatibility to consume it from the properties.
      • The new reference property will be added under metadata under policies, I think the existing functionality in policy models can persist metadata field in tosca. PDPs consume only the required fields from the tosca. So I don't think it will have any impacts.  Please correct me if my understanding is incorrect. 
      • I have attached the sample json for representing policy type impl entity. Please verify the changes.
      • I will check the possibilities to preload policy_type_impl in db during startup. 

      1. Ram Krishna Verma

        Ramesh Murugan Iyer Thanks for the reply.

        Couple of more observations based on recent updates in this document.

        • Do you have thoughts about how the policyTypeImpl entity will be created for apex-pdp? The current CLI editor or Tosca CLI editor will fail to do so.
        • I am doubtful that this can be achieved with minor additional changes in apex-pdp. Specially about the fact that multiple tosca policies may bring the same policy model implementation (events, task, state, policy etc.).
        • Drools-PDP might be doing strict validations of tosca policy against the policy type. But I guess keeping the new fields optional will help.
        • It will be good to have a swagger spec for the new API's being added to policy-api component.

        1. Liam Fallon

          Ram Krishna Verma Just some observations on the bullets

          1. The CLI editor does create the policy_type_impl but it doesn't have an option to create it separately, we need an option on the CLI editor to allow it to be generated separately or simply always create it in the "target" in maven as an extra file in the build, Ramesh Murugan Iyer we need a JIRA task or sub-task for this.
          2. Why would this be any different to what happens today in Apex, we're simply moving the location of a property? Ajith Sreekumar do you have any thoughts here?
          3. I don't think this will impact Drools-PDP, unless a policy-type-impl is specified with Drools-PDP as the PDP type, the policy received by a Drools-PDP instance will be the same as it is today. In the future this mechanism could be used to deliver metadata to a Drools-PDP instance, of course in that case there would most likely be development work to do on Drools-PDP to receive metadata in that way.
          4. I agree, we should have a Swagger spec and ideally geenrate the API code from the Swagger (Open API?) rather than the other way around.

          1. Ram Krishna Verma

            Liam Fallon Thanks for the update!

            • I mean the current CLI Editor won't be able to create the new structure of policy_type_impl entity (as defined above). It will only create the inner piece (apexpolicymodel) and user have to take it and draft the other fields to create the policy_type_impl entity payload.
            • Currently 2 tosca policies are not recommended to share the same entities. As they are created using independent maven module. And we have set guidelines to prefix any entity name with "usecase name" to make it unique. And the ToscaCLIEditor generates the whole tosca policy payload. For sure a user can go in and change the generated tosca policy manually and create duplicates. But we don't support that officially.
            • Personally I don't like swagger generating the java code. As it adds a lot of feathers. But we can give it a try for sure.

            The above 2 tasks on apex-pdp can be done separately after the support for policy_type_impl & policy_type_impl_ref is created in api & pap.


            1. Liam Fallon

              On the CLI editor points I think it wouldn;t be a lot of work to update the CLI editor to create the body of the REST call for the new API. Also, as you point out it can be done after the API calls are finished. Ramesh Murugan Iyer could you create a new story on the Apex epic to track this?

              On Swagger generating code, let's have a separate discussion on this point.

  2. Jorge Hernandez

    I don't see clearly what is being proposed, could you add an actual policy example to see the differences between what we have now and what you are proposing?   That will make it more clear.    Is this something currently only used in apex?

    1. Ramesh Murugan Iyer

      Hi Jorge  this is currently only used in apex. We are proposing to add policy_type_impl as a separate entity in Tosca generic for all PDPs. policy_type_impl_ref will be supported in tosca which is a Tosca concept Identifier field. PAP fetches the corresponding policy_type_impl from DB accordingly also supporting the backward compatibility of existing flow.  I have updated the page with example policy tosca for both current and proposed solutions including API-PAP and PAP-PDP interactions. Please provide your thoughts. 

      1. Jorge Hernandez

        Thanks a lot Ramesh Murugan Iyer for the feedback!    I am still having a little bit of trouble getting its value and how it can be generic.   In theory, if I remember correctly everything under properties is to be interpreted as a blob under the context of the policy type by the pdp that understands the policy type.   I am concerned with having custom apex behavior in the PAP which in theory is a neutral component.     There is currently some leakage of apex data into generic structures like for example in statistics with the engine stats that seems to be have meaning only under apex.    May be present in policy weekly, and we can have a discussion there with all committers.

        1. Liam Fallon

          Hi Jorge Hernandez ,

          Let's take this at the committers meeting on Tuesday or at one of the scrums. I think rather than making the PF more specific to Apex, we are actually doing the opposite, bringing the parameter handling and meaning of a Policy in APex more into line with its meaning in Drools and XACML, and this approach could be used in the future ti distribute rule sets for any PDP, not just Apex. This goes back to ideas we had in the architecture from Dublin. Let's discuss all this on a call.

  3. Pamela Dragosh

    Not sure I get this at all, I don't understand what is shown as different between the cases. I'll look forward to more information. Please note that the XACML PDP most definitely uses Policy Type implementations in order to retrieve metadata on how to generate policies derived from that policy type. As long as programmatically the changes are trivial and/or backward compatible, I'm ok with any work that can help the Apex PDP.

  4. Liam Fallon

    On performance, I can't see that that change will make much difference, for Apex the Policy will be the same size that's sent at deployment and also for the other PDPs. It'll be one if statement and a DB dip in the case of Apex, should be a minimal change.


  5. Liam Fallon

    The DB table for the metadata should also have a PDP type field. Therefore if a policy type has 2 implementations on 2 different PDPs, the PAP can send the correct implementation to the correct PDP when a policy of that type is deployed. SO the DB columns could be something like this:
    name
    version
    policy_type_name
    policy_type_version
    pdp_type
    policy_type_pdp_specific_metadata

    with the primary key being Name:Version, policy_type_name:policy_type_version is a foreign key to find the policy type and the pdd_type is a foreign key to find the PDP type (to be pedantic about it)