High Level Architecture for the Networking Plugin




Networking Plugin

The Networking Plugin has the ability to create, delete, query and update virtual networks. This will implement API's like CreateVL, DeleteVL, QueryVL and UpdateVL. Networking plugin on initialization will call discovery function to find out about the type of networking plugin available. It'll load the required plugin and call init functions of the appropriate plugin.

CreateVL:

Input: Name of the Virtual Network, Subnet IP, Gateway IP

Output: Success/Failure

  • Check in the database if a virtual network with same name already exists. If so return error.
  • Get the networking artifact.
  • Figure out the networking plugin to call
  • Call the appropriate plugin to initialize virtual network.
  • If success returned then add the Virtual network to the database
  • Return success

Create VL for OVN:

Input: Name of the Virtual Network, Subnet IP, Gateway IP

Output: Success/Failure

  • Figure out if the virtual network already exists. Use the ovn-nbctl call to figure that out. If already exist skip next step.
  • Create Logical switch with name as provided in input and  other-config:subnet=<Subnet-IP> set to the subnet required for the virtual network and external-ids:gateway_ip=<Gateway-IP> set to the gateway IP for the virtual network.

                      sudo ovn-nbctl --may-exist ls-add ovn-ls-24 -- set logical_switch ovn-ls-24 other-config:subnet=172.16.24.0/24 external-ids:gateway_ip=172.16.24.1/24


  • For East-West connectivity connect the switch to the distributed router:
    • Add switch port to the distributed router. Mac address needs to be generated. In the example below "k8smaster" is the name of the distributed router created for the cluster based on the name of the master node.
      sudo ovn-nbctl --may-exist lrp-add k8smaster rtos-ovn-ls-24 00:00:00:A0:96:B9 172.16.24.1/24
    • Connect distributed router port to the switch
      sudo ovn-nbctl -- --may-exist lsp-add ovn-ls-24 stor-ovn-ls-24 -- set logical_switch_port stor-ovn-ls-24 type=router options:router-port=rtos-ovn-ls-24 addresses=\"00:00:00:A0:96:B9\"


  • For  North-South connectivity update the gateway router with route and SNAT rules for the new switch. The example below is based on the assumption that there is only one Gateway Router created for the cluster. The name of the Gateway Router in the example is GR_k8smaster.
    • For external connectivity add route to the gateway router: (The "join" is allocated IP addresses in the range 100.64.1.0/24)

      ovn-nbctl  --may-exist lr-route-add GR_k8smaster 172.16.24.0/24 100.64.1.1
    • Add SNAT rules to the gateway router. "192.168.121.19" is the external_IP

                   ovn-nbctl  --may-exist lr-nat-add GR_k8smaster snat 192.168.121.19 172.16.24.0/24


  • Return success if initialization is successful

Multus Integration

One of the requirements for VNF's  is to support multiple virtual network interfaces and multiple IP addresses. Multus acts as a Multi plugin in Kubernetes and provides the multiple network interface support in a pod. https://github.com/intel/multus-cni. It'll be used in this project to provide a default management port based on Flannel to all VNF's. The other interfaces will all be based on ovn-kubernetes as discussed in the next section.

CNI configuration

Multus is configured to have default network as Flannel:

{

"type": "multus",
"kubeconfig": "/etc/kubernetes/admin.conf",
"delegates": [
   {
         "type": "flannel",
         "masterplugin": true,
          "delegate": {
              "isDefaultGateway": true
           }
    }
]

}

Create following network resource for OVN.

OVN network resources in Kubernetes:

apiVersion: "kubernetes.cni.cncf.io/v1"
kind: Network
metadata:
name: ovn-network
spec:
config: '{
"name": "ovn-kubernetes",
"type": "ovn-k8s-cni-overlay"
}'

With this setup a Pod can be annotated as below for OVN:

apiVersion: v1
kind: Pod
metadata:
  name: pod-ovn
  annotations:
     kubernetes.v1.cni.cncf.io/networks: '[
                { "name": "ovn-network"}
      ]'
spec: # specification of the pod's contents
containers:
     - name: pod-ovn
      image: "busybox"
      command: ["top"]
      stdin: true
      tty: true

With this setup and with above pod definition a Pod with two interfaces is created. This has a limitation though. The OVN interfaces always belongs to the subnet configured during initialization.

OPEN: To get above pod configuration to work ovn-kubernetes had to be modified and AddRoute for the Pod had to be removed. 

Ovn-kubernetes design and changes for multiple interface support

  • ovn-kubernetes current design doesn't support multiple interfaces and also a single default network is created at the time initialization time and all pods are connected to that.
  • Currently if a Pod is annotated with "ovn" then static IP, MAC can be assigned for the single interface that is created. And if the pod is not annotated with "ovn" then the ovn watcher annotates the pod with the "ovn" with dynamic address allocation. This information is then used by OVN CNI plugin to assign address for the interface. For backward compatibility this behavior will not change and pods can be created with single interface as before. The one interface is attached to a default network created at the time initialization time.
  • To create multiple interface the pods or to attach single interface to a different virtual network a new annotation is added to the Pods - "ovnNetwork". If "ovnNetwork" annotation is present then the "ovn " annotation is ignored.
  • Assumption CreateVL call is already made prior pod getting created with the virtual network needed for new Pod initialization.
  • To create a pod with multiple interfaces annotate the pod like below:

{ "name": "ovn-ls-25", "interface": "net0" },

{ "name": "ovn-ls-26", "interface": "net1", "ip_address":"172.16.26.3", "mac_address":"0a:00:00:00:00:15" }

        In this example net0 interface will be connected to a "ovn-ls-25" network and dynamic address allocation will be done for the addresses. net1 is connected to the ovn-ls-26 network and static address is used as specified.

    • "name": Virtual network name, used to connect interface to network
    • "interface": Name of the interface. Assumption: Current 10 interfaces are supported. All interfaces should be named with alphanumeric characters with the last character being a number between 0-9. This is required as this number is used to distinguish between interfaces within the pod namespace.
    • Ip_address: static ip. Optional if not given dynamically assigned
    • mac_address: static mac. Optional if not given dynamically assigned
  • ovn-kubernetes watcher based on the "ovnNetwork" annotation connects the interfaces to the right virtual network and creates a list of interfaces and corresponding addresses for the CNI to assign addresses to the newly created Pods.  It annotates the pod with the list "ovnIfaceList".

 For example:  ovnIfaceList=[{"ip_address":"172.16.25.3", "mac_address":"0a:00:00:00:00:24", "gateway_ip": "172.16.25.1"}, {"ip_address":"172.16.26.3", "mac_address":"0a:00:00:00:00:15", "gateway_ip": "172.16.26.1"}]

Pod Specification with Multus and multiple interface support in OVN. The pod created has eth0 interface configured by flannel and one interface net1 on "ovn-ls-24" network:

apiVersion: v1
kind: Pod
metadata:
  name: pod-ovn
  annotations:
    kubernetes.v1.cni.cncf.io/networks: '[ { "name": "ovn-network"}]'
   ovnNetwork: '[
      { "name": "ovn-ls-24", "interface": "net1", "ipAddress": "172.16.24.55", "macAddress": "0A:00:00:00:00:4E" },

      {"name": "ovn-ls-24", "interface": "net2"}

]'

spec: # specification of the pod's contents
containers:
- name: pod-ovn
image: "busybox"
command: ["top"]
stdin: true
tty: true

Configuration details

ONAP Network Definition  :

apiVersion: v1
kind: onapNetwork
metadata:
name: ovn-ls-24
spec:
   config: '{

      "cnitype" : "ovn",
      "name": " ovn-ls-24",
      "subnet": " 172.16.24.0/24",
      "gateway": " 172.16.24.1"
}'

If no cnitype defined default is ovn.

Pod/Service annotation:

apiVersion: v1

kind: Pod

metadata:

  name: pod-ovn

  annotations:

      kubernetes.v1.cni.cncf.io/networks: '[{ "name": "ovn-network"}]'

      onapnetworks: 

{  
   "ovnNetwork":[  
      {  
         "name":"ovn-ls-26",
         "interface":"net0",
         "defaultGateway":"true"
      },
      {  
         "name":"ovn-ls-24",
         "interface":"net1",
         "defaultGateway":"false",
         "ip_address":"172.16.24.2/24",
         "mac_address":"0a:00:00:00:00:1e",
         "gateway_ip":"172.16.24.1"
      }
   ],
   "ovnNetworkRoutes":[  
      {  
         "dst":"172.16.29.1/24",
         "gw":"172.16.24.1"
      }
   ]
}

Pod Annotation for CNI (Internal to the plugin)

ovnIfaceList=[{"ip_address":"172.16.26.2/24", "mac_address":"0a:00:00:00:00:1d", "gateway_ip": "172.16.26.1","interface":"net0",  “defaultGateway”:”true”},

                             {"ip_address":"172.16.24.2/24", "mac_address":"0a:00:00:00:00:1e", "gateway_ip": "172.16.24.1","interface":"net1",  “defaultGateway”:”false”}]

ovnNetworkRoutes: [{ "dst": "172.16.29.1/24", "gw": “172.16.24.1” }]

  • No labels

1 Comment

  1. Srinivasa Addepalli

    Hi Ritu,

    You might have seen this: https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/switch-per-node.pdf

    In my view when createVL is called, OVN plugin should do following:


    1. Check if the VL exists already. If not do following:
      1. Create distributed logical switch (let us not worry about per node logical switch, I don't think it matters. In future, we can think of creating multiple logical switches if we find some performance issues.   Number of logical switches are not per node basis.  When more than certain number of ports assigned to a logical switch, we can create new logical switch for a given VL).
      2. We will assume that the distributed router, gateway, join switch between distributed router and gateway and external switch between gateway and physical network are already created (Either by the site configuration itself or by the ONAP - TBD though, for now we will assume that site has created those logical entities).
      3. Create a port between newly created logical switch with the distributed router. 
      4. IP address of the above port in distributed router is 'gateway IP address' or the 1st IP address of the subnet.
      5. If there are multiple subnets, we need to create as many ports and associate them with the distributed router. 
    2. If VL exists already, check whether the subnet IP addresses also match, if not
      1. If some subnets don't match, delete the subnets that don't match. It involves removing the distributed router port that was created for that subnet before.
      2. If there are new subnets, then add new subnets.


    Srini