TL;DR

Today, ONAP uses a number of different container base images and different versions of the same base images. At the same time base images are not vetted by the security subcommittee.  For example, we currently use 5 different versions of node, 9 different versions of openjdk and 13 different versions of Python across ONAP projects.

This approach has led to duplication of efforts and inefficiencies, such as resource consumption, large footprint and long build times. Also, teams experience unnecessary pain when unexpected changes occur in upstream projects.

The solution is to adopt a set of common base images, (onap/base-xxx) with a well defined life cycle:

• Proposed
• Under review
• Adopted
• Maintained
• Retired

PTLs, the Security Sub-committee and the TSC will drive images through this life cycle.

Images will remain stable during each development cycle and will only be updated in-between development cycles.

The ONAP community already came together and made great progress in reducing image footprint. A lot of people contributed to make that a reality.

We still have work to do. This proposal gives us an opportunity to work together to uncover, and remove, efficiencies and to introduce time-saving strategies to help ease everybody’s pains.

Problem Statement and Vision

As a community we are dealing with a number of different base images, different versions of the same base images, images are not vetted by the security subcommittee.  This has led to duplication of efforts, inefficiencies such as resource consumption, large footprint and long build times. Also, teams, experience unnecessary pain when unexpected changes occur in upstream projects.

These problems are caused in part because everyone is spread too thin. Getting more contributors is not always feasible. We have a lot of moving parts to deal with. On top of that, we need to release on time while at the same time stabilize new features and reduce technical debt.

For example, the community came together and made great progress in reducing the footprint. A lot of people contributed to make that a reality. We still have work to do.  We have an opportunity to work together to uncover, and remove, efficiencies and introduce time-saving strategies to help ease everybody’s pains.

What if we could mitigate these problems by adopting common base images with a clear and predictable life cycle? What if these common base images were re-usable across a large number of ONAP projects?

Furthermore what if these common images were vetted by the community in terms of their footprint, stability and security?

Re-usability

Figure 1 shows how ONAP uses a number of different base images for similar purposes and different versions of the same base images.

For example, we currently use 5 different versions of node, 9 different versions of openjdk and 13 different versions of Python across ONAP projects. There is very little re-use and the commonalities across projects are not being exploited.

It is clear that there is significant commonality across ONAP projects. What if we benefited from that commonality by adopting a set of common base images that were shared across ONAP projects? These common images will increase re-use. Teams and PTLs will share stable base images across projects. Developers will have a common starting point for their services. Most importantly, developers will have one less thing to maintain and worry about.

Figure 1. Image versions

Security

Base image adoption is  not vetted by ONAP’s security subcommittee. We are missing the opportunity to adopt, in a scalable manner, security best practices and guidelines across all base images.

What if the Security sub-committee had the opportunity to review and provide input on the common images we adopt? We will be able to adopt, in a scalable manner, their recommendations and enable security best practices and guidelines across all base images.

Stability

Like any other open source project, we can’t control upstream projects. Upstream project updates have proven to disrupt our work and caused us pain. Case in point, the change in the nss library that affected a few ONAP projects. We can’t prevent these events but we can certainly take actions to mitigate their impact.

Figure 2 shows ONAP's most popular distributions for base images. It is clear that ONAP projects favor alpine for their development base images. "Development base images" are those used to develop ONAP services (e.g. python, java, node).

What if could store, share, update, analyze and secure a common alpine distribution? This approach will ensure both re-use and stability during each development cycle.

Proposed Solution

Common OS Repository

A practical way to achieve stability is to install a local OS repository, such as an Alpine distribution repository. This repository (i.e. mirror) can be easily installed, locked and stabilized for the duration of an ONAP release.

Figure 2: Percentage of Linux distribution usage for ONAP development images

Common Base Images

Other than creating production-ready images, there are benefits to development in being able to easily replace the base image.

For example, it's possible to create a development (i.e. debuggable) version of the base image, with default configuration enabling more logging, instrumentation, network capturing services, etc.  It also makes it easier to test the impact of an upgrade of a library across all of ONAP, for example moving from JDK 11 to JDK 12.

Common base image repository

A practical way to enable the use of common base images is to setup a common repository.

What would be the content of the repository? Besides the basic Alpine Linux distribution, its libraries and packages, the repository will contain base image abstractions or aliases.

Base Image Abstractions

Image abstractions can be used to increase re-usability and enhance stability.

For example, for ONAP Python projects the "FROM" statement could point to:

onap/base-python2

Likewise for java projects

onap/base-java-8

And node projects

onap/base-node-9

Each of these base images could in turn derive from the same minimal base, "onap/base", just with the additionally installed dependencies.

What would be the content of these images? For our reference implementation we propose Alpine Linux. We need to start somewhere where we have good adoption and experience.

End users, however, would be able to replace them with compatible alternatives.

Let's consider an example derivation path for one image:

onap/base → onap/base-java-8 → onap/so

If the reference implementation of "onap/base" Alpine Linux, "onap/base-java-8" would derive from that, and add an OpenJDK 8 distribution.

Then "onap/so" would use "onap/base-java-8" in its Dockerfile "FROM" field.

All Java 8 components in ONAP would be doing the same, so any consumer would only ever download that JDK once. It would be relatively easy to swap out that JDK. For example, a consumer might want to use a security-patched, custom-built version of OpenJDK and not wait for ONAP to make a new release. So, they would first rebuild the "onap/base-java-8" image, and then trigger a rebuild of all derived ONAP packages. Assuming integration tests are successful, they are good to go.

Dependencies

One potential challenge would be where to put dependencies: there might be a concern that various little dependencies would be pushed into "onap/base" and that it would become bloated.

There are two potential answers to this:

1) So what if it becomes a bit bloated? It is only ever downloaded once. And there's a big advantage to seeing our dependencies in one place.

2) Our set of base images can evolve over time to accommodate this. For example, let's say that a specific container requires a bunch of large PDF and PostScript manipulation libraries. It might make sense to create a new "onap/base-graphical" derivative, even if it's just for one project.

Implementing this change would be relatively easy to do across ONAP. For the most part, we already know what the set of base images would look like. The current quirky build processes for each project can remain the same and it would just be changing the "FROM" field.

Base Image Life Cycle

It is relatively easy to introduce a new base images to ONAP today. On the other hand, the process of maintaining those images and deciding when they no longer serve their purpose is not consistent across ONAP projects.

A well-known, predictable image life cycle will help implement a transparent process to adopt common base images. To accomplish this goal, Figure 3 shows the proposed life cycle for ONAP’s common base images.

Figure 3: Life Cycle of ONAP's Common Base Images

Governance

Today, developers and teams choose --independently for the most part-- their base images. It all makes sense because developers know what is best for their projects. We are clearly confronted with a case of local optimization.

What if we came together as a community (PTLs, TSC, Security Sub-committee), shared our experiences, reduced duplication and minimized the effort to maintain common components.

At the same time, what if we gave all stakeholders a voice in deciding what images and versions we should adapt?

Process and Stakeholders

Initial setup

As the first step of the proposed approach, the Linux Foundation infrastructure team sets up a stable repository of base images and libraries to hold and serve the current set of ONAP Normative Container Base Images.  This is a one-time event expected to take place before El Alto’s M1.

Stable process

After the initial setup, the process enters into a cycle that follows the cadence of ONAP releases as follows:

Roadmap

The proposed approach to this initiative is incremental and iterative as shown on the roadmap below.

Incremental because the Common Images are defined, contributed and adopted incrementally, release after release.

Iterative because the Local Base Image Repository is updated and locked with every release cycle.

Dates are subject to change in order to align with El Alto's newly proposed timeline.