The main of this page is to compare the existing k8S security tests versus the Kubescape tool developed according to NSA recommendations.

Security Tests

 Integration Security tests are deployed (some tests are developped internally) by integration Teams. 


Security Tests

Tests

Description

Code

Comments

root_pods

check that pods are nor using root user or started as root

bash script

kubectl

unlimitted_pods

check that limits are set for pods

bash script

kubectl

cis_kubernetes

perform the k8s cis test suite (upstream src aquasecurity)

bash script

kube-bench

nonssl_endpoints

check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels

Go script

kubetl, nmap

http_public_endpoints

check that there is no public http endpoints exposed in ONAP cluster

bash script

kubectl,nmap

jdpw_ports

check that there are no internal java ports

bash script

kubectl, procfs

kube_hunter

security suite to search k8s vulnerabilities (upstream src aquasecurity)

kube-Hunter

kube-Hunter

versions

check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains

python module

cerberus, kubernetes python lib,

tern

Check the component licenses within the ONAP dockers

bash script

kubectl


Example of a scorecard:


 


                                                   

The outcome includes the possibility to accept somes failures and to increase the outcome (main 100%).




Kubescape is the first open-source tool for testing if Kubernetes is deployed securely according to multiple frameworks: regulatory, customized company policies and DevSecOps best practices, such as the NSA-CISA and the MITRE ATT&CK® .
Kubescape scans K8s clusters, YAML files, and HELM charts, and detect misconfigurations and software vulnerabilities at early stages of the CI/CD pipeline and provides a risk score instantly and risk trends over time. Kubescape integrates natively with other DevOps tools, including Jenkins, CircleCI and Github workflows.

https://github.com/armosec/kubescape


Control name and description:

1) Allow privilege escalation:
Attackers may gain access to a container and uplift its privilege to enable excessive capabilities.

2)Allowed hostPath:
Mounting host directory to the container can be abused to get access to sensitive data and gain persistence on the host machine.

3) Applications credentials in configuration files:
Attackers who have access to configuration files can steal the stored secrets and use them. Checks if ConfigMaps or pods have sensitive information in configuration.

4) Automatic mapping of service account:
Potential attacker may gain access to a POD and steal its service account token. Therefore, it is recommended to disable automatic mapping of the service account tokens in service account configuration and enable it only for PODs that need to use them. 

5) CVE-2021-25741 - Using symlink for arbitrary host file system access:
A user may be able to create a container with subPath volume mounts to access files & directories outside of the volume, including on the host filesystem.
This was affected at the following versions: v1.22.0 - v1.22.1, v1.21.0 - v1.21.4, v1.20.0 - v1.20.10, version v1.19.14 and lower.

6) Cluster-admin binding:
Attackers who have Cluster-admin permissions (can perform any action on any resource), can take advantage of their high privileges for malicious intentions. Determines which subjects have cluster admin permissions.

7) Container hostPort:
Configuring hostPort limits you to a particular port, and if any two workloads that specify the same HostPort cannot be deployed to the same node. And if the scale of your workload is larger than the number of nodes in your Kubernetes cluster, the deployment fails.

8) Control plane hardening:
Kubernetes control plane API is running with non-secure port enabled which allows attackers to gain unprotected access to the cluster.

9) Dangerous capabilities:
Giving dangerous and unnecessary capabilities for a container can increase the impact of a container compromise.

10) Exec into container:
Attackers who have permissions, can run malicious commands in containers in the cluster using exec command (“kubectl exec”). Determines which subjects have permissions to exec into containers. 

11) Exposed dashboard:
If Kubernetes dashboard is exposed externally in Dashboard versions before 2.01, it will allow unauthenticated remote management of the cluster.

12) Host PID/IPC privileges: 
Containers should be as isolated as possible from the host machine. The hostPID and hostIPC fields in Kubernetes may excessively expose the host for potentially malicious actions. 

13) Immutable container files:
Mutable container filesystem can be abused to gain malicious code and data injection into containers. Use immutable (read-only) filesystem to limit potential attacks.
Deployment - nginx-deployment
Remediation: Set the filesystem of the container to read-only when possible. If the containers application needs to write into the filesystem, it is possible to mount secondary filesystems for specific directories where application require write access. 

14) Ingress and Egress blocked:
By default, you should disable Ingress and Egress traffic on all pods.
Deployment - nginx-deployment
Remediation: Define a network policy that restricts ingress and egress connections. 

15) Insecure capabilities:
Giving insecure and unnecessary capabilities for a container can increase the impact of a container compromise. 

16) Linux hardening: 
Often, containers are given more privileges than actually needed. This behavior can increase the impact of a container compromise. 
Deployment - nginx-deployment
Remediation: Make sure you define at least one linux security hardening property out of AppArmor, Seccomp, SELinux or Capabilities.

17) Network policies: If no network policy is defined, attackers who gain access to a single container may use it to probe the network. Lists namespaces in which no network policies are defined. 

18) Non-root containers:
Potential attackers may gain access to a container and leverage its privileges to conduct an attack. Hence it is not recommended to deploy containers with root privileges unless it is absolutely necessary. 

19) Privileged container:
Potential attackers may gain access to privileged containers and inherit access to the host resources. Therefore, it is not recommended to deploy privileged containers unless it is absolutely necessary. 

20) Resource policies:
CPU and memory resources should have a limit set for every container to prevent resource exhaustion.

21) hostNetwork access:
Potential attackers may gain access to a POD and inherit access to the entire host network. For example, in AWS case, they will have access to the entire VPC.



Example of a scorecard:

+-----------------------------------------------------------------------+------------------+-------------------+---------------+-----------+
|                             CONTROL NAME                              | FAILED RESOURCES | WARNING RESOURCES | ALL RESOURCES | % SUCCESS |
+-----------------------------------------------------------------------+------------------+-------------------+---------------+-----------+
| Allow privilege escalation                                            | 0                | 0                 | 1             | 100%      |
| Allowed hostPath                                                      | 0                | 0                 | 1             | 100%      |
| Applications credentials in configuration files                       | 0                | 0                 | 1             | 100%      |
| Automatic mapping of service account                                  | 0                | 0                 | 0             | NaN       |
| CVE-2021-25741 - Using symlink for arbitrary host file system access. | 0                | 0                 | 1             | 100%      |
| Cluster-admin binding                                                 | 0                | 0                 | 0             | NaN       |
| Container hostPort                                                    | 0                | 0                 | 1             | 100%      |
| Control plane hardening                                               | 0                | 0                 | 1             | 100%      |
| Dangerous capabilities                                                | 0                | 0                 | 1             | 100%      |
| Exec into container                                                   | 0                | 0                 | 0             | NaN       |
| Exposed dashboard                                                     | 0                | 0                 | 1             | 100%      |
| Host PID/IPC privileges                                               | 0                | 0                 | 1             | 100%      |
| Immutable container filesystem                                        | 1                | 0                 | 1             | 0%        |
| Ingress and Egress blocked                                            | 1                | 0                 | 1             | 0%        |
| Insecure capabilities                                                 | 0                | 0                 | 1             | 100%      |
| Linux hardening                                                       | 1                | 0                 | 1             | 0%        |
| Network policies                                                      | 0                | 0                 | 0             | NaN       |
| Non-root containers                                                   | 0                | 0                 | 1             | 100%      |
| Privileged container                                                  | 0                | 0                 | 1             | 100%      |
| Resource policies                                                     | 0                | 0                 | 1             | 100%      |
| hostNetwork access                                                    | 0                | 0                 | 1             | 100%      |
+-----------------------------------------------------------------------+------------------+-------------------+---------------+-----------+
|                                  21                                   |        3         |         0         |      17       |    82%    |
+-----------------------------------------------------------------------+------------------+-------------------+---------------+-----------+


  • No labels

2 Comments

  1. First tests executed on master daily (kubernetes 1.22.2 - jakarta target) with last version of kubescape
     
    root pods
    ********
    test bash
    https://logs.onap.org/onap-integration/daily/onap_daily_pod4_master/2021-12/17_02-30/security/root_pods/root_pods/root_pods.log

    kubescape shoudl have detected several pods under "waiver" in our bash scripts

    • aaf
    • dcae-cloudify # DCAEGEN2-2424
    • cassandra # OOM-2552
    • awx # used for use cases
    • netbox # used for use cases
    • multicloud-fcaps # rabbit-mq
    • oof-has-etcd # etcd
    • multicloud-k8s-etcd # etcd
    • robot # use for test cases + refactoring planned in Istanbul INT-1716


    but we got a 100%
     
    kubescape
    |                             CONTROL NAME                              | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % SUCCESS |
    +-----------------------------------------------------------------------+------------------+--------------------+---------------+-----------+

    | Non-root containers |        0         |         0          |      186      |   100%    |

    On the 186 listed objects => no pods..I do not knwo what is really processed....
    I performed the test with an export in the nice GUI => same behavior 
     
    limits
    *****
    test bash
    no tests on limits found in the different suite (NSA, Mitre, Amrosecurity)


    short sync with armo planned on the  

  2. Sum up after armo meeting

    The business model is more clear now:
     - Free for the problem finding
     - paid for the remediation

    kubescape (https://portal.armo.cloud/)  have a saas UI very complet but nothing for on-premise (only CLI) .

    We find all informations regarding the:

     - Calculate Kubernetes risk score instantly and see risk trends overtime
     - Multiple Frameworks in one click - NSA, MITRE, K8s Best Practices, or create your own custom one
     - Scan Code & Clusters - Scan running cluster or YAMLs and HELM charts in your IaC

    whatever, in ower case, kubescape didn't find issues because kubescape analyse the yaml files to find the problem. (need a point with Morgan Richomme to better understand the ONAP deployment. 

    We stay in touch with armo for any questions or product improvements.