Page History

...

#!/bin/sh
/opt/app/policy/bin/prepare_upgrade.sh ${SQL_DB}
/opt/app/policy/bin/db-migrator -s ${SQL_DB} -o upgrade
rc=$?
/opt/app/policy/bin/db-migrator -s ${SQL_DB} -o report
exit $rc

Update kubernetes/policy/templates/configmap.yaml (configmap.yaml)

...

State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
APP_FQI: policy@policy.onap.org
aaf_locate_url: https://aaf-locate.onap:8095
aaf_locator_container: oom
aaf_locator_container_ns: onap
aaf_locator_fqdn: policy
aaf_locator_app_ns: org.osaaf.aaf
DEPLOY_FQI: <set to the key 'login' in secret 'dev-policy-apex-pdp-cert-initializer-deployer-creds'> Optional: false
DEPLOY_PASSWORD: <set to the key 'password' in secret 'dev-policy-apex-pdp-cert-initializer-deployer-creds'> Optional: false
cadi_longitude: 0.0
cadi_latitude: 0.0
aaf_locator_public_fqdn: policy.onap.org
Mounts:
/opt/app/aaf_config/bin/aaf-add-config.sh from aaf-add-config (rw,path="aaf-add-config.sh")
/opt/app/aaf_config/bin/retrieval_check.sh from aaf-add-config (rw,path="retrieval_check.sh")
/opt/app/aaf_config/cert/truststoreONAP.p12.b64 from aaf-agent-certs (rw,path="truststoreONAP.p12.b64")
/opt/app/aaf_config/cert/truststoreONAPall.jks.b64 from aaf-agent-certs (rw,path="truststoreONAPall.jks.b64")
/opt/app/osaaf from dev-policy-apex-pdp-aaf-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from dev-policy-apex-pdp-read-token-x829k (ro)
Containers:
policy-apex-pdp:
Container ID:
Image: nexus3.onap.org:10001/onap/policy-apex-pdp:2.6-SNAPSHOT-latest
Image ID:
Port: 6969/TCP
Host Port: 0/TCP
Command:
sh
-c
Args:
if [ -f /opt/app/osaaf/local/.ci ]; then . /opt/app/osaaf/local/.ci; fi;/opt/app/policy/apex-pdp/bin/apexOnapPf.sh -c /home/apexuser/config/OnapPfConfig.json
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: tcp-socket :6969 delay=20s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :6969 delay=20s timeout=1s period=10s #success=1 #failure=3
Environment:
REPLICAS: 1
Mounts:
/etc/localtime from localtime (ro)
/home/apexuser/config from apexconfig (rw)
/opt/app/osaaf from dev-policy-apex-pdp-aaf-config (rw)
/var/log/onap from policy-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from dev-policy-apex-pdp-read-token-x829k (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
dev-policy-apex-pdp-aaf-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
aaf-agent-certs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-cert-wrapper-certs
Optional: false
aaf-add-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-policy-apex-pdp-cert-initializer-add-config
Optional: false
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
policy-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
apexconfig-input:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-policy-apex-pdp-configmap
Optional: false
apexconfig:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
dev-policy-apex-pdp-read-token-x829k:
Type: Secret (a volume populated by a Secret)
SecretName: dev-policy-apex-pdp-read-token-x829k
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned onap/dev-policy-apex-pdp-0 to esy-master-policy-002-k8s-02
Warning FailedMount 35m (x2 over 35m) kubelet MountVolume.SetUp failed for volume "apexconfig-input" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 35m (x3 over 35m) kubelet MountVolume.SetUp failed for volume "aaf-add-config" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 35m (x3 over 35m) kubelet MountVolume.SetUp failed for volume "dev-policy-apex-pdp-read-token-x829k" : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 35m kubelet Container image "docker.io/dibi/envsubst:1" already present on machine
Normal Created 35m kubelet Created container policy-apex-pdp-update-config
Normal Started 35m kubelet Started container policy-apex-pdp-update-config
Normal Started 33m (x3 over 35m) kubelet Started container policy-apex-pdp-cert-initializer-readiness
Normal Pulled 32m (x4 over 35m) kubelet Container image "nexus3.onap.org:10001/onap/oom/readiness:3.0.1" already present on machine
Normal Created 32m (x4 over 35m) kubelet Created container policy-apex-pdp-cert-initializer-readiness
Warning BackOff 37s (x123 over 33m) kubelet Back-off restarting failed container

Exit Code

State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137

 kubectl describe pods dev-policy-api-78976fbd5d-4xfx2 | grep -i exit
Exit Code: 0
Exit Code: 0
Exit Code: 137

...

The Exit code 137 is important because it means that the system terminated the container as it tried to use more memory than its limit. In order to monitor this, you always have to look at the use of memory compared to the limit.

State: Running
Started: Thu, 26 Aug 2021 15:10:16 +0000
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 26 Aug 2021 15:08:03 +0000
Finished: Thu, 26 Aug 2021 15:08:50 +0000
Ready: False
Restart Count: 5
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 3m
memory: 20Mi