1
0
-1

we are getting following error from ONAP mariadb-galera pod :

ubuntu@ip-172-31-44-93:~$ kubectl describe po -n onap dev-mariadb-galera-0

Name:         dev-mariadb-galera-0

Namespace:    onap

Priority:     0

Node:         onap-k8s-8/172.31.39.204

Start Time:   Tue, 21 Sep 2021 03:04:16 +0000

Labels:       app.kubernetes.io/instance=dev

              app.kubernetes.io/managed-by=Helm

              app.kubernetes.io/name=mariadb-galera

              controller-revision-hash=dev-mariadb-galera-85d6669ff9

              helm.sh/chart=mariadb-galera-8.0.1

              statefulset.kubernetes.io/pod-name=dev-mariadb-galera-0

Annotations:  cni.projectcalico.org/podIP: 10.42.0.165/32

              cni.projectcalico.org/podIPs: 10.42.0.165/32

Status:       Running

IP:           10.42.0.165

IPs:

  IP:           10.42.0.165

Controlled By:  StatefulSet/dev-mariadb-galera

Init Containers:

  fix-permission:

    Container ID:  docker://a1d5231d43e1f62d1b255199eb1443ad8ff27f9ac094f492087690888110dc46

    Image:         docker.io/busybox:1.32

    Image ID:      docker-pullable://busybox@sha256:ae39a6f5c07297d7ab64dbd4f82c77c874cc6a94cea29fdec309d0992574b4f7

    Port:          <none>

    Host Port:     <none>

    Command:

      /bin/sh

    Args:

      -c

      chown -R 10001:10001 /data

      chown -R 10001:10001 /bootstrap/

      chown -R 10001:10001 /tmp/

      cp /config/my.cnf /actual/my.cnf

      chown -R 10001:10001 /actual


    State:          Terminated

      Reason:       Completed

      Exit Code:    0

      Started:      Tue, 21 Sep 2021 03:04:19 +0000

      Finished:     Tue, 21 Sep 2021 03:04:19 +0000

    Ready:          True

    Restart Count:  0

    Environment:    <none>

    Mounts:

      /actual from mariadb-galera-actual-config (rw)

      /bootstrap from previous-boot (rw)

      /config/my.cnf from mariadb-galera-starting-config (rw,path="my.cnf")

      /data from dev-mariadb-galera (rw)

      /tmp from mariadb-tmp-folder (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from dev-mariadb-galera-read-token-76z5f (ro)

Containers:

  mariadb-galera:

    Container ID:  docker://182ac891f24c002b09d2ffe706abe393da5a91f714dbc2f95759122033929374

    Image:         docker.io/bitnami/mariadb-galera:10.5.8

    Image ID:      docker-pullable://bitnami/mariadb-galera@sha256:b4474a41812f284dfdf3b85cc7da35985bd493043859a8c96b5ebb45a8704d75

    Ports:         3306/TCP, 4567/TCP, 4568/TCP, 4444/TCP

    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP

    Command:

      bash

      -ec

      exec /opt/bitnami/scripts/mariadb-galera/entrypoint.sh /opt/bitnami/scripts/mariadb-galera/run.sh


    State:          Running

      Started:      Tue, 21 Sep 2021 03:36:18 +0000

    Last State:     Terminated

      Reason:       Completed

      Exit Code:    0

      Started:      Tue, 21 Sep 2021 03:33:24 +0000

      Finished:     Tue, 21 Sep 2021 03:36:15 +0000

    Ready:          False

    Restart Count:  9

    Liveness:       exec [bash -ec exec mysqladmin status -u$MARIADB_ROOT_USER -p$MARIADB_ROOT_PASSWORD

] delay=150s timeout=1s period=10s #success=1 #failure=3

    Readiness:  exec [bash -ec exec mysqladmin status -u$MARIADB_ROOT_USER -p$MARIADB_ROOT_PASSWORD

] delay=60s timeout=1s period=10s #success=1 #failure=3

    Environment:

      MY_POD_NAME:                          dev-mariadb-galera-0 (v1:metadata.name)

      BITNAMI_DEBUG:                        true

      MARIADB_INIT_SLEEP_TIME:              5

      MARIADB_GALERA_CLUSTER_NAME:          galera

      MARIADB_GALERA_CLUSTER_ADDRESS:       gcomm://mariadb-galera-headless.onap.svc.cluster.local

      MARIADB_GALERA_NODE_ADDRESS:           (v1:status.podIP)

      MARIADB_ROOT_USER:                    root

      MARIADB_ROOT_PASSWORD:                <set to the key 'password' in secret 'dev-mariadb-galera-db-root-password'>     Optional: false

      MARIADB_USER:                         <set to the key 'login' in secret 'dev-mariadb-galera-db-user-credentials'>     Optional: false

      MARIADB_PASSWORD:                     <set to the key 'password' in secret 'dev-mariadb-galera-db-user-credentials'>  Optional: false

      MARIADB_DATABASE:

      MARIADB_GALERA_MARIABACKUP_USER:      <set to the key 'login' in secret 'dev-mariadb-galera-db-backup-credentials'>     Optional: false

      MARIADB_GALERA_MARIABACKUP_PASSWORD:  <set to the key 'password' in secret 'dev-mariadb-galera-db-backup-credentials'>  Optional: false

    Mounts:

      /bitnami/mariadb from dev-mariadb-galera (rw)

      /opt/bitnami/mariadb/.bootstrap from previous-boot (rw)

      /opt/bitnami/mariadb/conf from mariadb-galera-actual-config (rw)

      /opt/bitnami/mariadb/tmp from mariadb-tmp-folder (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from dev-mariadb-galera-read-token-76z5f (ro)

  mariadb-galera-metrics:

    Container ID:  docker://e4441f9050db61b7a9f63c8520ccfebb67c0203e28814930b6685d56da563143

    Image:         docker.io/bitnami/mysqld-exporter:0.12.1-debian-10-r264

    Image ID:      docker-pullable://bitnami/mysqld-exporter@sha256:d69cacee774e5a5538fdb44d869a7c6fa077d109c2bea0c4c2441eb4de7b33ea

    Port:          9104/TCP

    Host Port:     0/TCP

    Command:

      sh

      -c

      DATA_SOURCE_NAME="$MARIADB_ROOT_USER:$MARIADB_ROOT_PASSWORD@(localhost:3306)/" /bin/mysqld_exporter $MARIADB_METRICS_EXTRA_FLAGS


    State:          Running

      Started:      Tue, 21 Sep 2021 03:04:24 +0000

    Ready:          True

    Restart Count:  0

    Limits:

      cpu:     500m

      memory:  256Mi

    Requests:

      cpu:      500m

      memory:   256Mi

    Liveness:   http-get http://:metrics/metrics delay=30s timeout=5s period=10s #success=1 #failure=3

    Readiness:  http-get http://:metrics/metrics delay=5s timeout=1s period=10s #success=1 #failure=3

    Environment:

      MARIADB_METRICS_EXTRA_FLAGS:

      MARIADB_ROOT_USER:            root

      MARIADB_ROOT_PASSWORD:        <set to the key 'password' in secret 'dev-mariadb-galera-db-root-password'>  Optional: false

    Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from dev-mariadb-galera-read-token-76z5f (ro)

Conditions:

  Type              Status

  Initialized       True

  Ready             False

  ContainersReady   False

  PodScheduled      True

Volumes:

  dev-mariadb-galera:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  dev-mariadb-galera-dev-mariadb-galera-0

    ReadOnly:   false

  previous-boot:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  mariadb-tmp-folder:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  mariadb-galera-actual-config:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  mariadb-galera-starting-config:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      dev-mariadb-galera-configuration

    Optional:  false

  dev-mariadb-galera-read-token-76z5f:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  dev-mariadb-galera-read-token-76z5f

    Optional:    false

QoS Class:       Burstable

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

  Type     Reason     Age                From               Message

  ----     ------     ----               ----               -------

  Normal   Scheduled  33m                default-scheduler  Successfully assigned onap/dev-mariadb-galera-0 to onap-k8s-8

  Normal   Pulling    33m                kubelet            Pulling image "docker.io/busybox:1.32"

  Normal   Created    33m                kubelet            Created container fix-permission

  Normal   Started    33m                kubelet            Started container fix-permission

  Normal   Pulled     33m                kubelet            Successfully pulled image "docker.io/busybox:1.32" in 2.444811183s

  Normal   Pulling    33m                kubelet            Pulling image "docker.io/bitnami/mariadb-galera:10.5.8"

  Normal   Pulling    33m                kubelet            Pulling image "docker.io/bitnami/mysqld-exporter:0.12.1-debian-10-r264"

  Normal   Pulled     33m                kubelet            Successfully pulled image "docker.io/bitnami/mariadb-galera:10.5.8" in 1.881453075s

  Normal   Created    33m                kubelet            Created container mariadb-galera

  Normal   Started    33m                kubelet            Started container mariadb-galera

  Normal   Pulled     33m                kubelet            Successfully pulled image "docker.io/bitnami/mysqld-exporter:0.12.1-debian-10-r264" in 1.80603054s

  Normal   Created    33m                kubelet            Created container mariadb-galera-metrics

  Normal   Started    33m                kubelet            Started container mariadb-galera-metrics

  Warning  Unhealthy  30m (x2 over 31m)  kubelet            Liveness probe failed: mysqladmin: connect to server at 'localhost' failed

error: 'Access denied for user 'root'@'localhost' (using password: YES)'

  Normal   Pulled     18m                    kubelet  Successfully pulled image "docker.io/bitnami/mariadb-galera:10.5.8" in 1.806930911s

  Warning  BackOff    8m40s (x7 over 9m43s)  kubelet  Back-off restarting failed container

  Warning  Unhealthy  3m32s (x89 over 32m)   kubelet  Readiness probe failed: mysqladmin: connect to server at 'localhost' failed

error: 'Access denied for user 'root'@'localhost' (using password: YES)'

ubuntu@ip-172-31-44-93:~$

ubuntu@ip-172-31-44-93:~$ kubectl logs -n onap -c mariadb-galera-metrics dev-mariadb-galera-0

time="2021-09-21T03:04:24Z" level=info msg="Starting mysqld_exporter (version=0.12.1, branch=HEAD, revision=48667bf7c3b438b5e93b259f3d17b70a7c9aff96)" source="mysqld_exporter.go:257"

time="2021-09-21T03:04:24Z" level=info msg="Build context (go=go1.12.7, user=root@0b3e56a7bc0a, date=20190729-12:35:58)" source="mysqld_exporter.go:258"

time="2021-09-21T03:04:24Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:269"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.info_schema.innodb_cmp" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.info_schema.innodb_cmpmem" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:273"

time="2021-09-21T03:04:24Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:283"

time="2021-09-21T03:04:33Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:04:43Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:04:53Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:04:57Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:03Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:07Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:13Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:17Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:23Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:27Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:33Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:37Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

time="2021-09-21T03:05:43Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: YES)" source="exporter.go:146"

    CommentAdd your comment...