General info from: https://www.golinuxcloud.com/openssl-create-certificate-chain-linux/

Create certificates

Create directory structure

Example under /home/<user> (can be changed)

mkdir -p ~/myCA/rootCA/{certs,crl,newcerts,private,csr}
mkdir -p ~/myCA/intermediateCA/{certs,crl,newcerts,private,csr}
echo 1000 > ~/myCA/rootCA/serial
echo 1000 > ~/myCA/intermediateCA/serial
echo 0100 > ~/myCA/rootCA/crlnumber
echo 0100 > ~/myCA/intermediateCA/crlnumber
touch ~/myCA/rootCA/index.txt
touch ~/myCA/intermediateCA/index.txt

Create config files

Create openssl_root.cnf (use the complete directory as <base-dir> in "dir")

[ ca ]                                                   # The default CA section
default_ca = CA_default                                  # The default CA name

[ CA_default ]                                           # Default settings for the CA
dir               = /<base-dir>/myCA/rootCA                        # CA directory
certs             = $dir/certs                           # Certificates directory
crl_dir           = $dir/crl                             # CRL directory
new_certs_dir     = $dir/newcerts                        # New certificates directory
database          = $dir/index.txt                       # Certificate index file
serial            = $dir/serial                          # Serial number file
RANDFILE          = $dir/private/.rand                   # Random number file
private_key       = $dir/private/ca.key.pem              # Root CA private key
certificate       = $dir/certs/ca.cert.pem               # Root CA certificate
crl               = $dir/crl/ca.crl.pem                  # Root CA CRL
crlnumber         = $dir/crlnumber                       # Root CA CRL number
crl_extensions    = crl_ext                              # CRL extensions
default_crl_days  = 30                                   # Default CRL validity days
default_md        = sha256                               # Default message digest
preserve          = no                                   # Preserve existing extensions
email_in_dn       = no                                   # Exclude email from the DN
name_opt          = ca_default                           # Formatting options for names
cert_opt          = ca_default                           # Certificate output options
policy            = policy_strict                        # Certificate policy
unique_subject    = no                                   # Allow multiple certs with the same DN

[ policy_strict ]                                        # Policy for stricter validation
countryName             = match                          # Must match the issuer's country
stateOrProvinceName     = optional                       # Must match the issuer's state
organizationName        = match                          # Must match the issuer's organization
organizationalUnitName  = optional                       # Organizational unit is optional
commonName              = supplied                       # Must provide a common name
emailAddress            = optional                       # Email address is optional

[ req ]                                                  # Request settings
default_bits        = 2048                               # Default key size
distinguished_name  = req_distinguished_name             # Default DN template
string_mask         = utf8only                           # UTF-8 encoding
default_md          = sha256                             # Default message digest
prompt              = no                                 # Non-interactive mode

[ req_distinguished_name ]                               # Template for the DN in the CSR
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name (full name)
localityName                    = Locality Name (city)
0.organizationName              = Organization Name (company)
organizationalUnitName          = Organizational Unit Name (section)
commonName                      = Common Name (your domain)
emailAddress                    = Email Address

[ v3_ca ]                                           # Root CA certificate extensions
subjectKeyIdentifier = hash                         # Subject key identifier
authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier
basicConstraints = critical, CA:true                # Basic constraints for a CA
keyUsage = critical, keyCertSign, cRLSign           # Key usage for a CA

[ crl_ext ]                                         # CRL extensions
authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier

[ v3_intermediate_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign


Create openssl_intermediate.cnf (use the complete directory as <base-dir> in "dir")

[ ca ]                           # The default CA section
default_ca = CA_default          # The default CA name

[ CA_default ]                                           # Default settings for the intermediate CA
dir               = /<base-dir>/myCA/intermediateCA                # Intermediate CA directory
certs             = $dir/certs                           # Certificates directory
crl_dir           = $dir/crl                             # CRL directory
new_certs_dir     = $dir/newcerts                        # New certificates directory
database          = $dir/index.txt                       # Certificate index file
serial            = $dir/serial                          # Serial number file
RANDFILE          = $dir/private/.rand                   # Random number file
private_key       = $dir/private/intermediate.key.pem    # Intermediate CA private key
certificate       = $dir/certs/intermediate.cert.pem     # Intermediate CA certificate
crl               = $dir/crl/intermediate.crl.pem        # Intermediate CA CRL
crlnumber         = $dir/crlnumber                       # Intermediate CA CRL number
crl_extensions    = crl_ext                              # CRL extensions
default_crl_days  = 30                                   # Default CRL validity days
default_md        = sha256                               # Default message digest
preserve          = no                                   # Preserve existing extensions
email_in_dn       = no                                   # Exclude email from the DN
name_opt          = ca_default                           # Formatting options for names
cert_opt          = ca_default                           # Certificate output options
policy            = policy_loose                         # Certificate policy

[ policy_loose ]                                         # Policy for less strict validation
countryName             = optional                       # Country is optional
stateOrProvinceName     = optional                       # State or province is optional
localityName            = optional                       # Locality is optional
organizationName        = optional                       # Organization is optional
organizationalUnitName  = optional                       # Organizational unit is optional
commonName              = supplied                       # Must provide a common name
emailAddress            = optional                       # Email address is optional

[ req ]                                                  # Request settings
default_bits        = 2048                               # Default key size
distinguished_name  = req_distinguished_name             # Default DN template
string_mask         = utf8only                           # UTF-8 encoding
default_md          = sha256                             # Default message digest
x509_extensions     = v3_intermediate_ca                 # Extensions for intermediate CA certificate

[ req_distinguished_name ]                               # Template for the DN in the CSR
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

[ v3_intermediate_ca ]                                      # Intermediate CA certificate extensions
subjectKeyIdentifier = hash                                 # Subject key identifier
authorityKeyIdentifier = keyid:always,issuer                # Authority key identifier
basicConstraints = critical, CA:true, pathlen:0             # Basic constraints for a CA
keyUsage = critical, digitalSignature, cRLSign, keyCertSign # Key usage for a CA

[ crl_ext ]                                                 # CRL extensions
authorityKeyIdentifier=keyid:always                         # Authority key identifier

[ server_cert ]                                             # Server certificate extensions
basicConstraints = CA:FALSE                                 # Not a CA certificate
nsCertType = server                                         # Server certificate type
keyUsage = critical, digitalSignature, keyEncipherment      # Key usage for a server cert
extendedKeyUsage = serverAuth                               # Extended key usage for server authentication purposes (e.g., TLS/SSL servers).
authorityKeyIdentifier = keyid,issuer                       # Authority key identifier linking the certificate to the issuer's public key.

Create and check root keypair

openssl genrsa -out ~/myCA/rootCA/private/ca.key.pem 4096
chmod 400 ~/myCA/rootCA/private/ca.key.pem
openssl rsa -noout -text -in ~/myCA/rootCA/private/ca.key.pem

Create and check root certificate

openssl req -config openssl_root.cnf -key ~/myCA/rootCA/private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out ~/myCA/rootCA/certs/ca.cert.pem -subj "/C=US/O=ONAP/OU=aaf@aaf.osaaf.org/OU=OSAAF/CN=aaf-sms"
chmod 444 ~/myCA/rootCA/certs/ca.cert.pem
openssl x509 -noout -text -in ~/myCA/rootCA/certs/ca.cert.pem

Create intermediate CA keypair and certificate request

openssl genrsa -out ~/myCA/intermediateCA/private/intermediate.key.pem 4096
chmod 400 ~/myCA/intermediateCA/private/intermediate.key.pem
openssl req -config openssl_intermediate.cnf -key ~/myCA/intermediateCA/private/intermediate.key.pem -new -sha256 -out ~/myCA/intermediateCA/certs/intermediate.csr.pem -subj "/C=US/O=ONAP/OU=OSAAF/CN=intermediateCA_9"

Sign intermediate CSR

openssl ca -config openssl_root.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in ~/myCA/intermediateCA/certs/intermediate.csr.pem -out ~/myCA/intermediateCA/certs/intermediate.cert.pem
chmod 444 ~/myCA/intermediateCA/certs/intermediate.cert.pem
cat ~/myCA/rootCA/index.txt
openssl x509 -noout -text -in ~/myCA/intermediateCA/certs/intermediate.cert.pem

Create certificate chain

cat ~/myCA/intermediateCA/certs/intermediate.cert.pem ~/myCA/rootCA/certs/ca.cert.pem > ~/myCA/intermediateCA/certs/ca-chain.cert.pem
openssl verify -CAfile ~/myCA/intermediateCA/certs/ca-chain.cert.pem ~/myCA/intermediateCA/certs/intermediate.cert.pem


Create files for OOM

Create ca-chain file for AAF-SMS:

cp ~/myCA/intermediateCA/certs/ca-chain.cert.pem ~/myCA/intermediate_root_ca.pem

File will be stored in https://git.onap.org/oom/tree/kubernetes/aaf/components/aaf-sms/resources/certs?h=kohn

Import CA-chain to cert-wrapper


cp ~/myCA/cacerts ~/myCA/truststoreONAPall.jks
keytool -import -alias onaptestca -keystore ~/myCA/truststoreONAPall.jks -file ~/myCA/intermediate_root_ca.pem -storepass changeit
keytool -list -keystore ~/myCA/truststoreONAPall.jks-old|grep onap
  • base64 encrypt file
base64 ~/myCA/truststoreONAPall.jks >~/myCA/truststoreONAPall.jks.b64
  • No labels

14 Comments

  1. Hans Torres

    Hello,

    Team, at the moment I am in the process of updating the expired certificate for the AAF module. I have followed all the steps previously, but the module still hasn't picked up the change to the new certificate. After deploying and attempting to upload the module, it still uses the previous certificate.

    The environment is deployed on Kubernetes AKS and managed through Helm.

    Thank you.

  2. Andreas Geissler

    Yes, same in my environment.
    I will now try the instructions in Bootstrapping AAF Components#DataDefinitions to create a new CA/intermediateCA...



  3. gamerslouis@gmail.com

    Update:

    This patch can work. Just need to change `openssl x509 -in certs/ca.crt -out certs/AAF_RootCA.cer -outform DER` to cp certs/ca.crt certs/AAF_RootCA.cer 

    If someone use dcae service, you need to rebuild the image nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container

    # create empty folder
# copy new truststoreONAPall.jks.b64 and truststoreONAP.p12.b64 to this folder
cat << EOF > Dockerfile
from nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:2.1.0
COPY . /opt/app/aaf_config/cert/
EOF
docker build -t nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:2.1.0


    ---

    I find that AAF components sign application certificates in initialize container with image aaf_config. The intermediate certificate "intermediateCA_9" which is the expired certificate is at  /opt/app/aaf_config/cert/demoONAPsigner.p12.b64 in the aaf_config. 

    This file is "not" overrided by onap_oom charts so the above steps will not work. 


    So I try to create new CA/intermediateCA with follow commands in an aaf_config. To ensure that all parameters such as issuer and subject names are all the same as original aaf certs.

    Then copy new certs to onap_oom and build another new aaf_config.

    docker run -it --name certs_builder nexus3.onap.org:10001/onap/aaf/aaf_config:2.1.23 sh
cd /opt/app/aaf_config/CA/
echo '8' > intermediate.serial
sed -i "/^#.*openssl/s/^#//" truststore.sh
bash newCA.sh // changeit
bash truststore.sh // changeit changeit changeit changeit
bash newIntermediate.sh // something easy / y / y
cd intermediate_9 
bash signerP12.sh // something easy / something easy / something easy
openssl x509 -in certs/ca.crt -out certs/intermediate_root_ca.pem -outform PEM 
base64 aaf_intermediate_9.p12 > demoONAPsigner.p12.b64
## replcae onap_oom kubernetes/aaf/components/aaf-sms/resources/certs/intermediate_root_ca.pem with  certs/intermediate_root_ca.pem
cd ..

openssl x509 -in certs/ca.crt -out certs/AAF_RootCA.cer -outform DER
openssl x509 -in certs/ca.crt -out certs/AAF_RootCA.pem -outform PEM
base64 truststore.p12 > truststoreONAP.p12.b64
## replace onap_oom kubernetes/common/cert-wrapper/resources/truststoreONAP.p12.b64 with truststoreONAP.p12.b64

# run commands below in onap_oom
base64 -d kubernetes/common/cert-wrapper/resources/truststoreONAPall.jks.b64 > truststoreONAPall.jks
keytool -delete -alias onaptestca -keystore truststoreONAPall.jks -storepass changeit
# cp intermediate_root_ca.pem from aaf_config to here
keytool -import -alias onaptestca -keystore truststoreONAPall.jks -file intermediate_root_ca.pem -storepass changeit // yes
base64 truststoreONAPall.jks > kubernetes/common/cert-wrapper/resources/truststoreONAPall.jks.b64

# rebuild onap oom charts
<skip>


# build new aaf_config image 
mkdir new_aaf_config
cd new_aaf_config
docker cp certs_builder:/opt/app/aaf_config/CA/certs/AAF_RootCA.cer .
docker cp certs_builder:/opt/app/aaf_config/CA/intermediate_9/demoONAPsigner.p12.b64 .
docker cp certs_builder:/opt/app/aaf_config/CA/truststoreONAP.p12.b64 .
cp ../kubernetes/common/cert-wrapper/resources/truststoreONAPall.jks.b64 .
cat << EOF > Dockerfile
from nexus3.onap.org:10001/onap/aaf/aaf_config:2.1.23
COPY . /opt/app/aaf_config/cert/
EOF
# This image is poblish on dockerhub
docker build -t gamerslouis/aaf_config:2.1.23 .

    I try to use this new aaf_config and oom resource to deploy onap.

    aaf_sms init container aaf-sms-aaf-config can successfully access aaf_locate, which means aaf modules successfully use new certs

    However, aaf-sms-aaf-config still failed with new Errors (see error log in attachment). And I have no idea what's wrong now.

    Hope this can be of some help.

    The aaf-patch.tar includes all new certs and keyfiles I used


    additional information:

      truststoreONAPall.jks contains root ca rather than intermediate ca.

      For root ca, the p12 password is changeit (same as aaf now)

      For intermediate ca, the p12 password is "something easy" (same as aaf now)

      The expired intermediate ca is intermediate_9. So I use the same serial number 9 to prevant new errors  


    aaf-patch.tar

    1. dajian zhang

      hi, gamerslouis,  I follow your workflow to  encountered the same issue, and after debugging the program, I found that I needed to replace the contents of 'AAF_RootCA.cer' with 'ca.crt,' and that solved the problem."  good luck for you!

      1. gamerslouis@gmail.com

        Thanks, dajian zhang 

        Your solution solved the problem in my environment.

        Hovever, pod such as dcae-* use truststoreONAPall.jks.b64 include in the init container tls-init with image nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:2.1.0. And this init-container does not use cert-wrapper.

        So we need to build this image or modify oom templates


  4. gamerslouis@gmail.com

    Here is a unofficial quick patch to temporarily fix deployment with oom and single node Kubernetes

    Use the new Root CA cert signed by me

    docker pull gamerslouis/aaf_config:2.1.23
docker tag gamerslouis/aaf_config:2.1.23 nexus3.onap.org:10001/onap/aaf/aaf_config:2.1.23
docker pull gamerslouis/org.onap.dcaegen2.deployments.tls-init-container:2.1.0
docker tag gamerslouis/org.onap.dcaegen2.deployments.tls-init-container:2.1.0 nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:2.1.0
docker run --name certs -d gamerslouis/aaf_config:2.1.23 sleep infinity
docker cp certs:/opt/app/aaf_config/cert/truststoreONAP.p12.b64 kubernetes/common/cert-wrapper/resources/truststoreONAP.p12.b64
docker cp certs:/opt/app/aaf_config/cert/truststoreONAPall.jks.b64 kubernetes/common/cert-wrapper/resources/truststoreONAPall.jks.b64
docker cp certs:/opt/app/aaf_config/cert/intermediate_root_ca.pem kubernetes/aaf/components/aaf-sms/resources/certs/intermediate_root_ca.pem

# rebuild oom charts
cd kubernetes 
make all -e SKIP_LINT=TRUE

  5. dajian zhang

    Thanks gamerslouis@gmail.com

    I haven't use dcaegen2 module.   we put the new truststoreONAP.p12.b64 and   truststoreONAPall.jks.b64 to 

    kubernetes/common/cert-wrapper/resources/ dir as your mentioned. and renew the aai_keystore file with
    intermediate_root_ca.pem and key
    for aai-schema-service.  Finally, to rebuild oom charts.

  6. Michal

    i've used mentioned instruction to generate new cert via openssl x509 cmds and copied

    openssl genrsa -out ~/myCA/intermediateCA/private/intermediate.key.pem 4096
chmod 400 ~/myCA/intermediateCA/private/intermediate.key.pem
openssl req -config openssl_intermediate.cnf -key ~/myCA/intermediateCA/private/intermediate.key.pem -new -sha256 -out ~/myCA/intermediateCA/certs/intermediate.csr.pem -subj "/C=US/O=ONAP/OU=OSAAF/CN=intermediateCA_9"

    cp *  aaf/components/aaf-sms/resources/certs/

    but still have issues with cert

    1. dajian zhang

      hi  mwlszk,  i suggest you follow 

      gamerslouis@gmail.com steps.  should be ok!

  7. Andreas Geissler

    I think I found a solution without patching the original aaf_config image (only via OOM).
    The following files need to be patched in the aaf_config after your investigations (thanks a lot !!!):

    • /opt/app/aaf_config/cert/truststoreONAPall.jks.b64 (exists in CM onap-cert-wrapper-certs)
    • /opt/app/aaf_config/cert/truststoreONAP.p12.b64 (exists in CM onap-cert-wrapper-certs)
    • /opt/app/aaf_config/cert/intermediate_root_ca.pem  (exists in Secret onap-aaf-sms-int-certs)
    • /opt/app/aaf_config/cert/demoONAPsigner.p12.b64 (not stored yet in CM/Secret)
    • /opt/app/aaf_config/cert/AAF_RootCA.cer  (not stored yet in CM/Secret)

    When I add the last 2 items in secrets/CMs and mount all files in the aaf_config (https://git.onap.org/oom/tree/kubernetes/aaf/components/aaf-templates/templates/_initContainers.tpl?h=kohn) it will work !!
    I tested it in my environment sucessfully.



  8. Andreas Geissler

    Hi all,
    with my patch (https://gerrit.onap.org/r/c/oom/+/135975) I was able to solve the problem for most of the components.
    The only failing pods are:

    • dmaap-provisioning-job
    • oof-has-controller

    For the rest of the components the fix should work. 

    1. Andreas Geissler

      I have checked i again in my enviroment and found still issues in the aaf-sms pod logs:

      ERROR: 2023/10/02 07:23:56 vault.go:483: Unable to create policy for approle creation
ERROR: 2023/10/02 07:23:56 vault.go:175: Token Check: Unable to initRole in checkToken
ERROR: 2023/10/02 07:23:56 handler.go:124: ListSecretHandler: Token check failed
ERROR: 2023/10/02 07:23:56 auth.go:226: Read from file: open auth/role: no such file or directory
WARNING: 2023/10/02 07:23:56 vault.go:389: Unable to find RoleID. Generating...
ERROR: 2023/10/02 07:23:56 vault.go:405: Creating Policy: Error making API request.
URL: PUT http://aaf-sms-db:8200/v1/sys/policy/smsvaultpolicy
Code: 503. Errors:

* Vault is sealed

      I think, the image of aaf-sms is not updated after the release 4.0.2
      If you look at: https://git.onap.org/aaf/sms/log/
      you can see, that the patch "

      Remove expired certs and key, use them from oom

      was never released...

  9. dajian zhang

    Andreas Geissler  you should rebuild docker image aaf_agent from the old aaf_agent ,  to replace the cert by your new  Root CA cert 

  10. dajian zhang

    Andreas Geissler  The Dockerfile as below for your reference: 

    from nexus3.onap.org:10001/onap/aaf/aaf_agent:2.1.20
    COPY truststoreONAP* /opt/app/aaf_config/cert/