This page will be updated with experiments, steps to reproduce on ways to store private keys and as well as perform sign and decrypt operations within the HW based Trusted Execution Environments.

Some background:

At this time, private keys would be present in two different places

  • At CA service - To store CA private keys and to perform signing operations on X.509V3 certificates.
  • At each ONAP Service or at the sidebar
    • To store private keys used for Mutual TLS communication with other ONAP services.
    • To perform sign operations in case PFS is enabled.
    • To perform decrypt operations in case PFS is disabled.


It is found that  cryptography libraries of different languages have PKCS11 support and in many cases, it is proven with softhsm2.

Hence softhsm2 is chosen for experimentation.

It is also an intention to hook HW-Security plugins under SoftHSM2 (This discussion will be in some other page).

CA Service:

  • Proposal is to use vault project.
    • Vault project is developed in golang and uses go native crypto library.
  • Experiment details:
    • Current state of Open source:
      • Go Crypto has signer and decrypter interface, which is used to develop plugins to accelerate these functions as well as for keeping keys away from the crypto library and associated applications.
      • PKCS11 interface from OASIS is standard to interface with hardware security modules.
      • PKCS11 wrapper for golang is developed and it is available in github: https://github.com/miekg/pkcs11
      • PKCS11 glue to Go Crypto signer and decrypter interface is also developed and it is here in : https://github.com/ThalesIgnite/crypto11
        • This project also has multiple test programs to test RSA, ECDSA and DSA etc..
        • This project also has demo program acting as HTTP Server using Mutual TLS with PKCS11 key label.
        • This project also tested it using nshield physical HSM and also softHSM 2.0 (https://github.com/opendnssec/SoftHSMv2)
    • Output of Experiment :
      • Ensure that all above open source combination work together.
      • Try with RSA2K, 3K, 4K and ECDSA with few popular curves.
      • Document the steps
  • Gap analysis:
    • What needs to be done in Vault project to secure keys and get the crypto operations performed using PKCS11 based HSMs?
    • Document the findings.
    • Create a blueprint to address the gaps.

Java based ONAP Services:

  • Experiment details:
    • Current state of Open Source:
    • Output of experiment:
      • Document steps of two experiments:
        • Creating the private key and public key using Java library to generate key pair (which inturn calls SoftHSM)  and ensuring the Mutual TLS communication works.
        • Creating private key and public key pair outside and importing them into softHSM and using these in Java program while establishing Mutual TLS communication.
      • Try above steps with RSA 2K and ECDSA using few popular curves.
  • Gap analysis:
    • What needs to be done in ONAP Java services?
    • Create guideline document

Python based ONAP Services:

  • Experiment details:
    • Current state of Open source:
      • Python SSL is based out of OpenSSL (That is to be verified). Since OpenSSL already has support to talk to SoftHSM via OpenSC,  current understanding is that SSL context creation can be passed with PKCS11 URI instead of private key file.
      • OpenSSL PKCS11 wrapper library:  https://github.com/OpenSC/libp11
      • Integration with SoftHSM2.0 is done via libp11 PKCS11 library. 
    • Output of experiment
      • Create a sample Python Mutual TLS program to connect to other service using PKCS11 URI instead of private key file (Using above open source)
      • Document steps for different kinds of private keys (RSA, ECDSA).
  • Gap Analysis:
    • Understand current ONAP services written in Python.
    • Create guideline documents for ONAP service developers to follow.

Side car(External Gateway and Internal Gateway of MSB)

  • Experiment details:
    • Background: 
      • Current MSB external gateway is based on OpenResty (https://openresty.org/en/)
      • OpenResty based on NGINX 
      • NGINX is developed in C language and uses OpenSSL.
      • OpenSSL library already has libp11 based PKCS11 wrapper for performing secure crypto operations. (https://github.com/OpenSC/libp11)
      • libp11 is proven with softHSM 2.0
    • Output of experiement:
      • Using NGNIX, OpenSSL, libp11 and SoftHSm 2.0, create Mutual TLS communication.
      • Document the steps to be followed for Ubuntu and CentOS (versions TBD)
  • Gap analysis:
    • Understand current MSB.
    • Create guideline documents for MSB external and internal gateways to adopt this technology.

SoftHSM2 with TPM:

  • SoftHSM2 today only has software based crypto support with private keys stored in file system.
  • What we would be doing:
    • Provide a way to hook HW-security blocks to SoftHSM2.
    • We will start with Private key operations such as Generate key pair,  sign and decrypt operations.
    • Create HW-Security interface in SoftHSM.
    • Create TPM plugin to attach to Hw-Security interface.
    • In future, create other plugins such as SGX plugin and ARM TZ plugin.
    • From design perspective, if HW is not available on the system, it falls back to original SoftHSM2 software security.
    • Experimentation to ensure that multipel container applications can talk one TPM HW via TSS and RM.
    • Experimentation to understand migration of keys among TPMs (Duplicate and Import functionality)
    • Output of this effort:
      • Steps to access TPM from multiple containers by showcasing simultaneous executation of tpm-tools for various operations.
      • Small write up/experiemention on migration of keys from one TPM to another TPM.
      • Create blueprint for HW-Security interface in SoftHSM2.
      • Create blueprint for TPM plguin.

Unsealing of Vault:

As per the Vault documentation on deploying valut (https://www.vaultproject.io/intro/getting-started/deploy.html),  after vault is initialized, it generates encryption key and master key. Master key itself is broken into multiple key shards.  Every time valut service is started, threshold number of key segments are expected to be presented to Vault for unsealing.  At this time, these key segments are expected to be sent from external systems.  

  1. Few security issues: Problem description:  Many times key segments are stored in the same system as Vault.  Whenever vault is restarted,  these key segments are provided as part of unsealing process in a startup script.
    1.  Key segments are clear in file system.  Even if it is encrypted using some password, these are also in the same system. If the disk/file is lost, then it is possible to get hold of secrets from the vault.
    2.  For vault to work in Active-backup mode, these key segements need to be present in backup vault file system too.  Hence, bigger attack surface.
  2. Possible solutions: External systems (could be other containers) providing key segments is indeed needed. Since vault process can be restarted for many reasons such as healing etc.., unsealing should happen with no human intervention.  Hence, the unseal process should not involve humans. External systems should monitor the vault process continuously, when it finds the vault process is restarted and waiting for unsealing, it should call /sys/unseal API with its key segment. It is expected that these external systems only have this service running, nothing else. Also, it is expected that these external systems are firewalled to ensure that only valid connections get into these unseal systems. Also, to ensure that key segments are not stored in the file system in clear, it is suggested that these key segments are placed in local TPM.  Password to access TPM would be in the file system, but not the key segment.  Even if the hard drive is stolen, one can't get the key segments as TPM is not part of the disk. 

Experiment 1:

  • Unseal application:
    • Is a daemon
    • Takes a configuration file that has following:
      • URLs of vault systems (Primary, multiple backups)
      • Key segemnt value (in base64)
      • Monitor timeout : 1 minutes default
    • Constants monitors the Vault systems - Both primary and backup (using /sys/seal-status). 
      • If it finds that the vault is in sealed state, it unseals it by sending the key by calling /sys/unseal API.
  • Check whether unsealing works without any human intervention when the vault service is restarted.


Experiment 2:

  • Same as above, expect that the configuration file will not have key segment itself. Rather, it will have TPM key blob and TPM password.
  • Unseal application is expected to read the key from TPM as part of unsealing.
  • BTW, one can use tpm tools to add key segment to the TPM and store the resulting keyblob in the configuration file.

Results will be published in the child page.

All documents are expected to be posted as child pages to this page.

Private key protection of CA private key using HSM (softHSM)

Vault project provides two different methods to create CA instance (PKI backend). 

  1. Asking Vault to create self-signed CA certificate & private key (pathGenerateRoot in path_root.go)
  2. Uploading root CA and private key (pathCAWrite() function in path_config_ca.go) 

Golang Crypto does not have interface to create key pairs and hence crypto11 does not have any interface to gocrypto to create key pair.

Vault depends on gocrypto.

Both of above methods are needed.

In case of generation of self-signed root certificate, suggestion is to create

  • New API instead of changing the exsiting API function - Call it ias PKCS11Generate.  It can take much of code from pathGenerateRoot, except for generating the key pair.

  • Key pair generation code can be borrowed from crypto11

    • See the code of GenerateRSAKeyPair from rsa.go for RSA keys.

    • See the code of GenerateECDSAKeyPair from ecdsa.go for ECDSA keys.

  • Ensure to follow the rest of the code as in pathGenerateRoot or its nested functions.

in case of uploading root CA and private key :  CA public/private key is generated elsewhere and upload those in CA  

Today, vault project, as part of pathCAWrite does not have ability to take PKCS11 keyLabel/ID to represent private key.  pathCAWrite functon expects both certificate and private key to be given to it. 

In case of PKCS11, there is no private key to be passed, only the PKCS11 keylable/id.  

Suggestion:

  • Create a new URL end point (similar to /config/ca), but with /config/pkcs11ca) - New API function.
  • In this new path, instead of calling parsePEMBundle,  we need to introudce new function parsePKCS11PEMBundle, where private key is not the PEM, only the certificate is put in PEM format.  private key is passed as string (pkcs11_key_label) as one of the JSON paramter.  This parsePKCS11PEMBundle needs to put  the siger interface from the PKCS11.  Some logic that can be used to make signer blob can be borrowed from the crypto11.

    • GenerateRSAKeyPairOnSession in crypto11 package creates the RSA key pair and then generates signer private key.  In our case, key pair is already generated and added to the HSM already.  Hence, instead of generating the key pair, it needs to find the object handle of the private key (based on key label) and prepare the priv signer.   This code is of interest in rsa.go of crypto11:  

      if pub, err = exportRSAPublicKey(session, pubHandle); err != nil {return nil, err |
      priv := PKCS11PrivateKeyRSA{PKCS11PrivateKey{PKCS11Object{privHandle, slot}, pub}}
      return &priv, nil


    • In case of ECDSA, this code from ecdsa.go is of interest:

      return nil, err
      }
      priv := PKCS11PrivateKeyECDSA{PKCS11PrivateKey{PKCS11Object{privHandle, slot}, pub}}
      return &priv, nil
  • My understanding is that  not many changes required during signCert, except for framing the private key signer data.


Creation of user certificates using CSR with Vault:  Some steps I followed are:

  1. Create PKI backend with vault mount -path=srini pki
  2. Create self signed certificate using openssl commands:

    1. openssl genrsa -out myrootca.key 2048 

    2. openssl req -x509 -new -nodes -key myrootca.key -sha256 -days 1024 -out myrootca.pem

  3. Upload the root certificate into the valut 
    1. Snce vault expects both private key and certificate be together. First concatenate them :   cat myrootca.pem myrootca.key > cabundle.pem
    2. Also create JSON file out of it as Vault expects this to be a JSON with pem_bundle attribute.  While you are at it, also ensure that there is a new line added to the end of  BEING CERTIFICATE line, right before END CERTIFICATE line, at the end of BEING RSA PRIVATE KEY line and right before END RSA PRIVATE KEY line. 
    3. curl --header "X-Vault-Token: 334ebe79-a2b8-d507-e5bd-544ffd46409e" --request POST  --data "@cabundle.pem"  $VAULT_ADDR/v1/srini/config/ca
  4. Create a role with following json file (role.json):

    {
    "allow_any_name": true,
    "allow_subdomains": true
    }

  5. Create a role xyz using : curl --header "X-Vault-Token: 334ebe79-a2b8-d507-e5bd-544ffd46409e" --request POST  --data "@role.json"  $VAULT_ADDR/v1/srini/roles/xyz
  6. Generate user certificate
    1. openssl genrsa -out usercert.key 2048
    2. openssl req -new -key usercert.key -out usercert.csr
    3. Convert to JSON format as expected by vault (Example shown below). Again make sure to add new lines at the end of BEGIN CERTIFICATE REQUEST line and before END CERTIFICATE REQUEST line.


      "csr":"-----BEGIN CERTIFICATE REQUEST-----\n
      MIICmzCCAYMCAQAwVjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQwwCgYDVQQK
      DANYWVoxEDAOBgNVBAMMB1hZWi5jb20xGjAYBgkqhkiG9w0BCQEWCzEyM0BYWVou
      Y29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6E42jttRxrcCWyAR
      dgd2iNmUQpUe1DgaktFUfX4Qv7xrqXW27x270iBpWVKm05QnrDPFY/xuPMHAPhhH
      BdcT8Nie2H5OoqOjsejzy4CR+UopXE5fmOwRJlxD16qfAmSjrsCTMXlSa01vhdEI
      Zcd3JPO6c4fLgfJrsQdrV3w45DUFziPDDAF5BiiZDXGKmvYH/BvkLUGI5Zj6j6U7
      Z4u2HcNzznBtzJa9j2S1wbBd1Z4lLEu0tHRQYymwOGWKLefAblDJGkeIPLmkmje2
      r4hEi7Oy1rvi54Hm3/8GGylRONFrao0BqFunjjbd8GsnRZZZo+I5FgvJqY+N5y9D
      TZ7GmQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAENgnHcDXJ3Z0UNbYK8gAzP+
      KzM8F8Zod4EOv1/i1o8rj4yuSXVcxNI6iR3ou5cpDgNn8QbPjqfL4vEy7LLmncTI
      8wuhknSynsZIx/ABpmgyFsMKD9soibEyVFEBz2j9BWKNRXLYhXK+LwiJH2Dlpai/
      mZMto0uFMtjWvMfLkN68QGuwjMCzu5EWusbi9KOSNN2u826YFrT3hrMzJzpOfYVR
      o2vdfKx1QcIJxeh1oOhfE0jNegvgzvrFw+DxQs5Kyd19y/GnxxrXxWb7ufPmZEz8
      EXd9BHLF650VnKvUkts3sYGJUDyRTEuIKXQd65iAtrnEvIrBaBvUD4zicee/AOg=\n
      -----END CERTIFICATE REQUEST-----",
      "common_name":"xyz.com"
      }

    4. Get the certificate signed : 

      curl --header "X-Vault-Token: 334ebe79-a2b8-d507-e5bd-544ffd46409e" --request POST --data "@usercert.json" $VAULT_ADDR/v1/srini/sign/xyz






  • No labels