Protecting OpenSSL private keys using PKCS#8

When generating private keys with OpenSSL, by default they are unprotected by a passphrase. For example:

$ openssl genrsa
-----BEGIN PRIVATE KEY----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCY+q1kOPM4RF5T
...
Rjet4T2TrJmFzIL1dsgJACU=
-----END PRIVATE KEY-----

generates a plaintext private key. By using the -aes256 parameters, the generated key is protected according to PKCS#8:

$ openssl genrsa -aes256
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQITQpYPEgrGN0CAggA
...
XIdi3hSPE8NoPWnROVGMnGbFOMm36g6064nZzQBD4r+4
-----END ENCRYPTED PRIVATE KEY-----

However, is it actually secure?

The default protection used by genrsa is PBKDF2 with 2048 (0x800) iterations of HMAC-SHA256. This can be verified by examining the PKCS#8 key, which is ASN.1 encoded.

$ openssl asn1parse -in key.priv 
    0:d=0  hl=4 l=1325 cons: SEQUENCE          
    4:d=1  hl=2 l=  87 cons: SEQUENCE          
    6:d=2  hl=2 l=   9 prim: OBJECT            :PBES2
   17:d=2  hl=2 l=  74 cons: SEQUENCE          
   19:d=3  hl=2 l=  41 cons: SEQUENCE          
   21:d=4  hl=2 l=   9 prim: OBJECT            :PBKDF2
   32:d=4  hl=2 l=  28 cons: SEQUENCE          
   34:d=5  hl=2 l=   8 prim: OCTET STRING      [HEX DUMP]:EAD8B97C8EAD1414
   44:d=5  hl=2 l=   2 prim: INTEGER           :0800
   48:d=5  hl=2 l=  12 cons: SEQUENCE          
   50:d=6  hl=2 l=   8 prim: OBJECT            :hmacWithSHA256
   60:d=6  hl=2 l=   0 prim: NULL              
   62:d=3  hl=2 l=  29 cons: SEQUENCE          
   64:d=4  hl=2 l=   9 prim: OBJECT            :aes-256-cbc
   75:d=4  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:290DB8F1786A9C8667829A4A394A8FB7
   93:d=1  hl=4 l=1232 prim: OCTET STRING      [HEX DUMP]:D828C11F170D22B82801A567A9136AA293D630692B88C1F07987ED256C29DC45C4625709A0D36039B22E5A8BCA2B400C590C2560CED3629D1F8C5D77AD...CC5F8027A3A5ED533A00D626E7DDC4ABC85547

A meager 2048 SHA256 iterations provide very little added protection on top of the password intrinsic entropy.

Better protection can be achieved by using scrypt to protect the PKCS#8 encoded key. You can upgrade an existing key’s protection to scrypt:

openssl pkcs8 -in key.priv -topk8 -scrypt -out key-scrypt.priv

The command will prompt for the key’s passphrase and re-encrypt it using a key derived from the same passphrase using scrypt.

openssl pkcs8 -in key3.priv -topk8 -scrypt -out key5priv

The default openssl scrypt parameters, N=0x4000=2^14, r=8, p=1, are simply weak. They are what scrypt’s author recommended to achieve 100ms per iteration on his 2002-era laptop. While better than the PBKDF2 protection offered by OpenSSL, you might want stronger protection.

Choosing better scrypt parameters

Before we dive into the parameters, let’s understand the scrypt algorithm briefly. Scrypt takes three input parameters: password, salt, and cost parameters. The password and salt are user-provided inputs, whereas the cost parameters are fixed for a given implementation.

The cost parameters determine the computational cost of the algorithm. Higher cost parameters result in increased computational requirements, making it harder to brute-force the derived key. The cost parameters are as follows:

• N: This is the CPU/memory cost parameter. It determines the number of iterations the algorithm performs. The value of N must be a power of two.
• r: This is the block size parameter. It determines the size of the blocks of memory used during the algorithm’s execution.
• p: This is the parallelization parameter. Doesn’t affect the memory consumption of the algorithm. However, despite it’s name, OpenSSL doesn’t perform any parallelization, instead opting to run the algorithm serially.

The original article recommends r=8 and p=1 as a good ratio between memory and CPU costs. This allows us to scale N almost arbitrary, increasing both CPU and memory costs. However, OpenSSL, by default, limits it’s memory usage.

$ openssl pkcs8 -in key.priv -topk8 -scrypt -scrypt_N 0x8000 -out key-scrypt2.priv
Enter pass phrase for key.priv:
Error setting PBE algorithm
40E778DD277F0000:error:030000AC:digital envelope routines:scrypt_alg:memory limit exceeded:../providers/implementations/kdfs/scrypt.c:482:
40E778DD277F0000:error:068000E3:asn1 encoding routines:PKCS5_pbe2_set_scrypt:invalid scrypt parameters:../crypto/asn1/p5_scrypt.c:59:

This limitation severely our parameters choice. The only option to increase security, is through the CPU cost parameter (p). We can set it (almost) arbitrary large, getting added security. For example:

$ openssl pkcs8 -in MOK.priv -topk8 -scrypt -scrypt_p 32 -passin pass:1234 -passout pass:1234 -out key-scrypt3.priv