Ratchet Mechanism to Enforce Cryptographic Agility

Introduction

The previous article, Crypto Agility: Make algorithms change safely, introduced a mechanism that, like a ratchet, moves only forward and prevents reversal, allowing new algorithms to be rolled out gradually while old ones are phased out. I have turned this concept into a simple Go library, riceball-tw/ratchet🔗.

Revisiting the Background and the Problem

Each algorithm and key without centralized management, they end up scattered everywhere:

// Encryption: each call site chooses its own algorithm and key
ct, _ := EncryptCBC(plaintext, key, iv)
// Decryption: each call site figures out the format of the stored data
if isGCM(stored) {
pt, _ = DecryptGCM(stored, newKey)
} else {
pt, _ = DecryptCBC(stored, key, iv)
}

This makes two common requirements painful to handle:

  • Upgrading algorithms: CBC, for example, provides no integrity verification. Switching to GCM, an AEAD mode, requires changes at every call site. Existing CBC data in the database must remain decryptable, so backward compatibility also needs attention.
  • Rotating keys: When a key is compromised or due for routine rotation, new data must use the new key while data encrypted with the old key must remain readable. Again, every call site needs to know about this.

ratchet acts like a key ring that keeps all the keys together. Callers only need to “lock” and “unlock.” Upgrading algorithms or rotating keys only requires changing the key ring configuration—not a single line at the call sites:

env, _ := r.Encrypt(plaintext, aad) // Encryption: always use the latest key
pt, _ := r.Decrypt(env, aad) // Decryption: automatically select the right key from the ciphertext label
RequirementUsed for Encryption (Current)Used Only for Decryption (DecryptOnly)
Upgrade CBC to GCMGCM v1/keyALegacy CBC
Rotate keysGCM v1/keyBGCM v1/keyA

The key ring relies on two things and enforces a ratchet-like policy:

  1. A defined format: Every ciphertext is labeled with the key used to encrypt it (algorithm version + key ID).
  2. A policy fixed at initialization: One scheme handles encryption; the others handle decryption only.

A Ratchet Policy?

Ratchet Gear: a gear that can turn in only one direction

Applied to encryption and decryption, this means newly encrypted data must use the new algorithm. Old schemes remain readable but are never used for encryption again.

Adopt ratchet

Migrate legacy data

Roll out v2

Migrate v1 data

Before adoption
━━━━
Encrypt: legacy scheme
Decrypt: legacy scheme

Transition
━━━━
Encrypt: v1
Decrypt: v1, unversioned

Stable
━━━━
Encrypt: v1
Decrypt: v1

Transition
━━━━
Encrypt: v2
Decrypt: v2, v1

Stable
━━━━
Encrypt: v2
Decrypt: v2

  • There is always exactly one encryption scheme, and it only moves forward: legacy scheme → v1 → v2
  • Old schemes are first demoted to decryption-only, then removed once their data has been migrated

Core Library Concepts

For concrete usage scenarios, skip ahead to the next section. This section examines how to use the library and the design decisions behind its implementation.

Initialization Parameters

v1, _ := aesgcm.New(ratchet.Ref{Version: 1, KeyID: "mk1"}, key)
r, _ := aead.New(aead.Config{
Current: v1,
DecryptOnly: []aead.Decrypter{
aead.DecrypterFunc(ratchet.Unversioned, legacyDecrypt),
},
})
env, _ := r.Encrypt([]byte("Hello"), aad)
RoleTypeEncryptDecryptPurpose
Currentaead.Cipher✅✅The only scheme used for all new data
DecryptOnlyaead.Decrypter❌✅Old schemes (or the next scheme being deployed)

Metadata Describing the Algorithm

To associate an algorithm with a “version” and the “key id,” we add Ref metadata to the ciphertext, forming an Envelope:

type Ref struct {
Version uint32 // Algorithm version, e.g., 1 = AES-256-GCM
KeyID string // Custom key ID, e.g., mk1
}
type Envelope struct {
Ref
Ciphertext []byte
}

Version and KeyID are separate to support rotation in different scenarios:

  • Change algorithms: v1/mk1 → v2/mk1
  • Change keys: v1/mk1 → v1/mk2

KeyID is simply the name identifying the key in use. The actual key is supplied when initializing aead.Cipher; you can store it in a KMS or an environment variable. Internally, ratchet looks up decrypters using map[Ref]Decrypter, so the same Ref cannot be registered twice.

Ratchet Envelope Demo
ratchet:1:mk1:aJHaGWyInGkfczT1HY3s3tNtyKU_mXRbKlyIRT1b0ogNjhc
│ │ │ └ base64url(ciphertext)
│ │ └ KeyID: which key ID
│ └ Version: which algorithm
└ Identifies data in the ratchet envelope format (no prefix means an Unversioned Ref)

Working with Envelopes

Convert an envelope to a string before storing it in the database, then convert it back to an Envelope when reading it:

stored := env.String() // Store: "ratchet:1:mk1:..."
env, err := ratchet.ParseEnvelope(stored) // Read: convert back to an Envelope

ratchet:2:...

ratchet:1:...

No ratchet: prefix

Other versions

Write

v2
Encrypt + Decrypt

Read

Envelope label

v1
Decrypt only

Unversioned
Decrypt only

ErrUnknownRef

A Minimal Practical Example

Creating an aead.Cipher

key := loadKey() // 32 bytes, managed by you
v1, err := aesgcm.New(ratchet.Ref{Version: 1, KeyID: "mk1"}, key)
aesgcm.go
// Package aesgcm is a reference aead.Cipher built on AES-GCM. It shows how
// to plug an algorithm into the ratchet; copy and adapt it, or write your own
// Cipher for any other algorithm.
package aesgcm
import (
"crypto/aes"
"crypto/cipher"
"encoding/binary"
"fmt"
"github.com/riceball-tw/ratchet"
"github.com/riceball-tw/ratchet/aead"
)
// New returns an AES-GCM Cipher. key must be 16, 24 or 32 bytes; prefer 32
// (AES-256) for long-lived data.
//
// Nonces are random (96-bit), so a single key should not encrypt more than
// about 2^32 messages; rotate the key ID well before that.
//
// The Ref is bound into the authenticated data, so an envelope whose version
// or key ID was tampered with fails to decrypt.
func New(ref ratchet.Ref, key []byte) (aead.Cipher, error) {
if ref == ratchet.Unversioned {
return nil, fmt.Errorf("aesgcm: needs a versioned Ref")
}
if err := ref.Validate(); err != nil {
return nil, err
}
block, err := aes.NewCipher(key)
if err != nil {
return nil, fmt.Errorf("aesgcm: %s: %w", ref, err)
}
gcm, err := cipher.NewGCMWithRandomNonce(block)
if err != nil {
return nil, fmt.Errorf("aesgcm: %s: %w", ref, err)
}
return &aesGCM{ref: ref, gcm: gcm}, nil
}
type aesGCM struct {
ref ratchet.Ref
gcm cipher.AEAD
}
func (c *aesGCM) Ref() ratchet.Ref { return c.ref }
func (c *aesGCM) Encrypt(plaintext, aad []byte) ([]byte, error) {
return c.gcm.Seal(nil, nil, plaintext, bindAAD(c.ref, aad)), nil
}
func (c *aesGCM) Decrypt(ciphertext, aad []byte) ([]byte, error) {
return c.gcm.Open(nil, nil, ciphertext, bindAAD(c.ref, aad))
}
// bindAAD prefixes the caller's AAD with an unambiguous encoding of ref.
func bindAAD(ref ratchet.Ref, aad []byte) []byte {
b := make([]byte, 0, 8+len(ref.KeyID)+len(aad))
b = binary.BigEndian.AppendUint32(b, ref.Version)
b = binary.BigEndian.AppendUint32(b, uint32(len(ref.KeyID)))
b = append(b, ref.KeyID...)
return append(b, aad...)
}

ratchet does not care how keys are generated or stored; it only recognizes Ref. aesgcm.New binds a Ref and a key into the same aead.Cipher, so KeyID is the name you give that key, allowing reads to locate the key originally used for encryption.

Wrapping Existing Decryption Logic

func legacyDecrypt(ciphertext, _ []byte) ([]byte, error) {
return oldDecryptCBC(ciphertext) // Unchanged
}
legacy := aead.DecrypterFunc(ratchet.Unversioned, legacyDecrypt)

Assembling an aead.Ratchet Instance

r, err := aead.New(aead.Config{
Current: v1, // The only scheme used for encryption
DecryptOnly: []aead.Decrypter{legacy}, // Decryption only
OnNewerData: func(found, current ratchet.Ref) {}, // Optional: called when reading a version newer than Current
})
if err != nil {
log.Fatal(err) // Refuse to start if the configuration is invalid
}

Usage

aad := []byte("users/42/password")
// Write: always use Current
env, _ := r.Encrypt([]byte("hunter2"), aad)
db.Save(env.String()) // ratchet:1:mk1:...
// Read: accept both new and old formats, dispatching automatically by label
env, _ = ratchet.ParseEnvelope(db.Load())
plaintext, _ := r.Decrypt(env, aad)
// Migrate: decrypt with the old scheme, then re-encrypt with Current
env, changed, _ := r.Rewrap(env, aad)

Scenario: Rolling Out a New Scheme Across Multiple Services

Services are updated independently, so use two deployments:

// Deployment A: make all servers capable of reading v2 while still writing v1
aead.Config{Current: v1, DecryptOnly: []aead.Decrypter{legacy, v2}}
// Deployment B: once A is complete everywhere, switch writes to v2
// Demote v1 to read-only
aead.Config{Current: v2, DecryptOnly: []aead.Decrypter{legacy, v1}}

If you skip A and deploy directly, services that have not yet been updated will receive an error when reading v2 data: ratchet: unknown scheme: v2/mk1.

Scenario: Updating Old Data with Rewrap

After switching to v2, only newly written data uses v2; existing v1 data in the database does not update itself. Without migration, v1 can never be retired. This is what Rewrap handles: it decrypts using the old scheme recorded in the envelope, then re-encrypts using Current.

newEnv, changed, err := r.Rewrap(env, aad)

With Current set to v2, for example:

InputOutputchanged
Unversioned legacy dataratchet:2:mk1:...true
ratchet:1:mk1:...ratchet:2:mk1:...true
ratchet:2:mk1:...Returned unchangedfalse

There are two approaches, typically used together:

// 1. Upgrade on read: frequently accessed data is migrated naturally
plaintext, _ := r.Decrypt(env, aad)
if newEnv, changed, err := r.Rewrap(env, aad); err == nil && changed {
db.Save(id, newEnv.String())
}
// 2. Batch migration: migrate rarely accessed data in one pass
for _, row := range db.FindNotPrefixed("ratchet:2:") {
env, _ := ratchet.ParseEnvelope(row.Value)
newEnv, changed, err := r.Rewrap(env, row.AAD)
if err != nil {
return err // Do not skip data that fails to decrypt; investigate it
}
if changed {
db.Save(row.ID, newEnv.String())
}
}

Once this is complete, confirm that no ratchet:1: data remains in the database, then remove v1 from DecryptOnly. The ratchet can now advance one notch.

Scenario: Managing and Rotating Multiple Keys

ratchet does not dictate where keys come from, but in practice you usually need two things: separate keys for separate purposes and the ability to rotate master keys. A common approach is to manage only a handful of master keys, use KeyID as the master key name, and derive purpose-specific keys with HKDF:

// Name → master key; the master key itself is stored in a KMS or environment variable
var masterKeys = map[string][]byte{
"mk1": load("MASTER_KEY_MK1"),
}
func newCipher(purpose string, ref ratchet.Ref) (aead.Cipher, error) {
master, ok := masterKeys[ref.KeyID] // Look up the master key by KeyID
if !ok {
return nil, fmt.Errorf("unknown key id %q", ref.KeyID)
}
// info, e.g., "myapp/mail.password/v1/mk1": different purposes and Refs produce unrelated keys
key, err := hkdf.Key(sha256.New, master, nil, "myapp/"+purpose+"/"+ref.String(), 32)
if err != nil {
return nil, err
}
return aesgcm.New(ref, key)
}

Suppose mk1 is compromised one day. Rotating the master key follows the same process as upgrading an algorithm, except this time you change KeyID rather than Version:

var masterKeys = map[string][]byte{
"mk1": load("MASTER_KEY_MK1"), // Old key, retained only for decryption
"mk2": load("MASTER_KEY_MK2"), // New key
}
current, _ := newCipher("mail.password", ratchet.Ref{Version: 1, KeyID: "mk2"})
old, _ := newCipher("mail.password", ratchet.Ref{Version: 1, KeyID: "mk1"})
r, _ := aead.New(aead.Config{
Current: current, // Use mk2 for new data
DecryptOnly: []aead.Decrypter{old}, // Old data remains readable
})
  1. Both ratchet:1:mk1:... and ratchet:1:mk2:... coexist in the database, each using its KeyID to locate the master key originally used for encryption
  2. Use Rewrap to migrate data from mk1 to mk2
  3. Once no ratchet:1:mk1: data remains, retire mk1 completely

Changing Only KeyID Is Not Key Rotation

Because Ref is included in HKDF’s info, changing only KeyID without replacing the master key will still produce a different derived key. But the derivation rules are not secret (info is in the source code, and Ref appears on every envelope); the only secret is the master key. If that master key is compromised, an attacker can follow the same rules to compute all derived keys for both mk1 and mk2, making the change ineffective. Best practice is therefore to:

  1. Associate each KeyID with an independently generated master key: Generate a fresh 32 bytes using a KMS or crypto/rand, rather than renaming the old key and registering it again.
  2. Check for duplicate master keys at startup to avoid accidentally assigning the same key to two names in the configuration.

Conclusion

Best practices in documentation get ignored in the rush to ship; mandatory workflows enforced in code are what people actually follow

The value of ratchet lies in providing an easy-to-understand abstraction while ensuring that the entire system moves only toward best practices. Adoption is just the beginning; continually converging on best practices is the goal.

Further Reading