Ratchet Mechanism to Enforce Cryptographic Agility
Introduction
The previous article, Crypto Agility: Make algorithms change safely, introduced a mechanism that, like a ratchet, moves only forward and prevents reversal, allowing new algorithms to be rolled out gradually while old ones are phased out. I have turned this concept into a simple Go library, riceball-tw/ratchet.
Revisiting the Background and the Problem
Each algorithm and key without centralized management, they end up scattered everywhere:
// Encryption: each call site chooses its own algorithm and keyct, _ := EncryptCBC(plaintext, key, iv)
// Decryption: each call site figures out the format of the stored dataif isGCM(stored) { pt, _ = DecryptGCM(stored, newKey)} else { pt, _ = DecryptCBC(stored, key, iv)}This makes two common requirements painful to handle:
- Upgrading algorithms: CBC, for example, provides no integrity verification. Switching to GCM, an AEAD mode, requires changes at every call site. Existing CBC data in the database must remain decryptable, so backward compatibility also needs attention.
- Rotating keys: When a key is compromised or due for routine rotation, new data must use the new key while data encrypted with the old key must remain readable. Again, every call site needs to know about this.
ratchet acts like a key ring that keeps all the keys together. Callers only need to “lock” and “unlock.” Upgrading algorithms or rotating keys only requires changing the key ring configuration—not a single line at the call sites:
env, _ := r.Encrypt(plaintext, aad) // Encryption: always use the latest keypt, _ := r.Decrypt(env, aad) // Decryption: automatically select the right key from the ciphertext label| Requirement | Used for Encryption (Current) | Used Only for Decryption (DecryptOnly) |
|---|---|---|
| Upgrade CBC to GCM | GCM v1/keyA | Legacy CBC |
| Rotate keys | GCM v1/keyB | GCM v1/keyA |
The key ring relies on two things and enforces a ratchet-like policy:
- A defined format: Every ciphertext is labeled with the key used to encrypt it (algorithm version + key ID).
- A policy fixed at initialization: One scheme handles encryption; the others handle decryption only.
A Ratchet Policy?
Ratchet Gear: a gear that can turn in only one direction
Applied to encryption and decryption, this means newly encrypted data must use the new algorithm. Old schemes remain readable but are never used for encryption again.
- There is always exactly one encryption scheme, and it only moves forward: legacy scheme → v1 → v2
- Old schemes are first demoted to decryption-only, then removed once their data has been migrated
Core Library Concepts
For concrete usage scenarios, skip ahead to the next section. This section examines how to use the library and the design decisions behind its implementation.
Initialization Parameters
v1, _ := aesgcm.New(ratchet.Ref{Version: 1, KeyID: "mk1"}, key)r, _ := aead.New(aead.Config{ Current: v1, DecryptOnly: []aead.Decrypter{ aead.DecrypterFunc(ratchet.Unversioned, legacyDecrypt), },})
env, _ := r.Encrypt([]byte("Hello"), aad)| Role | Type | Encrypt | Decrypt | Purpose |
|---|---|---|---|---|
| Current | aead.Cipher | ✅ | ✅ | The only scheme used for all new data |
| DecryptOnly | aead.Decrypter | ❌ | ✅ | Old schemes (or the next scheme being deployed) |
Metadata Describing the Algorithm
To associate an algorithm with a “version” and the “key id,” we add Ref metadata to the ciphertext, forming an Envelope:
type Ref struct { Version uint32 // Algorithm version, e.g., 1 = AES-256-GCM KeyID string // Custom key ID, e.g., mk1}
type Envelope struct { Ref Ciphertext []byte}Version and KeyID are separate to support rotation in different scenarios:
- Change algorithms:
v1/mk1→v2/mk1 - Change keys:
v1/mk1→v1/mk2
KeyID is simply the name identifying the key in use. The actual key is supplied when initializing aead.Cipher; you can store it in a KMS or an environment variable. Internally, ratchet looks up decrypters using map[Ref]Decrypter, so the same Ref cannot be registered twice.
ratchet:1:mk1:aJHaGWyInGkfczT1HY3s3tNtyKU_mXRbKlyIRT1b0ogNjhc│ │ │ └ base64url(ciphertext)│ │ └ KeyID: which key ID│ └ Version: which algorithm└ Identifies data in the ratchet envelope format (no prefix means an Unversioned Ref)Working with Envelopes
Convert an envelope to a string before storing it in the database, then convert it back to an Envelope when reading it:
stored := env.String() // Store: "ratchet:1:mk1:..."env, err := ratchet.ParseEnvelope(stored) // Read: convert back to an EnvelopeA Minimal Practical Example
Creating an aead.Cipher
key := loadKey() // 32 bytes, managed by youv1, err := aesgcm.New(ratchet.Ref{Version: 1, KeyID: "mk1"}, key)// Package aesgcm is a reference aead.Cipher built on AES-GCM. It shows how// to plug an algorithm into the ratchet; copy and adapt it, or write your own// Cipher for any other algorithm.package aesgcm
import ( "crypto/aes" "crypto/cipher" "encoding/binary" "fmt"
"github.com/riceball-tw/ratchet" "github.com/riceball-tw/ratchet/aead")
// New returns an AES-GCM Cipher. key must be 16, 24 or 32 bytes; prefer 32// (AES-256) for long-lived data.//// Nonces are random (96-bit), so a single key should not encrypt more than// about 2^32 messages; rotate the key ID well before that.//// The Ref is bound into the authenticated data, so an envelope whose version// or key ID was tampered with fails to decrypt.func New(ref ratchet.Ref, key []byte) (aead.Cipher, error) { if ref == ratchet.Unversioned { return nil, fmt.Errorf("aesgcm: needs a versioned Ref") } if err := ref.Validate(); err != nil { return nil, err } block, err := aes.NewCipher(key) if err != nil { return nil, fmt.Errorf("aesgcm: %s: %w", ref, err) } gcm, err := cipher.NewGCMWithRandomNonce(block) if err != nil { return nil, fmt.Errorf("aesgcm: %s: %w", ref, err) } return &aesGCM{ref: ref, gcm: gcm}, nil}
type aesGCM struct { ref ratchet.Ref gcm cipher.AEAD}
func (c *aesGCM) Ref() ratchet.Ref { return c.ref }
func (c *aesGCM) Encrypt(plaintext, aad []byte) ([]byte, error) { return c.gcm.Seal(nil, nil, plaintext, bindAAD(c.ref, aad)), nil}
func (c *aesGCM) Decrypt(ciphertext, aad []byte) ([]byte, error) { return c.gcm.Open(nil, nil, ciphertext, bindAAD(c.ref, aad))}
// bindAAD prefixes the caller's AAD with an unambiguous encoding of ref.func bindAAD(ref ratchet.Ref, aad []byte) []byte { b := make([]byte, 0, 8+len(ref.KeyID)+len(aad)) b = binary.BigEndian.AppendUint32(b, ref.Version) b = binary.BigEndian.AppendUint32(b, uint32(len(ref.KeyID))) b = append(b, ref.KeyID...) return append(b, aad...)}ratchet does not care how keys are generated or stored; it only recognizes Ref. aesgcm.New binds a Ref and a key into the same aead.Cipher, so KeyID is the name you give that key, allowing reads to locate the key originally used for encryption.
Wrapping Existing Decryption Logic
func legacyDecrypt(ciphertext, _ []byte) ([]byte, error) { return oldDecryptCBC(ciphertext) // Unchanged}
legacy := aead.DecrypterFunc(ratchet.Unversioned, legacyDecrypt)Assembling an aead.Ratchet Instance
r, err := aead.New(aead.Config{ Current: v1, // The only scheme used for encryption DecryptOnly: []aead.Decrypter{legacy}, // Decryption only OnNewerData: func(found, current ratchet.Ref) {}, // Optional: called when reading a version newer than Current})if err != nil { log.Fatal(err) // Refuse to start if the configuration is invalid}Usage
aad := []byte("users/42/password")
// Write: always use Currentenv, _ := r.Encrypt([]byte("hunter2"), aad)db.Save(env.String()) // ratchet:1:mk1:...
// Read: accept both new and old formats, dispatching automatically by labelenv, _ = ratchet.ParseEnvelope(db.Load())plaintext, _ := r.Decrypt(env, aad)
// Migrate: decrypt with the old scheme, then re-encrypt with Currentenv, changed, _ := r.Rewrap(env, aad)Scenario: Rolling Out a New Scheme Across Multiple Services
Services are updated independently, so use two deployments:
// Deployment A: make all servers capable of reading v2 while still writing v1aead.Config{Current: v1, DecryptOnly: []aead.Decrypter{legacy, v2}}
// Deployment B: once A is complete everywhere, switch writes to v2// Demote v1 to read-onlyaead.Config{Current: v2, DecryptOnly: []aead.Decrypter{legacy, v1}}If you skip A and deploy directly, services that have not yet been updated will receive an error when reading v2 data: ratchet: unknown scheme: v2/mk1.
Scenario: Updating Old Data with Rewrap
After switching to v2, only newly written data uses v2; existing v1 data in the database does not update itself. Without migration, v1 can never be retired. This is what Rewrap handles: it decrypts using the old scheme recorded in the envelope, then re-encrypts using Current.
newEnv, changed, err := r.Rewrap(env, aad)With Current set to v2, for example:
| Input | Output | changed |
|---|---|---|
Unversioned legacy data | ratchet:2:mk1:... | true |
ratchet:1:mk1:... | ratchet:2:mk1:... | true |
ratchet:2:mk1:... | Returned unchanged | false |
There are two approaches, typically used together:
// 1. Upgrade on read: frequently accessed data is migrated naturallyplaintext, _ := r.Decrypt(env, aad)if newEnv, changed, err := r.Rewrap(env, aad); err == nil && changed { db.Save(id, newEnv.String())}
// 2. Batch migration: migrate rarely accessed data in one passfor _, row := range db.FindNotPrefixed("ratchet:2:") { env, _ := ratchet.ParseEnvelope(row.Value) newEnv, changed, err := r.Rewrap(env, row.AAD) if err != nil { return err // Do not skip data that fails to decrypt; investigate it } if changed { db.Save(row.ID, newEnv.String()) }}Once this is complete, confirm that no ratchet:1: data remains in the database, then remove v1 from DecryptOnly. The ratchet can now advance one notch.
Scenario: Managing and Rotating Multiple Keys
ratchet does not dictate where keys come from, but in practice you usually need two things: separate keys for separate purposes and the ability to rotate master keys. A common approach is to manage only a handful of master keys, use KeyID as the master key name, and derive purpose-specific keys with HKDF:
// Name → master key; the master key itself is stored in a KMS or environment variablevar masterKeys = map[string][]byte{ "mk1": load("MASTER_KEY_MK1"),}
func newCipher(purpose string, ref ratchet.Ref) (aead.Cipher, error) { master, ok := masterKeys[ref.KeyID] // Look up the master key by KeyID if !ok { return nil, fmt.Errorf("unknown key id %q", ref.KeyID) } // info, e.g., "myapp/mail.password/v1/mk1": different purposes and Refs produce unrelated keys key, err := hkdf.Key(sha256.New, master, nil, "myapp/"+purpose+"/"+ref.String(), 32) if err != nil { return nil, err } return aesgcm.New(ref, key)}Suppose mk1 is compromised one day. Rotating the master key follows the same process as upgrading an algorithm, except this time you change KeyID rather than Version:
var masterKeys = map[string][]byte{ "mk1": load("MASTER_KEY_MK1"), // Old key, retained only for decryption "mk2": load("MASTER_KEY_MK2"), // New key}
current, _ := newCipher("mail.password", ratchet.Ref{Version: 1, KeyID: "mk2"})old, _ := newCipher("mail.password", ratchet.Ref{Version: 1, KeyID: "mk1"})
r, _ := aead.New(aead.Config{ Current: current, // Use mk2 for new data DecryptOnly: []aead.Decrypter{old}, // Old data remains readable})- Both
ratchet:1:mk1:...andratchet:1:mk2:...coexist in the database, each using itsKeyIDto locate the master key originally used for encryption - Use
Rewrapto migrate data frommk1tomk2 - Once no
ratchet:1:mk1:data remains, retiremk1completely
Changing Only KeyID Is Not Key Rotation
Because Ref is included in HKDF’s info, changing only KeyID without replacing the master key will still produce a different derived key. But the derivation rules are not secret (info is in the source code, and Ref appears on every envelope); the only secret is the master key. If that master key is compromised, an attacker can follow the same rules to compute all derived keys for both mk1 and mk2, making the change ineffective. Best practice is therefore to:
- Associate each
KeyIDwith an independently generated master key: Generate a fresh 32 bytes using a KMS orcrypto/rand, rather than renaming the old key and registering it again. - Check for duplicate master keys at startup to avoid accidentally assigning the same key to two names in the configuration.
Conclusion
Best practices in documentation get ignored in the rush to ship; mandatory workflows enforced in code are what people actually follow
The value of ratchet lies in providing an easy-to-understand abstraction while ensuring that the entire system moves only toward best practices. Adoption is just the beginning; continually converging on best practices is the goal.