Skip to content

Customer Data Retention (GDPR)

Why

End customers ("endCustomers") are stored in the users collection of dingoo-api (name, email, phone, addresses) for as long as they have orders associated with them. Without any cleanup mechanism, this data stayed there indefinitely even after a customer stopped ordering — which doesn't meet GDPR's data minimization principle (don't keep personal data longer than necessary).

The implemented solution (ticket DIS-1010) is a scheduler that, per business, identifies customers who have been inactive for X days and tokenizes their contact data reversibly — the data becomes unreadable, but can be recovered by someone authorized, unlike an irreversible hash or a hard delete.

Why not hash / why not delete?

The first version of this feature used a SHA-256 hash (irreversible) or a direct delete. It was changed at the business's request: sometimes there's a need to reverse an anonymization (e.g. a legal dispute, a request from the customer themselves, a business misconfiguration). A hash or a delete don't allow that — reversible tokenization via KMS (see below) does.

How it works

The job runs on bo-api, endpoint POST /api/cronjobs/end-customers/cleanup, scheduled via Cloud Scheduler every day at 3am UTC (cloudbuild.yaml, step schedule-end-customers-cleanup).

For each business (Business) with customerDataRetention.isActive !== false:

  1. Compute cutoff = today − daysSinceLastOrder days.
  2. A single MongoDB aggregation over the orders collection (for every business at once, not one at a time — the collection has no index on business/user, so avoiding repeated scans matters) computes the last delivery date (orderFlows.0.deliveryDate) per {business, user}.
  3. A User is considered inactive for that business if their last delivery is older than cutoff. A user who never had an order is not picked up by any business — there's no reliable way to associate them with a business without an order (see Known limitations).
  4. Inactive users (not yet processed, anonymizedAt: null) are tokenized: name, email, phone, addresses — except importedId, which is deliberately left alone (see limitations).
  5. That business's orders belonging to those users are also tokenized — orderId, confirmedBarcode and every entry of possibleBarcodes. These are external identifiers that, on their own, can also re-identify the customer, and confirmedBarcode/possibleBarcodes routinely fall back to (or directly embed) the plain orderId anyway — tokenizing orderId alone would leave it readable through these fields.

Code: app/bo/routes/v1/cronjobs.js.

Backoffice configuration

Each business has its own customerDataRetention (model Business, app/shared/models/business.js):

customerDataRetention: {
  isActive: Boolean,           // default true — turns cleanup off for this business
  daysSinceLastOrder: Number,  // default 30
  history: [                   // audit trail — never edited by hand, only by the backend
    {
      isActive: Boolean,
      daysSinceLastOrder: Number,
      changedBy: ObjectId,     // ref Staff — who made the change
      changedAt: Date,
    },
  ],
}

Editable from Businesses → [business] → General, "Customer Data Retention" section. Every change to isActive or daysSinceLastOrder is logged to history automatically (done in putBusiness, app/bo/routes/v1/businesses.js) — the frontend only needs to send the two values, the backend computes the diff and appends the audit entry.

Serious change

In the Backoffice, changing these values requires confirming a popup with a mandatory checkbox — this is a GDPR compliance change, it must match what's actually agreed in the contract with that business.

Tokenization and encryption (KMS)

This is the part that makes "reversal" possible, but only for someone authorized — and it's the part most worth understanding well, since a mistake here has compliance implications.

Envelope encryption

Instead of calling Cloud KMS directly to encrypt every field (slow — one network request per field, per user, every day), we use envelope encryption:

  1. At the start of each job run, a random 32-byte symmetric key is generated (the "DEK" — Data Encryption Key), in memory only.
  2. That DEK is encrypted once with Cloud KMS (kmsClient.encrypt) — the result is the "wrapped key".
  3. Every field to be tokenized in that run (potentially thousands) is encrypted locally, with AES-256-GCM, using that same in-memory DEK. No further network calls to KMS after step 2.
  4. Each encrypted value is stored as a self-contained string, format gdpr:v1:<wrappedKey>:<iv>:<tag>:<ciphertext> (all base64) — it includes the encrypted DEK itself, so any token can be reversed on its own, without needing to fetch the DEK from anywhere else.

Code: app/shared/utils/tokenization.js.

export const createTokenizer = async () => {
  const dek = randomBytes(32)
  const [{ ciphertext: wrappedKey }] = await kmsClient.encrypt({
    name: process.env.GDPR_KMS_KEY_NAME,
    plaintext: dek,
  })

  return (value) => {
    // encrypts `value` locally with `dek`, returns the "gdpr:v1:..." string
  }
}

Key rotation

The end-customer-pii key rotates automatically every 90 days (a new key version becomes primary). This does not break decryption of tokens created before the rotation, and it's not something our code has to handle — it's inherent to how KMS works:

  • Every ciphertext produced by kmsClient.encrypt embeds, internally, which key version produced it.
  • When you call kmsClient.decrypt({ name: KEY_NAME, ciphertext }), you only ever pass the key name, never a version — Cloud KMS reads the version info embedded in the ciphertext itself and uses that specific version to decrypt, whether or not it's still the primary one.
  • Rotation only changes which version is used for new encryptions. Old versions stay Enabled (and therefore usable for decrypt) indefinitely, unless someone manually disables or destroys them — automatic rotation never does that on its own.

This was verified directly: a value was tokenized while key version 1 was primary, then version 2 was created and made primary (simulating what a 90-day rotation does), and the original token — still referencing version 1 — decrypted correctly without any code change or version bookkeeping on our side.

Why this is secure, despite being reversible

A plain hash (e.g. sha256(phone)) of a low-entropy value like a phone number or email is crackable by brute force — the space of possible values is small enough to precompute every possible hash. Tokenization avoids this in two ways:

  • It's not a hash, it's real encryption (AES-256-GCM) — without the key, there's no practical way to reverse it.
  • The key (DEK) is never stored in the clear — it's always encrypted by KMS. Only someone with decrypt permission on the KMS key can even get to the DEK, and only then can they decrypt the field.

Permissions (who can reverse it)

The end-customer-pii key (keyring gdpr-tokenization, region europe-west1) is set up with two separate permissions:

  • roles/cloudkms.cryptoKeyEncrypter — granted to the apps-bo-api-sa service account (the one running bo-api in production). It can only encrypt. It cannot decrypt anything — even if the application code were compromised, it couldn't reverse the tokenization from there.
  • roles/cloudkms.cryptoKeyDecrypter — granted manually, person by person, to whoever is authorized to reverse a tokenization. There's no automation/pipeline that grants this permission — it's always a manual, deliberate step in the GCP Console.

Automated key provisioning (idempotent, runs in every environment) lives in cloudbuild.yaml, step ensure-gdpr-kms-key.

Never grant cryptoKeyDecrypter to the application's service account

If the app had decrypt permission, the "only one person can reverse it" guarantee would no longer exist — anything able to execute code on bo-api (a vulnerability, an RCE bug, etc.) would be able to decrypt all tokenized data.

Special case: addresses (User.addresses)

Addresses couldn't be tokenized field by field like name/email/phone, because location.lat/location.lng are Number in the schema — there's no way to store an encrypted string there without changing the field's type.

The solution: instead of encrypting each address field individually, the job bundles the whole address (address, number, floor, postCode, location, city, addressNotes) into a single JSON string, encrypts that whole string at once, and stores the result in a new field, token (added to the schema — app/shared/models/user.js). The old fields are left unset.

// Before
{ _id, address: "Street X, 123", postCode: "1000-001", location: { lat, lng }, city, addressNotes }

// After tokenization
{ _id, token: "gdpr:v1:..." }  // everything else is encrypted in there

To reverse it: decrypt the token, the result is the original JSON string, run JSON.parse() and recover {address, number, floor, postCode, location, city, addressNotes}.

Known limitations

  • User.importedId is left untouched. It's the field order-import flows use to find an existing User (orders/import/utils.js, orders.js, etc.). If it were tokenized, reimporting a customer who was already anonymized would no longer find the old record and would always create a new one — which, for other purposes, would actually be the correct GDPR behavior, but it would break the proximity-based address matching that also depends on this field. Left as a deliberate, accepted limitation, to be fixed once that matching no longer depends on the users collection.
  • Direct consequence of the point above: if an already-tokenized customer places new orders again (found again via importedId), name/email/phone/addresses get overwritten with the real data from the import, "undoing" the tokenization. Every write path that can do this — orders/import/utils.js (validateUser), orders.js (its own separate validateUser, plus the User.findOneAndUpdate/ updateOne calls in putOrder and importMasterData), master-data.js, and users/updateUserAddress.js — clears anonymizedAt back to null as part of the same update, so the User is re-evaluated normally by future job runs instead of being permanently skipped.
  • Order.anonymizedAt mirrors this for orderId/confirmedBarcode/ possibleBarcodes: the cleanup job only tokenizes orders where it's still null, and sets it once tokenized. Without this guard, a user revived through one of the paths above and later marked inactive again would have their already-tokenized past orders tokenized a second time (tokenizing already-opaque values) on the next run.
  • A User with no orders associated with any business is never picked up by the job (there's no way to know which business they belong to without an Order).