---
title: "2023-08-31 Accidental deletion of DNS records - INC0086487"
canonical: "https://wiki.patientsknowbest.com/space/REL/3794993153/2023-08-31%20Accidental%20deletion%20of%20DNS%20records%20-%20INC0086487"
format: markdown
---
| **Incident opened** | 31/08/2023 14:00 |
| --- | --- |
| **Incident closed** |  |

| ## <span style="color: #ffffff">Environments affected</span> |
| --- |

- [ ] Release candidate (RC)
- [x] Sandbox
- [x] UK Production
- [x] EU Production
- [x] EDU Production

| ## <span style="color: #ffffff">References</span> |
| --- |

- **PKB Reference:** <span style="color: #ff5630">**FD144767**</span>
- **Service Bridge incident reference number: **<span style="color: #ff5630">**INC0086487**</span>

| ## <span style="color: #ffffff">Issue Description </span> |
| --- |

The accidental deletion of DNS records caused production environments to be unreachable for most customers.

| ## <span style="color: #ffffff">Impact</span> |
| --- |

All PKB services that are accessed through a domain name were unreachable for customers for 2h15m. Some customers who used DNS providers that didn’t observe TTLs could not access PKB for >24h.

| ## <span style="color: #ffffff">Timeline & Resolving Activity</span> |
| --- |

- 13:50 - DNS records are removed due to running a deleterious script on an incorrect project
- 13:55 - Issue is identified and the restoration process is started
- 13:57 - Restoration of records complete
- 14:43 - Attempt is made to speed up DNS record propagation (at this point it’s still believed that replication is happening, as multiple DNS servers across the globe are picking up the records again)
- 14:46
  - Noticed that on re-creation of zones in Cloud DNS, Google Cloud selected different nameservers than what the registrar knew about, therefore made updates at the registrar’s side.
- 15:10 We suspected that the slow or lack of record propagation could be caused by DNSSEC misconfiguration
- 16:00 - Corrected DS records to match what the registrar was publishing, flushed caches at Cloudflare, Google, OpenDNS, propagation has started immediately

| ## <span style="color: #ffffff">Root Cause Investigations </span> |
| --- |

1. An infrastructure engineer was working on a minor task which involved editing DNS entries. Work was completed but the console was left open. A bit later, the engineer carried out another operation that involved the deletion of cloud resources in a test environment. The deletion operation was executed in the wrong console (that was left open earlier), leading to the deletion of DNS resources. Why was it possible to delete infrastructure components without further checks?
  1. The engineer used a flag called ‘autoapprove’, which forced skipping the pre-deletion check
  2. The engineer had over-granted access rights to critical infrastructure
  3. Critical infrastructure didn’t have explicit deletion/change protection
  4. Critical infrastructure wasn’t managed through the normal process of using version-controlled infrastructure-as-code (IaC)
2. 2 hours were required to restore the service due to DNSSEC misconfigurations. Why could PKB not fix this misconfiguration earlier?
  1. Not enough internal documentation and expertise for restoring DNSSEC from the ground up manually
  2. DNS setup is not fully controlled by code as the registrar does not offer good support for automation

| ## <span style="color: #ffffff">Follow-Up Activities & Mitigations</span> |
| --- |

Mitigations for RCAs:

1. Preventing accidental deletions and erroneous modifications of critical infrastructure: We will introduce a series of controls. Any of these would likely prevent this accident from reoccuring, but we expect benefits that have broader effects
  1. We will set up controls that will make ‘autoapprove’ irrelevant.
  2. 
    1. No engineer users (personas) will have permanent write access to production infrastructure. We are going to rely on service accounts (SA) that have task-specific access permissions. Engineers will need to elevate their permissions by impersonating SAs with narrow, specific access permissions when carrying out infrastructure work.
    2. In extreme cases when engineers themselves need elevated permissions, we will grant those temporarily (specifying an expiration timestamp) using IaC
  3. We will turn on deletion protection for all critical infra
  4. We will invest more into moving the rest of the system into IaC
2. DNS configuration will be documented and automated.
  1. DNSSEC configuration documentation will be documented.
  2. We will move to a domain registrar that allows automated DNS configuration so that the whole DNS infrastructure can be captured in IaC