Tools: The Forensic Black Box: Why Logs That Can Be Deleted Are Security Theater

Introduction: The Illusion of Observability ## Why Traditional Logs Fail Under Real Attacks ## The Forensic Requirement: Logs as Evidence, Not Data ## The Codecov Failure Mode ## The Forensic Black Box Model ## Implementation Pattern: WORM + Hash Chains ## Why SIEM Is Not Enough ## Forensics as a First-Class Security Control ## Bounding Blast Radius with Immutable Logs ## The Economic Argument ## Conclusion: A Log That Can Be Deleted Is Not a Log Most security teams believe they have “logs.” In reality, most organizations have rumors of past events stored in databases that attackers can modify, delete, or selectively corrupt after compromise. If an attacker gains privileged access to your cloud account, your SIEM, or your CI runner, the first thing they do is erase traces. Not because logs are unimportant — but because logs are dangerous to attackers. The Codecov breach persisted for months precisely because victims had no cryptographically reliable record of what actually executed in their pipelines. The question wasn’t just what was compromised — it was when, how often, and where the blast radius ended. Without immutable forensic memory, incident response becomes speculation. This article introduces the Forensic Black Box — a design pattern for making CI/CD activity tamper-evident and non-erasable, even under full cloud account compromise. Most logging pipelines share the same trust domain as the systems they observe: Once attackers control the environment, logs become attacker-controlled narratives. “What ran in our pipelines last month?” “Whatever we want you to think ran.” This is not observability. This is fiction. Logs should satisfy the same properties as legal evidence: If your logs can be modified by the same root account that runs your CI/CD, they are not evidence. They are suggestions. In the Codecov Bash Uploader breach: The root cause was not merely script compromise. It was the absence of an immutable execution record. There was no cryptographically trustworthy memory of: Without immutable logs, incident response cannot bound blast radius. It becomes guesswork. The Forensic Black Box treats logs as append-only forensic artifacts, not operational telemetry. A practical implementation of the Forensic Black Box uses: Even if attackers gain full cloud account control, they cannot retroactively alter the forensic record. SIEM systems aggregate logs for analysis. They are not designed as tamper-proof forensic memory. They are excellent for detection. They are poor for post-compromise truth. Forensic Black Boxes complement SIEMs. They do not replace them. Most security architectures treat forensics as an afterthought: “We’ll investigate if something happens.” Forensics must be designed before compromise. Once attackers control your environment, it is too late to build trustworthy logs. Immutable logging is not about compliance. It is about the ability to reconstruct reality after trust collapses. The value of immutable logs is not just knowing that you were compromised. This transforms incident response from open-ended damage assessment into bounded containment. Immutable logging costs money: Forensics is cheaper than ignorance. Security teams often say: “We’ll check the logs.” If your logs can be modified by attackers, you will be checking a story they wrote. The Forensic Black Box is the architectural decision to preserve truth in environments where trust is temporary. Logs are not telemetry. Logs are evidence. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse - CI logs stored in the same cloud account - Audit logs in the same IAM boundary - Application logs writable by root - SIEM pipelines authenticated by the same credentials attackers target - Append-only - Tamper-evident - Non-repudiable - Externally anchored - Survivable under full compromise of the primary environment - A single line of malicious code was injected into a widely used script - The script exfiltrated secrets - The attack persisted silently for months - Victims could not easily determine which pipeline runs executed the malicious version - Script versions - Execution times - Affected pipelines - Write-Once-Read-Many (WORM) storage - Hash chaining of entries - External retention enforcement - Separation of logging authority from execution authority - The system being logged will eventually be compromised. - The forensic system must survive that compromise. - Append-only object storage - Retention policies that even root cannot override - Cryptographic chaining between log entries - Produces a JSON record - Includes a SHA-256 hash of the previous record - Is written to WORM storage with a retention lock - Tamper evidence (hash chain breaks if modified) - Deletion resistance (retention policies prevent erasure) - Temporal integrity (attackers cannot rewrite history) - Depend on ingestion pipelines - Use mutable storage - Rely on credentials that attackers target - Prioritize query performance over immutability - Exactly which pipeline runs were affected - Exactly which artifacts were produced - Exactly which secrets may have leaked - Exactly when the compromise began - Engineering effort - Legal discovery - Regulatory fines - Breach notification - Customer churn - Long-term trust erosion