Audit Logging

An audit log is an immutable record of who performed what action, when, and from where within a system. It is an essential information foundation for investigating security incidents, meeting compliance requirements, and debugging system behavior.

Why Audit Logs Are Needed

When an incident occurs without audit logs in place, it becomes impossible to identify who performed an action or when it happened, making root-cause analysis and prevention extremely difficult.

Primary situations where audit logs are required:

• Security incidents: Investigating unauthorized access or insider threats
• Compliance: SOC 2, GDPR, ISO 27001, and similar requirements
• System debugging: Tracing the cause of unexpected behavior in production
• Access auditing: Periodic confirmation of who accessed which data and when

What to Log

Authentication Events

• Login success and failure
• Logout
• Password reset
• MFA (multi-factor authentication) enablement and disablement

Authorization Events

• Permission denial (every case where an access attempt is rejected)
• Role changes, grants, and revocations

Data Operation Events

• Access to sensitive data (reads)
• Data creation, update, or deletion (including before and after values)
• Data export or download

AI System-Specific Events

• Which user sent which prompt
• Model response (hash as needed)
• List of documents retrieved by RAG
• Token usage and cost

Audit Log Schema Design

Using a standardized JSON format makes ingestion into SIEM tools (described below) straightforward.

{
  "timestamp": "2026-05-13T10:30:00Z",
  "event_id": "evt_a1b2c3d4",
  "user_id": "u_12345",
  "user_email": "alice@example.com",
  "action": "rag_query",
  "resource": "document:contract-123",
  "resource_type": "document",
  "ip_address": "192.168.1.1",
  "user_agent": "Mozilla/5.0 ...",
  "result": "success",
  "metadata": {
    "query": "Changes to contract terms",
    "retrieved_doc_ids": ["doc-123", "doc-456"],
    "token_count": 1842
  }
}

Field reference:

Field	Required	Description
timestamp	Required	UTC time in ISO 8601 format
event_id	Required	Unique identifier for the log entry
user_id	Required	Identifier of the user who performed the action
action	Required	Type of action performed
resource	Required	Identifier of the resource operated on
result	Required	success / failure / error
ip_address	Recommended	IP address of the requester
metadata	Optional	Action-specific additional information

Log Pipeline Design

flowchart LR
    A[User action] --> B[Application]
    B --> C[Structured log output]
    C --> D[Append-only storage]
    D --> E[SIEM / analytics tool]
    E --> F[Alert notification]
    E --> G[Compliance report]

Log Immutability

Tampered audit logs lose their value as evidence. Techniques for ensuring immutability:

• Append-only storage: Object storage that prohibits overwrites and deletions (e.g., AWS S3 Object Lock)
• Hash chains: Include a hash of the previous log entry in the next entry to make tampering detectable
• Transfer to external storage: Real-time transfer to storage independent of the application server

Retention Period Requirements

Regulation / Standard	Required retention
GDPR	Duration of the processing purpose (6 months is a practical minimum)
SOC 2 Type II	Audit period (typically 12 months) plus additional buffer
ISO 27001	Per organizational policy (1–3 years is common)
PCI DSS	12 months (most recent 3 months must be available online)

Log Management Tool Comparison

Tool	Highlights	Best for
AWS CloudWatch	AWS-native, easy setup	AWS-centric infrastructure
Datadog	Feature-rich, real-time analytics	Medium to large SaaS
Splunk	Enterprise standard, powerful search	Large enterprises, finance, government
Elasticsearch + Kibana	OSS, highly customizable	Teams capable of self-hosting

Alerting on Suspicious Patterns

The following patterns should trigger immediate alerts:

• Repeated login failures in a short time: 5+ per minute → possible brute force attack
• Abnormally large data access: 10× the normal record view volume → possible insider threat or account compromise
• Off-hours access: Sensitive data access outside normal business hours
• Geographically implausible access: Same user accessing from different countries within a short time

Privacy Trade-offs

Logging too much accumulates large amounts of PII (personally identifiable information), creating a risk of violating GDPR and other privacy regulations.

Log subject	Recommended approach
User email address	Record user ID only; manage separately
AI prompt content	Record only a hash or summary
Passwords and tokens	Never log these
IP address	Mask as needed (e.g., zero the last octet)

Frequently Asked Questions

Q: Should I log the actual AI responses?

It depends on the use case. If compliance requirements exist (evidence of what was output is needed), logging is necessary. However, since responses may contain PII, it is recommended to run PII detection and masking before recording. Storing only a hash of the response for later tamper verification is also an effective design.

Q: How long should I keep audit logs?

It depends on the applicable regulations and standards (see the retention period table above). For organizations considering SOC 2 compliance, a minimum of 12 months is recommended. Without that requirement, 6 months is a practical lower bound. Use low-cost cold storage such as AWS S3 Glacier for long-term retention.

Q: What is the difference between application logs and audit logs?

Application logs serve primarily as debugging records, containing error and performance information. They are kept for a short period (1–4 weeks is common) in variable formats. Audit logs serve security and compliance purposes, recording who did what in a standardized format for long-term retention. Their purpose, retention period, and access controls should each be managed independently.

See the references for the external specifications and background sources used on this page.[1][2][3][4]

References

1. SOC 2 Type II Requirements
2. GDPR Article 30 — Records of Processing Activities
3. AWS CloudTrail Documentation
4. OWASP Logging Cheat Sheet