Security Logging — Explain what logs are useful for security (auth, network, application) (conceptual)

Historical Context & Motivation

Long before the term "cybersecurity" entered common parlance, system administrators recognized the value of recording what happens on their machines. The earliest Unix systems in the 1970s wrote simple text records of user logins and process events to flat files, primarily for debugging and capacity planning rather than for any security purpose. As networked computing expanded through the 1980s and 1990s, these humble records evolved into structured security logs — timestamped records of system events that analysts could review to detect unauthorized access, investigate incidents, and satisfy regulatory requirements. Today, security logging is not merely a best practice but a foundational requirement enshrined in frameworks such as NIST SP 800-92, PCI DSS, and HIPAA, reflecting the consensus that you cannot defend what you cannot observe.

This historical trajectory reveals a persistent question that security professionals still grapple with: which events, from which sources, should we capture to maximize our ability to detect, investigate, and respond to security incidents while keeping storage and processing costs manageable? Answering this question requires a principled understanding of the three primary log domains — authentication, network, and application — and the distinct security insights each provides.

Core Principles of Security Logging

Effective security logging rests on a set of foundational principles that determine what to record, how to protect it, and how to use it. These principles are technology-agnostic: whether you are instrumenting a Kubernetes cluster or a legacy mainframe, the same conceptual framework applies. Understanding these principles allows you to evaluate any logging architecture and identify gaps before an adversary exploits them.

The Three Pillars — A Visual Overview

Security-relevant logs can be broadly categorized into three pillars: authentication logs, network logs, and application logs. Each pillar answers a different set of questions about what happened in your environment. When these three data streams converge in a SIEM or log analytics platform, they produce a composite view that dramatically reduces the ambiguity an analyst faces during triage and investigation. The following diagram illustrates how each pillar feeds into a centralized collection point and the distinct security questions it addresses.

Notice that each pillar captures a fundamentally different dimension of activity. Authentication logs answer who questions (identity, credentials, access level). Network logs answer where and how questions (source/destination addresses, protocols, volumes). Application logs answer what questions (business logic execution, data access, errors). An attacker who exfiltrates a database record will leave traces across all three domains: an authentication event when they log in, a network flow when the data leaves the perimeter, and an application audit entry when the query executes. Viewing any single pillar in isolation may reveal only an innocuous-looking event; only the composite picture exposes the adversarial intent.

How Security Logging Works — The Log Pipeline

Security logging is not a single technology but a pipeline composed of several stages: generation, collection, normalization, storage, analysis, and retention. Each stage introduces design decisions that affect the fidelity and utility of the logs downstream. Understanding this pipeline is essential because a failure at any stage — an application that does not emit sufficient detail, a collector that drops events under load, or a retention policy that deletes evidence too early — can render the entire investment in logging moot.

The Six-Stage Log Pipeline

The normalization stage is particularly critical in heterogeneous environments. A firewall may label a denied connection as action=drop while an IDS records the same concept as result=blocked. Without parsing these into a common schema — such as the Open Cybersecurity Schema Framework (OCSF) or the Elastic Common Schema (ECS) — automated correlation rules will miss matches that a human analyst would immediately recognize. Normalization also involves enrichment: appending geolocation data to IP addresses, resolving hostnames, and mapping user identifiers to directory entries.

Deep Dive — Authentication, Network & Application Logs

Authentication Logs

Authentication logs record every attempt to prove identity to a system, whether that system is an operating system, a web application, or a cloud identity provider. On Linux, the primary sources are /var/log/auth.log (Debian-family) or /var/log/secure (Red Hat-family), capturing PAM events, SSH sessions, and sudo invocations. Windows environments generate Security Event Log entries, with Event IDs 4624 (successful logon) and 4625 (failed logon) being among the most security-critical identifiers any Windows analyst should know. Cloud providers like AWS emit authentication records through CloudTrail's ConsoleLogin and AssumeRole events. These logs are invaluable for detecting brute-force attacks, credential stuffing, lateral movement via pass-the-hash, and unauthorized privilege escalation.

Network Logs

Network logs capture the movement of data across infrastructure boundaries. The most common sources include firewall logs (recording allow/deny decisions), NetFlow or IPFIX records (summarizing traffic flows by source, destination, port, and byte count), DNS query logs, web proxy access logs, and intrusion detection/prevention system (IDS/IPS) alerts. Network logs are especially powerful for identifying command-and-control (C2) beaconing, data exfiltration, lateral movement, and reconnaissance scanning. A DNS query log that shows a workstation resolving a domain generated by a domain generation algorithm (DGA) is often the earliest indicator of a malware infection — long before application logs reveal any anomaly. Similarly, NetFlow records can reveal low-and-slow data exfiltration that bypasses signature-based controls by analyzing volume and timing patterns.

Application Logs

Application logs are generated by software running on top of the operating system — web servers (Apache/Nginx access and error logs), databases (MySQL slow-query logs, PostgreSQL pg_audit logs), custom business applications, and middleware such as message queues and API gateways. These logs capture the semantics of user interaction with business logic: which records were queried, which files were downloaded, which API endpoints were called, and which errors occurred. From a security perspective, application logs are indispensable for detecting injection attacks (SQL injection payloads in query logs), authorization bypass (a user accessing resources outside their role), and business-logic abuse (such as exploiting a race condition in a payment workflow). Because application developers control what gets logged, the security value of application logs varies enormously — an application that logs only "request processed" provides virtually no forensic value, whereas one that logs the authenticated user, the requested resource, the input parameters, and the response code becomes a rich source of investigative data.

Worked Example — Tracing an Intrusion Across Log Sources

Consider the following scenario: your organization's SOC receives an alert that a user account jdoe has been flagged for anomalous behavior. The alert was generated by a SIEM rule that detected the account accessing a sensitive database during off-hours. Walk through how an analyst would use each log type to reconstruct the attack chain.

Strengths and Limitations of Each Log Type

No single log type provides a complete picture. Each pillar has inherent strengths and blind spots, and a mature security operations program explicitly maps these gaps and compensates through multi-source correlation. The table below summarizes the key trade-offs that influence logging architecture design.

Connection to Advanced Security Operations

The conceptual understanding of security logging presented in this lesson is the starting point for several advanced disciplines within security operations. As you progress, you will encounter technologies and methodologies that extend the raw capabilities of log collection into sophisticated detection and response frameworks. It is useful to understand where basic logging ends and these advanced topics begin.

Importantly, all of these advanced technologies are only as effective as the log data they consume. A SIEM with incomplete authentication logs will produce blind spots. A UEBA model trained on poorly normalized data will generate excessive false positives. The conceptual foundations covered in this lesson — understanding which logs matter, what they contain, and how to preserve their integrity — remain the critical substrate upon which every advanced capability is built.

Practice Problems

Lesson Summary