Operating System Security Fundamentals

Explain Operating System Security Architecture.

Explain Kernel Space vs. User Space.

Explain The System Call Interface as Attack Surface.

Explain The Principle of Least Privilege.

An operating system is not a monolithic block of code.

Modern OS design separates the software into distinct layers with carefully defined trust boundaries.

Operating System Security Fundamentals

An operating system is not a monolithic block of code.

Modern OS design separates the software into distinct layers with carefully defined trust boundaries.

Operating System Security Architecture connects to risk, controls, and evidence.

The most fundamental architectural distinction in OS security is between kernel space and user space .

The kernel is the core of the operating system — the privileged software that directly manages hardware, memory, processes, and input/output.

Code running in kernel space operates with full, unrestricted access to all system resources.

The kernel is the crown jewel of system security: if an attacker achieves kernel-level code execution, they effectively own the machine entirely.

Every time a user-space application needs to do something privileged — read a file, open a network connection, allocate memory — it must issue a system call (syscall).

The syscall interface is a carefully managed boundary between unprivileged code and the kernel.

It is also a significant attack surface: flaws in how the kernel handles syscalls can allow user-space attackers to escalate privileges to kernel level, a class of vulnerability…

The set of all entry points, data inputs, and interfaces that an attacker could potentially exploit is called the attack surface of a system.

The Principle of Least Privilege (PoLP) states that any user, process, or system component should operate with only the minimum permissions necessary to perform its legitimate…

The principle has profound implications for system design and administration: - A web server process should not run as root (or SYSTEM on Windows) — it needs only to read web…

- A database application user should not have administrative access to the database — it needs only to perform the specific queries required.

- An employee in the finance department should not have access to HR records — only to the financial systems their job requires.

Modern operating systems are multi-user environments.

Each user has an account with an associated set of permissions that determines what they can do on the system.

Access control for files and resources is implemented through several models:

Discretionary Access Control allows the owner of a resource to control who can access it.

On a DAC system (the default model for most commercial operating systems), the creator of a file can grant or deny access to other users at their own discretion.

Unix/Linux file permissions and Windows ACLs are both DAC implementations.

DAC is flexible but has weaknesses: it relies on users making good access control decisions, and malware running as a user inherits all of that user's permissions.

Mandatory Access Control removes discretion from resource owners.

Instead, a central security policy (enforced by the OS or a security module) determines access based on labels assigned to both subjects (users, processes) and objects (files,…

A process cannot access a resource unless the security policy explicitly permits it, regardless of what the resource owner wants.

MAC systems are common in high-security environments (government, military).

Role-Based Access Control assigns permissions to roles rather than directly to individuals.

Users are assigned to roles, and inherit that role's permissions.

An organization might have roles such as "Database Administrator," "Read-Only Analyst," and "HR Manager," each with different permission sets.

RBAC simplifies administration (especially in large organizations) and makes it easier to enforce least privilege.

Modern operating systems implement process isolation to prevent one process from interfering with another.

Each process runs in its own virtual address space — the OS provides each process with the illusion that it has the machine's entire memory to itself, mapped through a Memory…

In reality, the physical memory is shared, but the mapping ensures that process A cannot read or write process B's memory.

- Address Space Layout Randomization (ASLR): Randomizes the memory addresses at which system components (stack, heap, libraries) are loaded, making it harder for an attacker to…

Microsoft Windows is the dominant desktop operating system and a frequent target of attackers.

Understanding its security model is essential for any security professional.

The Windows Security Model connects to risk, controls, and evidence.

The Security Account Manager (SAM) is a database stored in the Windows registry that holds user account credentials.

Passwords are stored as hashed values (historically using NTLM hashing).

The SAM database is locked from access while Windows is running, but attackers have developed various techniques (pass-the-hash, SAM dumping via tools like Mimikatz) to extract…

Windows uses Access Control Lists to implement DAC.

- A SACL (System Access Control List): Controls which access events are audited (logged).

Windows permissions are highly granular — you can separately control Read, Write, Execute, Modify, Take Ownership, and many other permission types on any object.

User Account Control was introduced in Windows Vista as a mechanism to limit the damage that can be done by malware or by users making mistakes.

Even an administrator account does not run with full administrative privileges by default — when an action requiring elevated privileges is attempted, UAC prompts the user for…

Standard users are prompted for administrator credentials; administrators are asked to confirm.

UAC is not a security boundary — it is a consent mechanism.

Windows ships with built-in antivirus and antimalware capabilities through Windows Defender Antivirus (now part of the broader Microsoft Defender platform).

Modern versions provide real-time protection, cloud-based threat intelligence, exploit protection, and endpoint detection and response (EDR) capabilities.

Windows Security Center provides a centralized dashboard for monitoring the health of various security components.

BitLocker is Windows' full-disk encryption feature, available in Pro and Enterprise editions.

It encrypts entire volumes using AES encryption, protecting data at rest from attackers who gain physical access to the drive.

BitLocker can be configured to require a PIN at boot, rely on a Trusted Platform Module (TPM) chip, or use a USB recovery key.

Linux powers the majority of the world's servers, cloud infrastructure, network devices, and embedded systems.

Its security model is foundational knowledge for any security professional.

The Linux Security Model connects to risk, controls, and evidence.

Linux's traditional privilege model centers on the root account — the superuser with unrestricted access to everything on the system.

Running services as root is extremely dangerous: if any root-owned process is compromised, the attacker inherits root privileges.

Best practice is to run services as dedicated unprivileged users.

The sudo utility allows specific users to execute commands as root (or as another user) on a per-command basis, based on a policy defined in /etc/sudoers.

Properly configured, sudo allows administrators to perform privileged tasks without needing to log in directly as root, and creates an audit trail of privileged actions.

The principle of least privilege is enforced by granting sudo access only to specific commands, not to a shell.

Linux file permissions follow the classic Unix model: each file has an owner (a user), a group, and permissions assigned to three categories — owner, group, and other.

Permissions include read (r), write (w), and execute (x).

These are represented as octal values (e.g., chmod 644 sets owner read/write, group read, others read) or symbolic notation (e.g., chmod u+x).

Understanding and correctly configuring file permissions is a fundamental Linux security skill — misconfigured permissions are a common source of vulnerabilities.

SELinux (Security-Enhanced Linux) is a kernel security module that implements Mandatory Access Control on Linux.

Originally developed by the NSA, SELinux enforces a detailed policy that defines exactly what each process is allowed to do — which files it can read, which network ports it can…

A compromised web server running under SELinux policy is constrained to the specific file paths and resources it legitimately needs, dramatically limiting an attacker's ability to…

AppArmor is an alternative MAC system, used by default on Ubuntu and SUSE.

Operating System Security Architecture

Kernel Space vs. User Space

The System Call Interface as Attack Surface

The Principle of Least Privilege

Where Operating System Security Architecture applies.

Where Kernel Space vs. User Space applies.

How the difference changes the security decision.

Identify the asset or process at risk.

Choose a preventive, detective, or corrective control.

Explain what evidence would prove the control is working.

Define one core concept in plain language.

Compare two controls or threats from the week.

Apply one idea to a campus or business system.

Evaluate why a solution might fail in practice.

Collect evidence, not just screenshots.

Explain what the artifact proves.

Connect the proof back to risk and control selection.

Operating System Security Fundamentals

Security is a decision process, not just a tool list.

Use the reading to justify practical choices.