File System Concepts: Storage, Management, and Operations

Posted on Sep 9, 2025 in Computers

File System Fundamentals

A file system manages how data is stored and retrieved on disk.

Disk Organization

  • A disk is a sequence of fixed-size blocks.
  • Only two primary operations: read(k) and write(k).

Core File System Requirements

  1. Store large amounts of data persistently.
  2. Retain data after processes terminate.
  3. Allow multiple processes to access the same data concurrently.

File Types and Structures

Common File Types

  • Executable: Programs that can be run.
  • Text: Human-readable character sequences.
  • Archive: Collections of files, often compressed.

File Structures

  • Byte Sequence: Simple stream of bytes (e.g., .txt files).
  • Record Sequence: Structured data as a series of records (e.g., databases).
  • Tree: Hierarchical structure (e.g., XML files).

File Operations

  • Create: Establish a new file.
  • Delete: Remove an existing file.
  • Open: Prepare a file for access.
  • Close: Release file resources.
  • Read: Retrieve data from a file.
  • Write: Store data into a file.
  • Append: Add data to the end of a file.
  • Seek: Change the current read/write position.
  • Get/Set Attributes: Manage file metadata (e.g., permissions, size).
  • Rename: Change a file’s name.

A read-only attribute prevents write operations on a file.

Directory Structures

Types of Directory Structures

  • Single-Level: All files reside in one main directory.
  • Hierarchical Tree: A tree-like structure with subdirectories, common in UNIX-style systems.

Pathnames

  • Absolute Path: Starts from the root directory (e.g., /home/user/document.txt).
  • Relative Path: Starts from the current working directory (e.g., ../notes.txt).

Directory Operations

  • Create: Make a new directory.
  • Delete: Remove an empty directory.
  • OpenDir: Open a directory for reading its contents.
  • CloseDir: Close an opened directory.
  • Readdir: Read entries from an opened directory.
  • Rename: Change a directory’s name.
  • Link: Create a new link (hard link) to an existing file.
  • Unlink: Remove a filename entry from a directory. The file is only deleted if no more hard links exist.

File System Layout: Storage Regions

Disk space is divided into several regions for file system management:

  • Boot Block: Contains bootstrap code, used during system startup.
  • Superblock: Stores critical file system information (e.g., block size, number of inodes, file system state).
  • Free Space Bitmap/List: Tracks which disk blocks are available for use.
  • Inodes or FAT: Structures that hold metadata and pointers to data blocks for files.
  • Data Blocks: Store the actual content of files.

File Allocation Methods

Different strategies for allocating disk blocks to files:

  • Contiguous Allocation:
    • Stores a file in back-to-back blocks.
    • Pros: Fast sequential access.
    • Cons: Difficult to resize files, prone to external fragmentation.
  • Linked List Allocation:
    • Each block contains a pointer to the next block in the file.
    • Pros: Easy to expand files.
    • Cons: No random access, slow sequential access due to scattered blocks.
  • File Allocation Table (FAT):
    • A table in memory tracks the chain of blocks for each file.
    • Pros: Faster random access than simple linked lists.
    • Cons: The FAT itself can consume significant RAM for large disks.
  • Inode-based Allocation (UNIX Style):
    • An inode (index node) holds file metadata and pointers to data blocks.
    • Pros: Scalable, supports large files efficiently.
    • Mechanism: Uses direct pointers for small files and indirect pointers (single, double, triple) for larger files to point to blocks of pointers.

Directory Implementation

How directory entries are stored and managed:

  • Inline Metadata: File metadata (e.g., size, permissions) is stored directly within the directory entry.
  • Inode Reference: Directory entries simply point to an inode, which contains the actual file metadata.
  • Long Names: Short filenames might be stored inline; longer names often use a pointer to a heap or separate storage area.

Shared Files: Hard Links

Hard links allow multiple directory entries (names) to point to the same inode.

  • The file is only deleted when all hard links pointing to its inode are removed.
  • When the link count for an inode reaches zero, the system reclaims the inode and its associated data blocks.

Journaling File Systems

Journaling enhances data integrity and recovery.

Why Journaling?

Prevents data loss and file system corruption if a system crash occurs during a write operation.

How Journaling Works

Write operations are first logged to a journal (a special area on disk) before being applied to the actual file system. This ensures atomicity and recoverability.

Evolution of Journaling in Linux File Systems

  • EXT2: A non-journaling file system, fast but risky for data integrity during crashes.
  • EXT3: Introduced journaling capabilities to EXT2, improving reliability.
  • EXT4: An optimized journaling file system, addressing startup issues and improving performance over EXT3. It is the default for many Linux distributions.

Deleting Files in UNIX

The process of removing a file in UNIX-like systems involves several steps:

  1. Remove the entry (filename and inode number) from its parent directory.
  2. If no other hard links point to the inode, release the inode to the free inode pool.
  3. If the inode is released, return all associated data blocks to the free block list.

Virtual File System (VFS)

Purpose of VFS

The Virtual File System (VFS) provides a common application programming interface (API) that abstracts away the differences between various file system types.

  • It makes diverse file systems (e.g., EXT4, FAT32, NTFS) appear uniform to the operating system and applications.
  • Enables plug-and-play compatibility for different file systems.

Free Space Management

Methods for tracking available disk blocks:

  • Bitmap:
    • Uses a bit array where 1 indicates a used block and 0 indicates a free block.
    • Pros: Fast to scan for free blocks.
    • Cons: Can consume significant memory for very large disks.
  • Linked List:
    • Maintains a linked list of all free blocks.
    • Pros: Low memory overhead.
    • Cons: Slow to search for a contiguous block of a specific size.

Disk Quotas

Disk quotas allow system administrators to set limits on the amount of disk space or the number of files that individual users or groups can consume.

  • Prevents single users from monopolizing disk resources.
  • Managed via quota tables maintained by the file system.

Backup Systems

Strategies for data protection and recovery:

  • Full Backup: Copies all selected files every time the backup runs.
  • Incremental Backup: Only backs up files that have changed since the last full or incremental backup.
  • Purpose: Used for disaster recovery, data migration, or recovering from user errors.
  • Tracking Changes: Bitmaps are often used to efficiently track which files have been modified.

File System Consistency

Ensuring the integrity of the file system structure.

Common Issues

  • Orphaned Blocks: Data blocks that are marked as used but are not linked to any file or directory.
  • Duplicate Blocks: The same data block is accidentally linked to multiple files or multiple times within the same file.

FSCK Tool

The FSCK (File System Consistency Check) utility detects and fixes corruption within a file system.

  • It checks inodes, block lists, and link counts to identify and repair inconsistencies.

File System Performance

Optimizing speed and efficiency of file operations.

Buffer Cache

A portion of memory that stores recently accessed disk blocks.

  • Helps speed up read operations by serving data from RAM instead of disk.
  • Reduces disk I/O and wear.

Modified LRU Policy

A variation of the Least Recently Used (LRU) algorithm used in buffer caching.

  • Considers factors like whether a block is likely to be reused soon or if it’s critical for system integrity.

Disk Layout Optimization

Techniques to improve disk access times:

  • Cylinder Groups: Grouping related inodes and data blocks into logical “cylinder groups” on the disk.
  • Benefit: Reduces seek time by minimizing the movement of the disk head, as related data is physically closer.

Real File System Examples

  • FAT32 (File Allocation Table 32):
    • Pros: High cross-platform compatibility (Windows, macOS, Linux).
    • Cons: Lacks modern features like permissions, limited to 4GB maximum file size (though 2GB is a common practical limit for older tools/APIs), and 2TB partition size.
  • NTFS (New Technology File System – Windows):
    • Features: Stores all file system metadata in a Master File Table (MFT).
    • Supports: Robust permissions, journaling, encryption, compression, large file and partition sizes.
  • EXT2 (Second Extended File System – Linux):
    • Characteristics: Non-journaling, known for its speed.
    • Risk: Prone to data loss or corruption during unexpected shutdowns.
  • EXT3 (Third Extended File System – Linux):
    • Enhancement: Added journaling capabilities to EXT2, improving data integrity.
  • EXT4 (Fourth Extended File System – Linux):
    • Improvements: More efficient journaling, addresses startup issues, and reduces journaling overhead compared to EXT3.
    • Status: Default file system for many modern Linux distributions.
  • Btrfs (B-tree File System – Linux):
    • Features: An optional Linux file system offering advanced features like snapshots, checksums, and flexible metadata support.
  • HFS+ (Hierarchical File System Plus – Apple):
    • Compatibility Note: Generally not directly compatible with Windows without third-party tools.

Advanced File System Concepts

UNIX File Permissions

Permissions are stored as three octal digits, representing access for the User (owner), Group, and Other.

  • Values: Read (4), Write (2), Execute (1).
  • Example 1: 700 means owner has read, write, execute (4+2+1=7); group and others have no permissions (0).
  • Example 2: 664 means owner has read/write (4+2=6); group has read/write (4+2=6); others have read (4).
  • Permissions are saved consistently across machines.

File Table

A per-process table that tracks all files currently opened by that specific process.

Paging File and Swap Space

Mechanisms used by the OS to extend available RAM:

  • Paging File: A special file on disk used when physical RAM runs out. The OS temporarily moves parts of memory (pages) from RAM to this file, acting as virtual memory backup. This prevents system crashes due to insufficient RAM.
  • Swap Space: A dedicated partition or reserved section on disk, specifically used by the OS to store memory pages that do not fit in RAM. It functions as an extension of RAM.

Mount Partitions

A mount partition is a separate section of the disk (a partition) that has been assigned its own mount point within the file system hierarchy. This allows the operating system to access the files and directories on that partition.

Backwards Compatibility

The ability of newer systems or software to work with older standards or data formats. Often, maintaining backwards compatibility can introduce complexities or make systems more prone to errors with new standards (e.g., 64-bit systems interacting with older 32-bit components).

File Size Limits

  • FAT32: Maximum file size is 4 GB (though 2 GB is often cited for practical reasons or older APIs).
  • 64-bit Systems: Can support extremely large file sizes, typically up to 16 EB (exabytes).

NTFS Structure

All information in NTFS, including metadata, is stored in a single, highly structured Master File Table (MFT).

Disaster Recovery

Backup systems are crucial for recovering data from disasters or accidental user errors.