CUDA Programming and Distributed Systems Fundamentals

Posted on Jul 7, 2026 in Mathematics and Computer Science

CUDA and GPU Architecture

  • Core Intuition: Data parallelism (same operations on many independent data points).
  • GPU vs. CPU Cores: GPU cores are weaker individually but significantly more numerous (10,000+ in modern data centers).
  • CPU Strengths: Sequential, branchy, and latency-bound workloads.
  • CPU Silicon Investment: Branch predictors, caches, and deep pipelines.
  • Latency Hiding: GPUs use thread oversubscription (warp switching).
  • SIMT (Single Instruction, Multiple Threads): Allows per-thread program counters.
  • CUDA Kernel: Describes the execution of a single thread.
  • Warp Mechanics: Warp size is 32 threads; warp divergence leads to serialization.
  • Streaming Multiprocessor (SM): GPU unit containing CUDA cores, schedulers, registers, and fast memory.
  • Memory Hierarchy: Registers (thread-local) > Shared Memory (block-local) > Global Memory (slowest, host-accessible).
  • CUDA Programming: Uses __global__ qualifiers, nvcc compilation, and cudaDeviceSynchronize() for asynchronous management.
  • Grid-Stride Loops: Efficiently process multiple elements across threads.

Distributed Systems Principles

Fallacies and Engineering Reflexes

  • 8 Fallacies (Peter Deutsch): Includes assumptions like “the network is reliable” or “bandwidth is infinite.”
  • Engineering Reflexes: Expect loss, respect cost, and trust nothing.
  • Failure Types:
    • Crash: Node stops completely (easiest to handle).
    • Omission: Lost messages (requires ACKs and retries).
    • Byzantine: Arbitrary or malicious behavior (requires 3f+1 nodes).

Clocks and Causality

  • Physical Clocks: Subject to drift; require GPS-disciplined atomic clocks for microsecond accuracy.
  • Lamport Clocks: Capture causality but cannot detect concurrency.
  • Vector Clocks: Provide exact causality and concurrency detection at the cost of O(n) per message.

Consistency and CAP Theorem

  • CAP Theorem: Consistency, Availability, and Partition Tolerance. You can only choose two; partitions are inevitable.
  • Consistency Models:
    • Strong: Single-copy illusion.
    • Causal: Preserves causal order.
    • Eventual: Replicas converge over time.
  • Consensus: Guarantees agreement, validity, and termination.