Digital Communication and Information Theory Fundamentals
T1 — Noiseless Digital Communication
Analog signal: Continuous value, varies continuously in time.
Digital signal: Discrete values only (e.g., 0/1).
ADC (Analog-to-Digital Converter): Sampling, then Quantization, then Encoding.
Sampling and Quantization
- Sampling: Periodic snapshots of the analog signal.
- Nyquist rate: Sample at least 2x the max frequency to reconstruct without loss.
- Quantization: Rounding each sample to the nearest discrete level.
- Quantization error/noise: Real value vs. assigned level; imperceptible with enough bits.
Bits per level (round UP if not integer): n = log2(L)
Nyquist Bit Rate (noiseless max): R = 2 x B x log2(L) bits/s
Latency and Throughput
Latency = propagation + transmission + queuing + processing delay.
Propagation time = distance / speed (cable/fiber approx 2.4×10^8 m/s).
Transmission time = bits / bandwidth.
Throughput = actual achieved rate, always ≤ bandwidth.
Confusion: Bandwidth = ceiling (capacity); throughput = real rate. They are not the same.
T2 — Noisy Communication and Power
Shannon’s Model
Source → Transmitter/Encoder → Channel (+Noise) → Receiver/Decoder → Destination.
Encoding = adding controlled redundancy so the receiver can detect/correct errors.
Noise Types
- Thermal (Johnson-Nyquist): Random electron motion from temperature; in every device; power proportional to T x B; sets minimum detectable signal (sensitivity).
- White noise: Flat PSD (equal power, all frequencies). Models thermal noise well.
- Pink/flicker (1/f): From material defects; PSD proportional to 1/f (more energy at low frequencies); affects long-term stability.
- AWGN: Additive (adds to signal) + White (flat PSD) + Gaussian (normal-distributed amplitude). Standard model.
Power, dB, and dBm
Electrical power: P = VI = R x I^2 = V^2/R
Relative decibels: N(dB) = 10 x log10(P1/P2)
dBW: Absolute power, ref. 1 W. dBm: Absolute power, ref. 1 mW.
dBm = dBW + 30
Why dB? Gains/losses add/subtract instead of multiply/divide, simplifying link budgets.
Capacity with Noise
Shannon Capacity (with noise): C = B x log2(1 + S/N)
Confusion: Nyquist assumes NO noise; Shannon capacity is the real limit WITH noise. Shannon is always ≤ Nyquist.
Random Variables
Assigns a number to each outcome of a random process.
Discrete RV (x): Digital, 0 or 1. Continuous RV (x(t)): Analog, any value over time.
T3 — Information Theory
Origin: Boolean logic / relay switches (on/off) led to the digital bit.
Self-Information (“Surprise”)
I(X) = -log2(p(X)) bits
Rare (low-probability) events carry MORE information; certain events carry NONE.
If p=1 (certain), I=0: no surprise, no information.
Entropy
- Entropy (average information of a source): H(X) = -sum of p_i x log2(p_i)
- Binary entropy function: H(p) = -p x log2(p) – (1-p) x log2(1-p)
- Max entropy (binary) = 1 bit, at p=0.5 (most unpredictable).
- Entropy = 0 when outcome is certain (p=0 or p=1).
- Information = what resolves uncertainty. Predictable source = low entropy; equiprobable = high entropy.
Conditional and Mutual Information
Conditional entropy H(Y|X): Uncertainty left about Y after knowing X.
Mutual information: I(X;Y) = H(X) – H(X|Y)
Channel capacity: C = max of I(X;Y)
Confusion: Entropy is NOT “the message itself” – it’s the average uncertainty of the source.
T5 — Design and Lifecycle of a System
Problem and Requirements
Understand the problem before solving it (“55 min on the problem, 5 on the solution”).
Requirements describe WHAT the system must do, never HOW. Hierarchical, verifiable.
Case: Mars Climate Orbiter (1999): lost due to units mismatch (imperial lbf-s vs SI N-s). Poorly defined requirements/interfaces = major cause of cost overruns.
V-Model
Each design phase is paired with a corresponding test phase.
Verification: “Did we build it right?” Validation: “Did we build the right thing?”
Popular for complex communication systems (rigorous testing).
Methodologies
- Waterfall: Sequential phases; rigid, costly late changes.
- Agile: Short iterative cycles, customer feedback; hard to estimate cost.
- Lean: From manufacturing; max customer value, eliminate waste.
- Design Thinking: User-centered ideation process.
No methodology is universally “best” – depends on context. Tools: Kanban, Scrum, Six Sigma.
Lifecycle and “Ilities”
PLM: Concept → design → development → production → deployment → operation/maintenance → decommissioning.
“Ilities”: Reliability, scalability, modularity, quality, safety, sustainability – non-functional quality attributes.
Case: Iridium (1990s): deployed entire constellation upfront, bankrupt 1999 (about 5 billion dollar loss). Lesson: staged/phased deployment reduces financial risk, often Pareto-optimal (cost-risk-performance trade space).
T4 — Quantum Communication
Why a New Physics
Classical physics failed: blackbody radiation, photoelectric effect, atomic stability.
Solved by: energy is quantized – discrete packets, not continuous (Planck, E = h x nu).
Quantum Weirdness
- Wave-particle duality: Photon behaves as wave or particle depending on observation.
- Superposition: System in multiple states at once, until measured.
- Measurement/collapse: Measuring disturbs the system, collapses it to one state.
- Entanglement: Correlated particles at any distance (“spooky action”). Does NOT allow faster-than-light communication.
Qubit state: |psi> = alpha|0> + beta|1>, where |alpha|^2 + |beta|^2 = 1
Qubit = superposition of 0 and 1 until measured; collapses with probability |alpha|^2 / |beta|^2.
No-Cloning Theorem
An unknown quantum state cannot be copied exactly. Consequences: no quantum backups, no classical-style repeaters, basis of QKD security.
QKD and BB84
QKD distributes a SECRET KEY – does NOT encrypt the message itself.
Security is information-theoretic (physics), not computational.
Needs 2 channels: quantum (photons) + public classical (basis comparison).
BB84: Alice encodes bits in photon polarization, 2 random bases; Bob measures with random basis; publicly compare WHICH basis (not value); keep matches = sifting (about 50%).
Why Eavesdropper is Detectable
Key idea: Eve doesn’t know the correct basis, so she measures wrong about 50% of the time, disturbing the qubit (measurement collapses it). This introduces detectable errors (QBER). Secure if QBER < 11% (target 3-5%).
Quantum Threat and Response
Shor’s algorithm (quantum computer) could break RSA.
Response: QKD + post-quantum cryptography (PQC, e.g., CRYSTALS-Kyber).
Classical capacity (Shannon) vs quantum capacity bound (Holevo bound).