Core Design Principles and Architectural Models for Distributed Systems

Posted on Nov 21, 2025 in Computers

Distributed Systems Architectures (CPSC 5520, Fall 2022)

Instructor: Kevin Lundeen

Design Considerations in Distributed Systems

This section covers the fundamental criteria and challenges involved in designing robust distributed systems.

Design Criteria vs. Motivation

Compare this design considerations topic to the motivations presented in the Overview Lecture (slide 16):

Key Design Criteria

  1. Support Sharing of Resources

    Examples:

    • Cloud-based shared storage and files
    • Peer-to-peer assisted multimedia streaming
    • Shared services (e.g., outsourced mail systems and CDNs)
  2. Distribution Transparency

    Make the system appear as a single centralized system to the user.

  3. Openness

    Be able to interact with services from other open systems, irrespective of the underlying environment. This requires:

    • Well-defined interfaces
    • Easy interoperability
    • Portability
    • Extensibility
  4. Scalability

    Scalability has many dimensions:

    • Size
    • Geography
    • Administration

Design Criteria Ranking

How would you rank these four criteria?

  • Scalability
  • Distribution Transparency
  • Sharing of Resources
  • Openness

Criterion: Distribution Transparency

Distribution transparency aims to hide the distributed nature of the system.

Two Kinds of Clients

  • High Level: Hide distribution from users.
  • Low Level: Hide distribution from software.

Distribution Transparency Categories

Can you think of examples of each category? How would you rank their importance?

Degree of Transparency

Aiming at full distribution transparency may be too much because:

  • Communication latencies cannot be hidden.
  • Completely hiding failures of networks and nodes is (theoretically and practically) impossible.
  • You cannot distinguish a slow computer from a failing one.
  • You can never be sure that a server actually performed an operation before a crash.

Furthermore, full transparency often costs performance, potentially exposing the distribution of the system:

  • Keeping replicas exactly up-to-date with the master takes time.
  • Immediately flushing write operations to disk is required for fault tolerance.

Exposing distribution may be beneficial:

  • Making use of location-based services (e.g., finding nearby friends).
  • When dealing with users in different time zones.
  • When it makes it easier for a user to understand what is happening (e.g., reporting a server as failing when it does not respond for a long time).

Distribution transparency is a nice goal, but achieving it is a different story, and it should often not even be aimed at.

Criterion: Scalability

Many developers of modern distributed systems easily use the adjective “scalable” without making clear why or how their system actually scales.

Three Components of Scalability

  1. Size Scalability: Number of users and/or processes.
  2. Geographical Scalability: Maximum distance between nodes.
  3. Administrative Scalability: Number of administrative domains.

Observation: Most systems account only, to a certain extent, for size scalability. Often, the solution involves multiple powerful servers operating independently in parallel. Today, the primary challenge still lies in geographical and administrative scalability.

Size Scalability Challenges

Root causes for scalability problems with centralized solutions:

  • The computational capacity, limited by the CPUs.
  • The storage capacity, including the transfer rate between CPUs and disks.
  • The network between the user and the centralized service.

The common solution is to decentralize through:

  • Partitioning data and computations.
  • Replication and caching.

Geographical Scalability Challenges

Systems cannot simply transition from a Local Area Network (LAN) to a Wide Area Network (WAN).

Issues include:

  • Many distributed systems assume synchronous client-server interactions (client sends request and waits for an answer). Latency may easily prohibit this scheme over WANs.
  • WAN links are often inherently unreliable. Simply moving streaming video from LAN to WAN is bound to fail.
  • Lack of multipoint communication, meaning a simple search broadcast cannot be deployed. The solution is to develop separate naming and directory services (which introduce their own scalability problems).

Administrative Scalability

Design Challenges in Distributed Systems

  1. Pitfalls: Based on prejudices evolved from building single-node systems.
  2. Failures: No longer atomic nor simple to model and reason about.
  3. No Global Knowledge: Data consistency is not a given. Accurate knowledge of actual time (Clock) is not a given. Sharing knowledge requires Inter-Process Communication (IPC).
  4. Dynamic Topology: Need to be able to add and remove components. The system must still work if failed components are removed, and integrate replacements when brought in.
  5. Security.
  6. Development: Notoriously difficult to test, verify, stage, deploy, and debug distributed systems.

Design Pitfalls: False Assumptions

Many distributed systems are needlessly complex, caused by mistakes that required patching later on. Many false assumptions are often made, including:

  • The network is reliable.
  • The network is secure.
  • The network is homogeneous.
  • The topology does not change.
  • Latency is zero.
  • Bandwidth is infinite.
  • Transport cost is zero.
  • There is one administrator.

Handling Failures

You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done.

— Leslie Lamport
Failure Management

Failure is the norm, not a rare event! In local systems, failure is usually all-or-nothing. In distributed systems, we experience partial failure (e.g., sending a request but not getting a response).

Design must deal with detection, recovery, and restart.

  • Availability: The fraction of time the system is usable. Achieved with redundancy, but consistency becomes an issue.
  • Reliability: Data must not get lost. This includes security considerations.
Types of Failures
Fail-Stop
The failed component stops functioning. Ideally, it notifies other components first. Halting means stopping without notice. Detection is often done via timeouts, but timeouts are unreliable if network latency is variable or the network is unreliable. We mark the component “dead” if it times out.
Fail-Restart
The component stops but then restarts. Danger: stale state.
Omission
Failure to send or receive messages (e.g., queue overflow in routers, corrupted data, receive buffer overflow).
Timing
Messages take longer than expected. We may assume a system is dead when it is not. Unsynchronized clocks can alter process coordination (e.g., mutual exclusion, timestamped log entries).
Partition
The network fragments into two or more sub-networks that cannot communicate with each other.
Byzantine Failures
Instead of stopping, a component produces faulty data. This can be due to bad hardware, software, network problems, or malicious interference.

The overall goal is to avoid single points of failure.

Challenge: No Global Knowledge

Nobody has the true global state of a distributed system because there is no shared memory.

A process only knows its current state. It may know the last reported state of other processes, and it may periodically report its state to others. There is no foolproof way to detect failure in all cases.

Components of Local Knowledge

State
Current values of the data, network connection info, process memory, list of clients with open files, lists of which clients finished their tasks.
Replicas
Redundant copies of data.
Cache
Local storage of frequently-accessed data to reduce latency.
Clock
Local time reference.

Challenge: Security

The distributed environment introduces security complexities:

  • Public networks
  • Remotely-managed services
  • Third-party services

Common Security Issues

  • Malicious interference, bad user input, impersonation of users and services.
  • Protocol attacks, input validation attacks, time-based attacks, replay attacks.

Security relies on authentication, encryption, and good programming practices. Users also demand convenience, such as single sign-on and controlled access to services.

Distributed System Architectures

Architectural models are built upon fundamental tools, ideas, services, and peer relationships.

Architectural Tools

Inter-Process Communication (IPC) Approaches

  • Shared Memory
  • Messages
    • Bus-based messaging (various topologies)
    • Internet protocols: TCP/IP and UDP/IP

Data Distribution Techniques

  • Partitioning
  • Replication
  • Caching

Distributed Algorithms

Analysis of algorithms helps design scalability, including algorithms for:

  • Coordination
  • Cooperation
  • Fault Tolerance and Recovery

Messaging Tools: IP

Anything non-local usually uses TCP/IP or UDP/IP, typically implemented with a socket library to access the operating system’s network protocol stack. This originated at UC Berkeley in 1983 for their version of the Unix OS, implementing the file Abstract Data Type (ADT): “Everything is a file.”

Architectural Ideas and Concepts

Established Concepts

Abstraction
APIs, Layers/Tiers, Operating Systems, Middleware.
Structured Programming
Procedures/Functions, Remote Procedure Calls (RPC).
Object-Oriented Design
Encapsulation/Information Hiding, Remote Objects/RMI.
Uniformity and Simplicity
Resource-Based Systems (RESTful APIs), Publish/Subscribe.

Emerging Concepts

  • Service-Based Architectures
  • Network of Peers
  • Structured and Unstructured Peer-to-Peer Topologies

Layered Architectures

Functionality is broken into multiple layers. Each layer handles a specific abstraction, hiding implementation details and specifics of hardware, OS, network abstractions, and data encoding.

Tiered (Multi-Tier) Architectures

Tiered architectures are the distributed systems analogy to a layered architecture. Each tier (layer):

  • Runs as a network service.
  • Is accessed by surrounding layers.

The classic client-server architecture is a two-tier model:

  • Clients: Typically responsible for user interaction.
  • Servers: Responsible for back-end services (data access, printing, etc.).

Service-Based Architectures

Monolithic Centralized Model

This model involves no networking (e.g., a traditional time-shared system, single workstation/PC, or direct connection of multiple terminals to a computer). It uses one or several CPUs but is not easily scalable due to limiting factors like the number of CPUs and contention for resources (memory, network, devices).

Client-Server Model (Many-to-One Topologies)

Clients send requests to servers. A server is a system that runs a service, is typically always on, and processes requests from clients. Clients generally do not communicate directly with other clients (e.g., FTP, email, web).

Roles in Client-Server
  • Server: Listens and accepts many clients.
  • Client: Initiates requests and is served.

Servers can act as clients to other services, and clients of one service can be servers of others (multi-tiered centralized system architectures).

Nature of the Service

Services can involve:

  • Data
  • Computation
  • Objects
  • In general, any resource

Peer-to-Peer (P2P) Models (One-to-One Topologies)

P2P models involve machines (peers) communicating directly with each other, with no reliance on centralized servers for core operations.

P2P Goals

  • Robustness: Expect that some systems may be down.
  • Self-Scalability: The system can handle greater workloads as more peers are added.

Examples include BitTorrent and Skype.

Overlay Networks

P2P systems utilize overlay networks, which can be categorized as:

  • Structured Overlay Topologies
  • Unstructured Overlay Topologies
  • Hybrids (Combinations of Client/Server and P2P, and/or combinations of Structured/Unstructured Overlay Nets)

Structured P2P Topologies

Structured P2P often uses a semantic-free index where each data item is uniquely associated with a key, which is used as an index. Common practice is to use a hash function, implementing a Distributed Hash Table (DHT).

key(data item) = hash(data item’s value)

The P2P system is responsible for storing (key, value) pairs.

Example: Chord

Nodes are logically organized in a ring, and each node has an m-bit identifier. Each data item is hashed to an m-bit key. The data item with key k is stored at the node with the smallest identifier id ≥ k, called the successor of key k. The ring is extended with various shortcut links to other nodes.

Unstructured P2P Topologies

Each node maintains an ad hoc list of neighbors, and the resulting overlay resembles a random graph.

Searching Methods
Flooding
The issuing node passes the request for data d to all neighbors. The request is ignored if the receiving node has seen it before. Otherwise, the node searches locally for d (recursively). This may be limited by a Time-To-Live (TTL), a maximum number of hops.
Random Walk
The issuing node u passes the request for data d to a randomly chosen neighbor, v. If v does not have d, it forwards the request to one of its randomly chosen neighbors, and so on. Random walks are more communication efficient but might take longer to find the result.

Hybrid and Specialized Architectures

Hybrid Models

Many P2P architectures still rely on servers for tasks like lookup, tracking users/peers, tracking content, and coordinating access. However, traffic-intensive workloads are delegated to peers.

Super-Peer Networks

It is sometimes sensible to break the symmetry in pure P2P networks:

  • Having index servers improves performance when searching in unstructured P2P systems.
  • Deciding where to store data can often be done more efficiently through brokers.

Edge-Server Architecture

Systems deployed on the Internet where servers are placed at the edge of the network (at the boundary between enterprise networks and the actual Internet).

Processor Pool Model

A collection of CPUs that can be assigned processes on demand (e.g., render farms).

Cloud Computing

Resources are provided as a network (Internet) service.

Software as a Service (SaaS)
Remotely hosted software (e.g., Salesforce.com, Google Apps, Microsoft Office 365).
Infrastructure as a Service (IaaS)
Compute + storage + networking (e.g., Microsoft Azure, Google Compute Engine, Amazon Web Services).
Platform as a Service (PaaS)
Deploy and run web applications without setting up the infrastructure (e.g., Google App Engine, AWS Elastic Beanstalk).
Storage
Remote file storage (e.g., Dropbox, Box, Google Drive, OneDrive).

***

The End of Architectures