crash-only software

Paper: Crash-Only Software by George Candea and Armando Fox (USENIX HotOS 2003).

Written to drive home the point of crash-only software. Most of the bugs in large scale systems are due to transient or intermittent bugs.

Variety of ways to stop: shutdown clean, panic, hang, crash, lose power, etc.

Crash reboots are faster than clean reboots based on the paper’s claim. If software is crash-safe, why support additional, non-crash mechanisms for shutting down? For higher performance. The usual design is to value steady performance more than robustness and recoverability.

Micro-reboots to improve availability of a soft state (state can be built from other sources) system that was crash-safe. In this paper they advocate crash-only (crash safety + fast recovery).

In a crash-only system, stop = crash and start = restart.

crash-only property #

Idempotent power off/on switch and they are outside the component’s functionality code and not relying on the component’s internal behavior. For large systems, the small component switches are wired together.

Example: kill -9 in unix processes.

The component crasher gives immense confidence. Recovery code deals with exceptional situations and runs on every startup of the system.

Design of crash-only is inspired from transactions in data storage and retrieval systems.

failure handling #

SSM: crash-only hashtable-like session state store.

Crash-only system makes it such that every detected failure goes as component level crashes. Questionable, but it really is simple. But how do we make sure we get traces and everything, and also this assumes there will always be a cascading failure. It might be that it can go rogue for that I/O. What about in cases of critical components such as a scheduler?

This changes the reality as faults equate to crashes and recovery states for restarting. This could very well work where the restart is very cheap. We can also follow a micro-reboot strategy for suspect components before they fail. It can restart when it notices fail stutter behavior, based on predictive math models of software aging.

properties of crash-only software #

• All non-volatile states of the components must be kept in dedicated state stores and they must be crash-only
• The system as a whole should be fault tolerant as they need to tolerate neighbor component failure or downstream component failure and crashes
• Strong modularity
• Timeout-based communications
• Lease-based resource allocation
• Self-describing requests that carry TTL and information on whether they are idempotent
• Restart/retry architecture

intra-component properties #

All important non-volatile state is managed by dedicated state stores, leaving application with just program logic.

Specialized state stores:

• Relational and object data stores
• File system appliances
• Distributed data structures
• Non-transactional hashtables
• Session state stores

App becomes stateless, which will make it more idempotent and easy to recover from crashes. Example: 3-tier internet architecture, where middle layer is stateless and relies on backend databases to store data.

state stores need to be crash-only #

Most of them are crash-safe, but not crash-only as they recover slowly. Many products offer knobs to fine-tune the recovery time through factors. Example: Oracle DBMS allows for more frequent checkpointing.

Pure crash-only state store is Postgres, which uses non-overwriting storage and maintains all data in single append-only log. More reliable state stores are required for building fast, simple applications.

Berkeley DB: storage system supporting B+tree, hash and record abstractions. 4 interface levels from no concurrency control/no transaction/no disaster recovery to multi-use, transactional API with logging, fine-grained locking with data replication.

State store types:

• ACID stores (common DBs)
• Non-transactional persistent stores (destor for user profiles)
• Session state store (SSM for shopping carts)
• Simple read-only store (file system appliances for static HTML)
• Soft state stores (web caches)

inter-component properties #

Because they are crash prone, all components need to tolerate failure of their neighbors. We need to decouple components from each other, from the resources they use, and from the requests they process.

• Components have externally enforced boundaries: strong fault containment, thorough isolation kernels, VMs, task-based intra-JVM isolation, OS processes (Denali isolation kernel)
• All interactions between components have a timeout. If the timeout is exceeded for response, the component is deemed failed and reported to the recovery agent for further reboots and restarts. The components are not fail-stop, but the failure containment is improved.
• All resources are leased. Infinite timeouts or leases are not acceptable.
• Requests are entirely self-describing: state and context are explicit. Idempotency and TTL are also included.

restart/retry architecture #

Component infers failures of a peer component based on raised exception or a timeout. Then the component is reported to the recovery agent for recovery. It tries to restart or reboot it. We assume idempotency for all component requests. We send the retryAfter for all the components that are trying to use it. This hides the component failure as latency before it comes up again.

Timeout-based failure detection is supplemented with traditional heartbeats and progress counters.

We can propagate recovery times through retryAfter exceptions. This can be estimated by recovery agent, components and others. We can configure a max retry in the global application policy with lease duration and timeouts.

We can use a stall proxy to prevent new requests from entering while recovering.

discussion #

Not easy to build crash-only systems but not impossible. See the ecosystem of J2EE and its usage of component-based architecture. In order to be HA, the application needs to be idempotent. This does not handle the byzantine failures or data errors. In such cases we can use triple modular redundancy or clever state replication.

In today’s systems, fast recovery is obtained by overprovisioning and counting on rapid failure detection to trigger failover. Crash-only is complementary to this approach. Fast recovery means less redundancy. Crash-only can also reintegrate recovered components automatically and faster.

Performance/throughput loss still stands as a problem. They expect it will evolve and solve the problems.

conclusion #

By using crash-only, better reliability, HA in internet systems. Fault modes are simplified. The goal is to get a system that is recursively restartable. The perceived reliability of the system is increased because the failures are recovered in fast times because of crash-only components.