[{"content":"","date":null,"permalink":"/tags/databases/","section":"Tags","summary":"","title":"Databases"},{"content":"Paper: OLTP Through the Looking Glass, and What We Found There by Stavros Harizopoulos, Daniel J. Abadi, Samuel Madden, and Michael Stonebraker (SIGMOD 2008).\nOLTP features # On-disk data structures for table storage Heap files and B-tree support Multiple concurrent queries via locking-based concurrency control Log-based recovery Efficient buffer management This was in the 70s where the main memory was little compared to the DB and it was so expensive to run, costing hundreds of thousands or millions.\nIn today\u0026rsquo;s situation, it is trivial to get a good system for cheap and mostly the full OLTP DB can be in the memory. Institutions have their own network-clustered machines operating with hundreds of gigabytes of memory.\nOS and network conferences: with the rise of the internet, there are more database-like system proposals than the full suite.\nalternative DBMS architectures #Optimizing OLTP for main memory is a good idea when it fits in the RAM.\nLog-less DB: recovery is not needed or it recovers from other sites in a cluster (Harp, Harbor and C-Store) Single-threaded databases Transaction-less databases: transactions are not needed in many systems. Most distributed systems prefer eventual consistency than actual. Or lightweight forms of transaction like all reads are done before writing How well will all of this perform if built?\nmeasuring the overheads of OLTP #Shore (DB). Benchmarked with a subset of TPC-C benchmark. In their environment, the initial transactions per second (TPS) is 640.\nThey try to remove features to check the increase in performance. At the end, they had a tiny kernel with query processing code that could process around 12.7K TPS. The kernel is single-threaded, lock-free, main memory DB without recovery.\nWhat did they remove to get this jump:\nLogging: helps in recovery and slows things down but if there are other means of recovery then we can remove it Locking: traditional two-phase locking poses a sizeable overhead since all accesses to DB are governed by lock manager Latching: in a multi-threaded DB, many data structures have to be latched before access. Removing this feature and going to a single-threaded approach has a noticeable performance impact Buffer management: main memory DB does not need to access pages through a buffer pool trends in OLTP #Cluster computing: RDBMS were originally written for shared memory multi-process systems. And added support for shared disk architecture. Gamma-style shared-nothing DBs.\nMemory-resident databases: because of increase in RAM capacity in these systems, we have more room to play with and most of the OLTP systems fit within a clustered system\u0026rsquo;s RAM. Optimized indices, eschewing disk-optimized tuple formats and page layouts.\nSingle threading in OLTP systems: dig on transactions and it is fast now compared to earlier and latching commands slowing down the performance. Making it single-threaded will help with latching reduction to zero and it will affect transactions, but they argue if there are no disk writes because of memory-resident systems, it will be fast as there will just be index lookups and etc. The other idea was to consider each DB in each thread with other threads in the same systems as clusters to stand for throughput concerns for transaction systems.\nHA vs logging\nTransaction variants: this is about eventual consistency and that is preferred in most systems for performance to trade off availability. Less strict forms are available and if the two-phase requirement is stripped down to their exact needs it would be like perform all of their reads before any of their writes.\nMIT H-Store: remove these features and there was two order of magnitude speedup in transaction throughput.\nShore #Scalable Heterogeneous Object Repository. To be typed, persistent object system borrowing from both file system and object-oriented DB.\nLayers: type system, Unix compatibility, language heterogeneity.\nThe storage manager SSM had all the features of a DBMS like concurrency control and recovery with two-phase locking and WAL and B-trees. The project ended in the 1990s but the SSM was developed more and updated.\nShore architecture #It has managers for everything: lock manager, log manager, buffer manager.\nAll invocations happen through a concurrency layer. Shared objects are accessed through latching. Latch is similar to locks and lightweight and comes with no deadlock detection mechanism.\nThread support: Shore used its own user-level, non-preemptive thread package derived from NewThreads, providing a portable OS interface API. Since it is user-level it is single process and multiplexes the threads. For this performance experiment, we are using single-threaded process and removing the multi-threads feature.\nLocking and logging: two-phase locking, transactions with ACID. Supports hierarchical locking. WAL. LSN (Log Sequence Number).\nBuffer manager: other modules read and write pages through this. A page read is issued by the fix method call to buffer manager.\nRecording a record involves:\nLocking the record (and page, per hierarchical locking) Fixing the page in the buffer pool Computing the offset within the page of the record\u0026rsquo;s tag Reading a record is performed by issuing a pin/unpin method call.\nremoving Shore components #Modifications:\nRemove logging: log-less architecture Remove locking (when applicable): partitioning, commutativity Single thread, remove locking and latching: one transaction at a time Remove buffer manager, directory: main memory resident Avoid transaction bookkeeping: transaction-less databases Steps taken:\nRemoving logging Removing locking or latching Removing latching or locking Removing core related to buffer manager Optimizations: B-tree key evaluation, accelerate directory lookups, increase page size to avoid frequent allocation, remove transactional sessions Removals are done through code block removal or with if-else statements. If it adds too much overhead, then they rewrite the methods.\nperformance study #OLTP workload #Benchmark is derived from TPC-C: models the wholesale parts supplier operating out of a number of warehouses and their associated sales districts. Each database for each warehouse is 100 MB. Involves mix of 5 concurrent transactions of different types and complexity (NewOrder, Payments, Delivering Orders, Checking Status of Orders, Monitoring Stock). This paper experimented with the first 2 transactions.\nNew Order: places an order for 5-15 items, the ask was changed a little bit than the mandate of 90% local warehouse + 10% remote. Eased to 100% local warehouses. We don\u0026rsquo;t abort any transaction and if we find an invalid one, we abort without redoing changes in DB.\nPayment: lightweight transaction that updates balances and creates a history record.\nsetup and measurement methodology #Single-core Pentium 4, 3.2 GHz with 1 MB L2 cache, hyperthreading disabled, 1 GB RAM running Linux 2.6. Use iostat to monitor disk activity and verify main memory residency of the DB.\nFor detailed instruction and cycle count, they instrument the benchmark application code with calls to PAPI library which gives access to performance counters.\nMicro-architectural behavior:\nInstruction count: deterministic Cycle count: susceptible to various parameters experimental results #Shore platform: single-threaded executing one transaction at a time, memory-resident, never flushed to disk.\n11 different switches to remove functionality from Shore.\nEffect on throughput:\nNo I/O: avg of 80 microseconds per transaction or 12,700 TPS The useful work in this is about 22 microseconds per transaction or 46,500 TPS. This difference is because of the complexity in code removal and some parts of the transaction code is left within the call stack. Out of the box Shore: with logging, 620 TPS. Without logging, 1,700 TPS. Modifications yield a significant improvement by a factor of 20. Payment transaction #Reduction in instructions as we optimized:\nB-tree key eval: standard practice in high-performance DBMS Removed logging: affects mainly commits and updates. These two removed 18% of the instruction count Locking: removed 25% instruction count, for pin and unpin Latching: 13% instructions, create record and B-tree lookups Buffer management: 30% total instruction count, now directly uses malloc Operations:\n3 B-tree lookups 3 pin/unpin ops 3 updates (through B-tree) 1 record creation Commit call After the optimizations, the remaining kernel requires about 5% of the total initial instruction count, which is about 6 times the total instructions of the optimal system.\nAll 6 components are significant Until all optimizations are applied, the reduction in instruction count is not dramatic New Order transaction #Operations:\n13 B-tree inserts 12 record creations 11 updates 23 pin/unpin ops 23 B-tree lookups Optimizations:\nB-tree key eval: 16% of the total instructions Logging and locking: 12% and 16% of the instructions Buffer management: 14% reduction in total instructions. Biggest win. Tuning the page size so that frequency of page allocation decreases and B-tree depth is reduced. implications for future OLTP #Concurrency control: 19% of cycles of the overhead from this, and we need to find the best way to do this as the simulated methods in the past had disk-based loads with disk stalls in mind. Redoing the simulation for newer systems and hardware might help the case.\nMulti-core support: many-core machines are prevalent, to handle the transactions with concurrent cores with a single site. For this, latching is required. The overhead for latching is 10%. You can make it better with new implementations. Or virtualize each core as its own machine at DBMS or OS layer, or exploit intra-query parallelism.\nReplication management: active-active approach, nearly instantaneous failovers. Two-phase commits and transaction consistency causes a lot of latency and leads to more failover time because of active-passive.\nWeak consistency: eventual consistency and paying for consistency errors in a non-technical way.\nCache-conscious B-trees: they did not use this and this would yield a modest performance boost.\nconclusion #Performance impact of Shore components under the TPC-C benchmark while being stripped of instructions and functionalities yielded serious performance gains up to a factor of 20. Major contributors are buffer management, logging and latching optimization.\nThe performance of a memory database without transactions or logging is better than a conventional database that could fit in RAM. When you provide a stripped-down system (single-threaded, fits in memory, reduced functionality), the performance is orders of magnitude better than an unmodified system. ","date":"23 July 2026","permalink":"/posts/oltp-through-the-looking-glass/","section":"posts","summary":"notes on Harizopoulos et al\u0026rsquo;s paper on measuring the overhead of traditional OLTP components","title":"OLTP through the looking glass"},{"content":"","date":null,"permalink":"/tags/papers/","section":"Tags","summary":"","title":"Papers"},{"content":"","date":null,"permalink":"/posts/","section":"posts","summary":"","title":"posts"},{"content":"","date":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"Notes on systems programming, binary exploitation, databases, and math.\n","date":null,"permalink":"/","section":"xphract","summary":"","title":"xphract"},{"content":"","date":null,"permalink":"/tags/systems/","section":"Tags","summary":"","title":"Systems"},{"content":"Paper: The Linux Scheduler: a Decade of Wasted Cores by Jean-Pierre Lozi, Baptiste Lepers, Justin Funston, Fabien Gaud, Vivien Quema, and Alexandra Fedorova (EuroSys 2016).\nWhat is TPC-H? An org that audits benchmarks for DB, storage engines. Frequently mentioned in research papers. DBT-3 (open source version) helps you run the same tests. TPC-H gives you QphH (queries per hour via geometric mean of power and throughput). Useful in capacity planning.\nThey built their own tools for identifying these bugs while scheduling.\nWhat is Dennard Scaling? Because of increasing complexity of the surrounding systems (hardware, cache), the implementation of the scheduler is becoming more complex and maybe more bloaty.\nIt fails in a very basic function: runnable threads use idle cores.\nThis paper discusses 4 performance bugs. A common symptom: the scheduler unintentionally and for a long time leaves cores idle while there are runnable threads waiting in runqueues.\nTools built: sanity checker (detects invariant violations).\nthe Linux scheduler #CFS on single CPU #CFS (Completely Fair Scheduling) is based on Weighted Fair Queueing (WFQ) scheduling algorithms. It time-slices the CPU among the running threads.\nDecisions made in the scheduler:\nHow to determine a thread\u0026rsquo;s timeslice? How to pick the next thread to run? The scheduler defines a fixed time interval in which every thread should run at least once. The interval is divided among threads with respect to weight. The resulting interval is the timeslice for that thread.\nThread\u0026rsquo;s weight is based on priority/niceness. When a thread runs, it accumulates vruntime (runtime/weight).\nThe thread gets pre-empted:\nWhen vruntime exceeds the timeslice Another smaller vruntime thread is awoken If there are other threads available on the CPU Threads are organized as a runqueue (red-black tree), sorted in increasing order of their vruntime. The CPU chooses the leftmost node in the tree, which contains the smallest vruntime.\nHow do I see the vruntime or the runqueue?\nCFS on multi-core #CFS becomes quite complex. Per-core runqueue. Context switches will be expensive if we go away from local runqueue to global shared runqueue across cores.\nRunqueues should be balanced across cores. If there are 2 cores and one has N high priority threads and the other core has 1 low priority thread, the high priority threads will have less CPU time than the low priority thread if each core only sees their local runqueue. To solve this problem, they introduced the load balancer.\nLoad balancing is expensive in communication and compute. When runqueues are unbalanced, cores go idle. Scheduler tries not to run the load balancer and goes to great lengths to avoid it. Goal is to have lower overhead and save power.\nload balancer algorithm #It will be bad to load balance based on weights and numbers alone. We need a metric to balance so that we balance the CPU time.\nLoad balancing with respect to weights would make CPU idle when high priority threads sleep and get more CPU time, which might lead to work stealing. Or if each runqueue has the same number of threads, one is low priority and another high priority. Looking at it this way, we see that this is not what we want. We don\u0026rsquo;t want the same CPU time for low and high priority threads.\nCFS balances runqueues based on a load metric: thread weight and average CPU utilization. This thereby helps us balance threads based on load. If the thread is not on CPU a lot, then it would have less load.\nIn multithreading in different processes, say one process with a lot of threads and another with few threads. This will make it such that the first process with lots of threads will starve the other process. To make sure there is fairness, there is a feature for this: scheduling groups.\nScheduling groups, scheduling domains.\nbugs #group imbalance bug #This happens because of the problem of taking load average of the scheduling group to assign or steal work. This makes the lowest thread in that group free and thus it is not fully utilized, causing inefficiencies among the group.\nIt is caused because the load metric of the group is the average. The simple solution is to take the lowest load in the group. Finding this would be in the same time as finding the average. So in terms of complexity, it is the same. But we get the performance improvement.\nWhen the NAS Benchmark was run, lu ran 13x faster after the fix.\nscheduling group construction bug #This happens because of the way scheduling groups are constructed and load balanced.\nIn the default cases of group construction with respect to node 0, there will be nodes that are two hops apart in some NUMA systems, which can be used for binding the application using tasksets. Now because they might also be in both groups, this will make the average probing of load to balance not effective for more work as the load averages will be the same in both groups. Making it a bug.\nThe fix is so that each node can form these groups with overlap in mind and not have the nodes in different groups, making this bug obsolete from the root cause.\noverload-wakeup bug #Because of cache optimizations, a short-term sleep thread may be woken up on a core other than a long-term idle core. Which makes the system non-optimal in performance.\nThe wakeup might not be power efficient as the long-term idle cores may be in a low power state and waking up means going on full throttle.\nFix: add this as a factor to choose and schedule work. Choose long-idle cores before short-term idle cores when needed.\nmissing scheduler domain bug #This bug occurs when a core gets deactivated and the number of cores gets updated but the scheduler still schedules in the past groups. Meaning it schedules on the old cores for new work. Significantly overloading the single core or group.\nFix: add the function that updates the group on deactivation, which was missed in the code refactor.\nshould we change the design of the Linux scheduler? #No, it has had immense engineering throughout the years. The high algorithmic complexity made it slower. CFS is useful for multithreaded workloads with changing hardware and software. Now, they are changing the load metric calculation to be made simpler in terms of code complexity.\nMeasurement of these bugs is not possible or a pain with traditional performance monitoring tools as these are not panic problems that the system shows but slow performance-eating bugs.\ntools #sanity checker #It checks for idle cores and waiting threads in the runqueue. It does not assert because it allows short-term discrepancy as the scheduler will fix it. So the tool runs to check in S interval and after a violation it profiles the events for M time. Systemtap can also be used to profile but it has performance implications compared to the in-house tool.\nThis tool gives us the bugs that eat away at performance.\nscheduler visualization tool #Helps us see the events visually throughout the paper to see the bug. The overload-wakeup bug!\nIt gets the runqueue, load and core usage events and stores them in a global array for further visuals. Each event is 20 bytes and it has no performance implications.\nlessons learned #There were a lot of changes and new ideas proposed for the scheduler by a lot of researchers and community. But they couldn\u0026rsquo;t be included in the scheduler safely. We need to think about a scheduler design that allows it to add more things safely as the hardware grows/changes. The proposed way is to have modules like main and optimization modules that help the main module schedule when possible. The optimization module may be cache-based, power-based, memory-based.\nAnother lesson is visualization should be more adopted and standardized to find more of these bugs.\nrelated work #performance bugs # The Linux kernel performance project LITMUS kernel correctness #RacerX, Eraser. Like these tools that detect system crashes, we need these tools to extend and detect performance bugs. The big issue in this arena is to know that the short invariant violation is acceptable in our system. But long-term violation isn\u0026rsquo;t. So we need tools that should keep track of that.\nModel checking is used to detect bugs in kernels. Useful to find bugs before they happen. It can detect crash bugs but not subtle performance bugs.\nFormal verification: proving software correctness by analyzing source code. They work by describing the system with states, pre and post conditions. It works in single-threaded. And it does not work yet with the long-term invariant violation detection. Having Linux developers write formal specs will also be a hurdle.\ntracing #Tracing can be helpful in finding these bugs, but the sheer volume of data is overwhelming.\nEvent Tracing in Windows DTrace (Solaris, FreeBSD, macOS) SystemTap, Ftrace, KernelShark (Linux) The online sanity checker used SystemTap to find the traces and events to build the bug cases with their visualization tool.\nconclusion #Scheduler (dividing CPU cycles among threads) is not a solved problem. And the bugs that arise from these implementations are hard to detect because of the subtlety of them. We see the problem of runnable threads stuck in the runqueue while there are idle cores. The Linux scheduler also violates the basic work-conserving invariant. They mention that more tools and fixes will be available.\n","date":"16 July 2026","permalink":"/posts/linux-scheduler-wasted-cores/","section":"posts","summary":"notes on Lozi et al\u0026rsquo;s paper on performance bugs in the Linux CFS scheduler","title":"the linux scheduler: a decade of wasted cores"},{"content":"Paper: Quantifying the Cost of Context Switch by Chuanpeng Li, Chen Ding, and Kai Shen (ExpCS 2007).\nMeasuring the indirect cost of the context switch is a challenge. This paper does it in an experimental setup through synthetic workloads and also shows the impact of OS background interrupt handling on the measurement.\nintroduction #Multitasking system: context switch is switching of the CPU from one thread to another. It makes multitasking possible but also causes system overhead.\ndirect cost # Process registers need to be saved and restored The OS kernel code (scheduler) needs to execute TLB entries need to be reloaded Process pipeline must be flushed indirect cost # Cache sharing between multiple processes causes performance degradation Varies for different memory access patterns and workloads measurement approach #Ousterhout measured the direct cost using a benchmark with 2 processes communicating with 2 pipes. McVoy measured using lm-bench.\nThis paper uses pipe communication to implement frequent context switches between 2 processes.\nC1: direct cost of the context switch using Ousterhout method where the process makes no data access C2: total cost of context switch where each process allocates and accesses an array of floating point numbers Indirect cost = C2 - C1 direct cost measurement (C1) #Two processes send single byte messages to each other via two pipes. During each round-trip, there are 2 context switches + one read and write syscall in each process. Measure the time cost of 10K round-trip communications (T1).\nSingle process simulating 2-process communication by sending a single byte message to itself via one pipe. Each round trip includes one read and one write syscall. Measure the time cost of 10K round-trip communications (T2).\nDirect time cost per context switch: C1 = T1/20K - T2/10K\ntotal cost measurement (C2) #Same as before but with array access before writing message to the other process, then blocking on the next read operation.\nS1: execution time of 10K round-trip communications between two test processes S2: execution time of 10K simulated round-trip communications Total time cost per context switch: C2 = S1/20K - S2/10K\nParameters changed during different runs: array size, access stride.\navoiding potential interference #We need to avoid interference from other processes or background interrupt handling of the OS.\nUse a dual processor machine. Background handlers default to one processor, we use the other for experiments and simulations. Real-time scheduling with max priority. Linux syscalls sched_setaffinity() and sched_setscheduler() are used to effect the design. time measurement #High resolution timer that relies on a counting register in the CPU. It measures the time by counting the number of cycles the CPU has gone through since startup. We measure the cost of a large number of context switches and then report the average to offset the overhead of the timer itself.\nexperimental results #Machine: IBM eServer with dual 2.0 GHz Intel Pentium Xeon CPUs. L2 cache: 512 KB, cache line size: 128 B. OS: Linux 2.6.17 kernel with Redhat 9. Compiler: gcc 3.2.2 (no optimization for compilation).\nAverage direct context switch cost (C1): 3.8 microseconds.\nTotal cost per context switch (C2): several microseconds to thousands.\neffect of data size # Each process sequentially traverses an array of size D between context switches. Each process does read, write or RMW on each array element.\nArray size 1-200 KB: context switch 4.2 to 8.7 microseconds. The entire dataset combined between both processes fits in the L2 cache.\nArray size 256-512 KB: context switch 38.9 to 203.2 microseconds. The dataset does not fit in the L2 cache for both processes but still fits for the simulated process. Upon each context switch, the process needs to refill the L2 cache with its own data. Additional costs also from frequent cache warmups. At array size 384 KB, the costs start to differ based on data access type.\nArray size \u0026gt; 512 KB: context switch is even more. The dataset does not fit in the cache for both cases. We also get cache interference. There are cache misses. Interference manifests as cache warmups in the case of context switches. Non-linear effect of cache sharing.\neffect of access stride # Each process accesses an array of floating point numbers in a stride pattern and does RMW operation on each element.\nArray size 32-128 KB: datasets fit in cache, no difference in context switch cost when we change the stride.\nWhen the datasets don\u0026rsquo;t fit in the cache, there will be a cost of context switch with respect to stride:\nStride 8 B: cost 44.1 - 183.9 microseconds Stride 128 B: cost 133.8 - 1496.1 microseconds The data access pattern can affect the cost of context switch significantly. Stride affects the cache warmups.\neffect of experimental environment # Dual processor: with the design of deploying the processes with real-time scheduling priority Single processor: same setup but with disabled multiprocessor support in Linux kernel It was run on a quiet environment and external noisy environment. There was interference in the environment.\nsummary #The indirect cost of context switches ranges from several microseconds to more than one thousand microseconds. When the data size is larger than the L2 cache, the cost of context switches increases monotonically with the data size and also based on access stride. The larger the stride, the more the cost.\n","date":"9 July 2026","permalink":"/posts/quantifying-context-switch-cost/","section":"posts","summary":"notes on Li et al\u0026rsquo;s paper on measuring the indirect cost of context switches","title":"quantifying the cost of context switching"},{"content":"Paper: Crash-Only Software by George Candea and Armando Fox (USENIX HotOS 2003).\nWritten to drive home the point of crash-only software. Most of the bugs in large scale systems are due to transient or intermittent bugs.\nVariety of ways to stop: shutdown clean, panic, hang, crash, lose power, etc.\nCrash reboots are faster than clean reboots based on the paper\u0026rsquo;s claim. If software is crash-safe, why support additional, non-crash mechanisms for shutting down? For higher performance. The usual design is to value steady performance more than robustness and recoverability.\nMicro-reboots to improve availability of a soft state (state can be built from other sources) system that was crash-safe. In this paper they advocate crash-only (crash safety + fast recovery).\nIn a crash-only system, stop = crash and start = restart.\ncrash-only property #Idempotent power off/on switch and they are outside the component\u0026rsquo;s functionality code and not relying on the component\u0026rsquo;s internal behavior. For large systems, the small component switches are wired together.\nExample: kill -9 in unix processes.\nThe component crasher gives immense confidence. Recovery code deals with exceptional situations and runs on every startup of the system.\nDesign of crash-only is inspired from transactions in data storage and retrieval systems.\nfailure handling #SSM: crash-only hashtable-like session state store.\nCrash-only system makes it such that every detected failure goes as component level crashes. Questionable, but it really is simple. But how do we make sure we get traces and everything, and also this assumes there will always be a cascading failure. It might be that it can go rogue for that I/O. What about in cases of critical components such as a scheduler?\nThis changes the reality as faults equate to crashes and recovery states for restarting. This could very well work where the restart is very cheap. We can also follow a micro-reboot strategy for suspect components before they fail. It can restart when it notices fail stutter behavior, based on predictive math models of software aging.\nproperties of crash-only software # All non-volatile states of the components must be kept in dedicated state stores and they must be crash-only The system as a whole should be fault tolerant as they need to tolerate neighbor component failure or downstream component failure and crashes Strong modularity Timeout-based communications Lease-based resource allocation Self-describing requests that carry TTL and information on whether they are idempotent Restart/retry architecture intra-component properties #All important non-volatile state is managed by dedicated state stores, leaving application with just program logic.\nSpecialized state stores:\nRelational and object data stores File system appliances Distributed data structures Non-transactional hashtables Session state stores App becomes stateless, which will make it more idempotent and easy to recover from crashes. Example: 3-tier internet architecture, where middle layer is stateless and relies on backend databases to store data.\nstate stores need to be crash-only #Most of them are crash-safe, but not crash-only as they recover slowly. Many products offer knobs to fine-tune the recovery time through factors. Example: Oracle DBMS allows for more frequent checkpointing.\nPure crash-only state store is Postgres, which uses non-overwriting storage and maintains all data in single append-only log. More reliable state stores are required for building fast, simple applications.\nBerkeley DB: storage system supporting B+tree, hash and record abstractions. 4 interface levels from no concurrency control/no transaction/no disaster recovery to multi-use, transactional API with logging, fine-grained locking with data replication.\nState store types:\nACID stores (common DBs) Non-transactional persistent stores (destor for user profiles) Session state store (SSM for shopping carts) Simple read-only store (file system appliances for static HTML) Soft state stores (web caches) inter-component properties #Because they are crash prone, all components need to tolerate failure of their neighbors. We need to decouple components from each other, from the resources they use, and from the requests they process.\nComponents have externally enforced boundaries: strong fault containment, thorough isolation kernels, VMs, task-based intra-JVM isolation, OS processes (Denali isolation kernel) All interactions between components have a timeout. If the timeout is exceeded for response, the component is deemed failed and reported to the recovery agent for further reboots and restarts. The components are not fail-stop, but the failure containment is improved. All resources are leased. Infinite timeouts or leases are not acceptable. Requests are entirely self-describing: state and context are explicit. Idempotency and TTL are also included. restart/retry architecture #Component infers failures of a peer component based on raised exception or a timeout. Then the component is reported to the recovery agent for recovery. It tries to restart or reboot it. We assume idempotency for all component requests. We send the retryAfter for all the components that are trying to use it. This hides the component failure as latency before it comes up again.\nTimeout-based failure detection is supplemented with traditional heartbeats and progress counters.\nWe can propagate recovery times through retryAfter exceptions. This can be estimated by recovery agent, components and others. We can configure a max retry in the global application policy with lease duration and timeouts.\nWe can use a stall proxy to prevent new requests from entering while recovering.\ndiscussion #Not easy to build crash-only systems but not impossible. See the ecosystem of J2EE and its usage of component-based architecture. In order to be HA, the application needs to be idempotent. This does not handle the byzantine failures or data errors. In such cases we can use triple modular redundancy or clever state replication.\nIn today\u0026rsquo;s systems, fast recovery is obtained by overprovisioning and counting on rapid failure detection to trigger failover. Crash-only is complementary to this approach. Fast recovery means less redundancy. Crash-only can also reintegrate recovered components automatically and faster.\nPerformance/throughput loss still stands as a problem. They expect it will evolve and solve the problems.\nconclusion #By using crash-only, better reliability, HA in internet systems. Fault modes are simplified. The goal is to get a system that is recursively restartable. The perceived reliability of the system is increased because the failures are recovered in fast times because of crash-only components.\n","date":"2 July 2026","permalink":"/posts/crash-only-software/","section":"posts","summary":"notes on Candea and Fox\u0026rsquo;s crash-only software paper","title":"crash-only software"},{"content":"I work in security and build things with code. I like understanding how systems work at the lowest level and breaking things to learn how they fail.\nThis site is where I write up what I\u0026rsquo;m studying. Not polished tutorials. Working notes, lab writeups, exploit walkthroughs, paper breakdowns.\nwhat I\u0026rsquo;m working on # Binary exploitation (ROP, heap, kernel) OS internals (MIT 6.S081) Linear algebra Database systems (CMU 15-445) tools #Arch Linux, qtile, alacritty, tmux, neovim.\nfind me # GitHub x [at] xphract [dot] dev ","date":null,"permalink":"/about/","section":"xphract","summary":"","title":"about"}]