For a long time, storage devices were much slower than CPUs. A hard disk had to physically move a read/write head and wait for the correct sector to rotate underneath it. That mechanical delay dominated everything else, so software did not need an especially efficient way to keep the device busy. SSDs removed the moving parts. SATA SSDs became much faster, but requests from multiple CPU cores still pass through one shared command queue. The cores must coordinate through this

Today, data is growing at an exponential rate, and the need to analyze and execute queries in near real time on petabytes of data has pushed OLAP (Online Analytical Processing) engines to evolve well beyond their batch-oriented origins. Historically, OLAP systems were designed to process large datasets in batch mode, with efficient query execution treated as a secondary concern. Modern workloads, however, increasingly demand that OLAP engines behave more like OLTP (Online Tra

This is part 1 of deep dive series on understanding internals of Apache Spark Window Functions. The full series in based upon Spark version 3.1x . Introduction A window function in query processing is a function that processes a subset of data, often called window frame , over a large dataset. A window frame defines the boundaries (ie. start and end row) within which a window function is applied to a set of rows. Consider column values as [1, 2, 3, 4, 5] and we want to calc

Today's distributed systems are complex and varied, requiring different data consistency guarantees. Linearizability and strict serializability provide strong and intuitive guarantees, but they can require coordination that increases latency or reduces availability during failures, especially at global scale. Many systems therefore choose weaker consistency models for parts of their workload. In this article, we will learn about one such model called causal consistency. What

Data is exploding exponentially and so is the requirement for massively scalable distributed storage systems. A distributed storage system is a complicated beast which has to ensure performance, scalability, high tolerance and high availability. Today's topic is more targeted toward the availability aspect of data. Distributed storage systems ensure availability by replicating the data. An RFn (Replication Factor n) would replicate a data block to n nodes. For eg. in RF3, dat