Memory hierarchy & cache

By the end of this section, you will be able to place cache in the memory hierarchy, distinguish a hit from a miss, and calculate an average access time. You will then apply those ideas to three published ITPEC FE questions.

A CPU can work on data much faster than a large memory can always deliver it. Very fast memory is expensive per bit, so a computer uses several layers. Small, fast layers hold the data needed immediately. Larger, slower layers hold more of the program and its data.

Registers are small storage locations inside the CPU used directly by instructions. Cache keeps copies of recently or frequently accessed memory blocks near the CPU. Main memory (RAM) holds programs and data currently in use. Secondary storage, such as an SSD, retains files when power is off. Cache and RAM are usually volatile; an SSD is persistent storage.

Cache memory ↗ · 3 questions stores copies of data from a slower memory. When the CPU requests an item, a cache hit means that the item is found in the cache. A cache miss means it is absent, so the request must continue to another level. In a simple cache-and-RAM model, that next level is main memory.

The cache helps because programs tend to reuse nearby data. Temporal locality means an item accessed recently is likely to be accessed again. Spatial locality means nearby addresses are likely to be used. Caches transfer blocks, rather than guessing one independent byte at a time.

The hit ratio, usually written as h, is the fraction of accesses that hit. If 80 out of 100 accesses hit, then h = 0.8 = 80%. The miss ratio is 1 − h, which is 0.2 = 20% in this example. The two ratios add to 1.

Do not confuse a percentage with a count. A hit ratio of 0.8 says how often a hit occurs, not that the cache has 80 locations or that a hit takes 0.8 nanoseconds.

If a hit takes T_hit and the complete miss path takes T_miss, the average time is:

T_average = h × T_hit + (1 − h) × T_miss

The word “complete” matters. If a question gives the additional time needed after a failed cache check, use the cache-check time plus that miss penalty. Equivalently, average time = cache-check time + miss ratio × miss penalty. Read the question’s timing assumptions before substituting values.

1. Identify whether the question gives a hit ratio or a miss ratio.
2. Check the units: keep all times in nanoseconds, or convert them consistently.
3. Decide whether the main-memory time includes checking the cache.
4. Write the two possible access paths and weight each by its probability.
5. Check that the result lies between the fastest and slowest complete path times.

If two systems have the same effective time, write an expression for each system using the same unknown hit ratio, then set the expressions equal. You will use this technique in the linked practice.

Cache holds copies of memory blocks to reduce average access time. A hit finds the item in cache; a miss continues down the hierarchy. Average access time is a weighted average, and the correct formula depends on whether the stated miss time is a complete path or an extra penalty.