Performance concept - Caching

Caching improves performance by retaining frequently used data in high speed memory, reducing access times to data. Some sources that provide data are slower than others. The approximate speed of retrieving data from various sources is shown below.   

Especially in situations where retrieving data takes relatively long (for instance reading form hard disk, CD- ROM and the network) caching can improve performance significantly.

In case of hard disks, before data can be read it must be located on the disk. Disks are mechanical devices, where the read head must be positioned above the correct track of the disk platter. Then the system must wait for the desired information to spin under the read head. This so-called seek time can take a long time: about 12 ms. When the data is actually read, streaming it is much faster: reading 512 bytes of data (a typical disk block) takes only 0.8 ms.

To speed up the reading of data from disk all disk drives contain caching memory. This caching memory stores all data recently read from disk and some of the disk blocks following the disk blocks that were read. When the data is to be read again, or (more likely) the data of the next disk block is needed, it is fetched from high speed cache memory, and without the seek time overhead.

The same principle goes for DVD drives (and CD-ROM drives, Blue-ray drives, etc). Here seek time includes not only the steps described above, but also the adjustment of disk speed. When the read head of the CD- Rom drive (the laser reading the disk) moves from the beginning to the end of the CD-ROM (or the other way around) the speed of CD-ROM drives changes accordingly. When data at the inner circles of the disk are read, the disk spins at a higher speed than when data is read at the edge of the disk. The drive’s motor must adjust the speed and this takes a considerable amount of time.

While networking connections are much faster, here also cache memory is used. And all CPUs today use internal caching as well (for more information on CPU cache.

Caching can be implemented in several ways, like using disk caching, web proxies, Operational Data Stores, web front end servers and even in- memory databases.

The best known example of using caching to increase performance is disk caching. Disk caching can be implemented in the storage component itself (for instance cache used on the physical disks or cahce implemented in the disk controller), but also in the operating system. A general rule of thumb is that adding memory in servers usually improves performance. This is due to the fact that all non-used memory in operating systems is used for disk cache. Over time all memory is filled with stored previous disk requests and pre-fetched disk blocks, speeding up data management.

Another example of caching is using web proxies. When users browse the Internet, instead of fetching all requested data from the Internet, earlier accessed data can be cached in a proxy server and fetched from there. This has two benefits: the user gets his data faster than when it would be retrieved from a distant web server, and all other users are provided more bandwidth to the Internet, as the data did not have to be fetched again.

An Operational Data Store (ODS) is a replica of a part of a database for a specific use. Instead of accessing the main database for retrieving information, often used information is retrieved from a separate small ODS database, not degrading the performance of the main database. A good example of this is a website of a bank. Most users want to see their actual balance when they login (and maybe the last 10 mutations of their balance). When every balance change is not only stored in the main database of the bank, but also in a small ODS database, the website only needs to access the ODS to provide users with the data they most likely need. This not only speeds up the user experience, but also decreases load on the main database.

In web facing environments storing most used (parts of) pages (like the JPG pictures used on the landing page) at the web front-end server lowers the amount of traffic to back-end systems enormously. Reverse- proxies can be used to cache most wanted data as well.

In special circumstances even complete databases can be run from memory instead of from disk. These so- called in-memory databases are used in situations where performance is crucial (like in real-time SCADA systems). Of course special arrangements must be made to ensure data is not lost when a power failure occurs.

Perceived performance

Performance is a typical hygiene factor. Nobody notices a highly performing system. But when a system is not performing well enough, people start complaining quickly. As an example, a 2006 research by Akamai found that 75% of the 1,058 people asked would not return to websites that took longer than four seconds to load. Five years earlier, in 2001, a similar research revealed that people were willing to wait eight seconds.  

The increase in bandwidth (in 2001 people mostly used modems to dial-in to the Internet), and the positive experience of very fast loading websites (like Google’s lightweight landing page) dropped this figure to four seconds and it would not surprise me that it will drop even more in the forthcoming years.

In general, the performance of a system can be defined as the amount of useful work that is accomplished by a system compared to the time and resources it used. 

Perceived performance refers to how quickly a software feature appears to perform its task. Most people understand that running a very complex report in a BI environment takes longer than let’s say opening an email. But while people have intrinsic expectations about performance, they are seldom expressed in hard figures.

In general people tend to overestimate their own patience. Most people value predictability in performance. When the performance of a system is fluctuating, they get a bad experience, even if the fluctuation is relatively rare.

For instance, even when one gets a bad response time of a system once a week, it will color the perception of the system for a long time. In the mind of the users the system is often slow.

It is therefore important to have a system with a predictable and consistent performance. It is always best to inform the user about how long something will take (when it is not instantaneously of course). When the user knows she has to wait for 40 seconds to get a task performed, and is informed about it (for instance using a progress bar), she accepts it more quickly. On the other hand when the system seems unresponsive without a reason, people get irritated very quickly. Increasing the real performance of a system is one way to increase the perceived performance.

But when real performance cannot be increased (for instance due to physical limitations) or the cost of improving the performance is very high, some techniques can be used to increase perceived performance. Two of those techniques are splash screens and progress bars.

Of course the amount of time an application takes to start up, or the time it takes for a file to download, is not made any faster by showing a splash screen or a progress bar. However, showing these screens satisfies a typical human need: splash screens and progress bars provide visual feedback to inform the users that the system is handling their request and is busy performing work for them.

Drawing and refreshing a progress bar while loading a file satisfies the user who is waiting, but steals time from the process that is actually loading the file. However, usually this is only a very small amount of time and the benefit of a satisfied user is much higher.