The history of Storage

This is a part of chapter "A brief history of IT Infrastructures" of my forthcoming book "Infrastructure Architecture". Please feel free to comment using my email address stated in the right column of this website.  

Storage

Storage persistently stores and retrieves data. Storage systems store data blocks than can be retrieved either as raw data or as files. Persistency in storage means that data stored on a storage device remains on it, even without supplied power.

Early storage systems

The earliest computers used no persistent storage system. When these first computers were switched on, using manual switches some instructions were loaded in main memory and executed. These instructions instructed the computer to load a program in main memory from a paper tape or from punched cards. Output was printed on paper. Data was stored on paper tape or punched cards, leading to typical batch processing systems, where the output from one step was fed as input into the next step.

Drum memory

Drum memory was one of the first magnetic storage systems. It was widely used in the 1950s and into the 1960s. For many computers, a drum formed the main working memory of the machine. Drums were so commonly used as computer’s main memory that these computers were often referred to as drum machines.

Drum memory consists of a large metal cylinder that was coated on the outside surface with a ferromagnetic recording material. It could be considered the precursor to the hard disk platter, but in the form of a drum rather than a flat disk. In most cases multiple rows of fixed read-write heads were placed along the long axis of the drum, one head for each track on the drum.

A difference between a drum as described and a modern disk is that on a drum the heads do not have to move to the track to access; the controller simply waits for the data to appear under the relevant head as the drum turns. In a disk drive the head takes a certain time, the seek time, to move into place, while the performance of a drum with fixed heads is determined almost entirely by the rotational speed.

Hard disks

A computer disk consists of a set of rotating disk platters and magnetic read/write heads mounted on a movable arm. This allows the read/write heads to reach any place on the disk.

The disk platters are covered with high density magnetic material that allows the storage of data by magnetizing the particles in two directions (representing 0's and 1's).

The first commercial digital disk storage device, was the IBM RAMAC 350 shipped in 1956. It could store a whopping 4.4 MB of data! It was composed of fifty 24-inch (61 cm) diameter disks. The system was 60 inches (152 cm) long, 68 inches (172 cm) high and 29 inches (74 cm) deep and it weighed over a ton.

IBM 1311 Disk Storage Drive was announced in 1962 as being the first disk drive with a removable disk pack. Each pack weighed about ten pounds, held six disks, and had a capacity of 2 MB.

In 1980 Seagate Technology created the first hard disk drive for microcomputers (called the ST506). The disk could store 5 MB, which was five times as much as a standard 5 ¼" floppy disk at that time. In the mean time drive systems had shrunk to fit in the space of a 5 ¼" floppy disk drive. Upon releasing its first product, Seagate quickly drew such big- name customers as Apple Computer and IBM. Within a few years, it had sold 4 million units, making Seagate the market leader for small size hard disk drives.

Since the introduction of the RAMAC 350 disk sizes shrunk and disk capacity raised every year. Storage predictions show the historic average disk size since 1980 and a projection for the next years.

This picture shows that the average disk capacity has followed a logarithmic increase in size for the last 30 years (be aware that the Y-axis is logarithmic instead of linear). Kryder's law states that "the density of information on hard drives has been growing at a rate, increasing by a factor of 1000 in 10.5 years, which corresponds to a doubling roughly every 13 months".

Floppy disks

In 1971 IBM invented the 8-inch floppy diskette. It quickly won widespread acceptance as a program and data-storage medium. Unlike hard drives, a user could easily transfer a floppy from one drive to another.

The 5 ¼" flexible disk drive and diskette were introduced by Shugart Associates in 1976. This was the result of a request by Wang Laboratories to produce a disk drive small enough to use with a desktop computer, since 8" floppy drives were considered too large for that purpose.

5 ¼" floppy drives were used in the first IBM PCs and their clones. A disk could store 360KB of data, which was more than enough at the time since the PCs only had 64KB of main memory. Since the original IBM PC did not include a hard disk drive, the 5 ¼" floppy was the only available persistent storage media (apart from the analogue tape connection of course).

Sony introduced and shipped the first 3.5" floppy drives and diskettes in 1981. These floppy disks became very popular as they had a reasonable capacity of 1.44 MB and a very attractive form factor. The hard cover also made the disks more robust than the previous 8" and 5 ¼" floppies. IBM started to use 3.5" floppy drives in their PS/2 line of products, after which many others followed. This effectively stopped the use of 5 ¼" floppies in just a few years.

Nowadays floppy drives are mostly replaced by USB based storage (flash disk drives).

Tape drives

Magnetic tape allows for inexpensive mass storage of information. The IBM 726, announced in 1952 was one of the first practical magnetic tape systems. The Model 726 could store 2 million digits per tape—an enormous amount at the time. The system used a unique ‘vacuum channel’ method of keeping a loop of tape circulating between two points allowing the tape drive to start and stop the tape in a split-second.

In 1984 IBM announced the 3480 cartridge tape system that was to replace the traditional reels of magnetic tape with a 4” x 5” (10 x 12 cm) cartridge that could store 200MB of data. Other computer makers began making 3480- compatible storage systems for several years after that, offering increased storage capacity in the same physical format.

In 1987 Exabyte and Sony introduced Helical scan tape drives. These tape drives used a rotating drum with multiple read/write heads spinning round the tape in a specific angle, a technology also used in Video Tape Recorders (VCRs).

This allowed for a much more efficient use of the overall surface of the physical tape, increasing the capacity of the tape. Due to the complex and delicate technology Helical drives were very expensive and were not widely used.

In 1984 DEC introduced the Digital Linear Tape (DLT) drive. DLT drives could store up to 20GB of data on a tape with a small form factor. The tape medium could be made small because the tape housing contained only one reel. The other reel was located in the DLT drive unit.

A variant with higher capacity is called Super DLT (SDLT), which can store up to 300GB.

Linear Tape Open (or LTO) is originally developed in the late 1990s as an open standards alternative to the proprietary magnetic tape formats that were available at the time. The standard form-factor of LTO technology goes by the name Ultrium, the original version of which was released in 2000 and could hold 100 GB of data in a single cartridge. The most recent version was released in 2010 and can hold 1.5 TB in the same size cartridge.

LTO cartridges look similar to DLT cartridges and have roughly the same size, but they are not interchangeable. Both (S)DLT and LTO are still in use today.

Optical storage

Data can be stored on an optically readable medium. The best known examples of this are the CD-ROM, DVD or Blu-ray disk. The medium is spinning and a small laser beam reads microscopic pits in the disk. The pits represent data (0's and 1's).

Laserdisc technology, using a transparent disc, was already invented in 1958. Laserdisc used a light source behind the rotating disk showing information on the disk to a transducer head that read it on the opposite side of the disk.

By 1969 Philips had developed a videodisc in reflective mode, which has great advantages over the transparent mode, since it could be read by a laser and could store a much higher density of information. MCA and Philips first publicly demonstrated the videodisc in 1972, which could be used to show video on a TV. Laserdisc was first available on the market in 1978, two years after the VHS VCR and four years before the CD, which is based on Laserdisc technology.

The 1985 “Yellow Book” standard developed by Sony and Philips adapted the CD format to hold any form of binary data. It led to the introduction of CD-ROMs, rewritable media like CD-RW, and many other formats.

When CD-ROMs were introduced they had more capacity than computer hard drives common at the time. The reverse is now true, where hard drive capacity far exceeds that of CDs, DVDs and Blu-ray disks.

CD-ROM drives were rated with a speed factor relative to music CDs (1× or 1- speed which gives a data transfer rate of 150 Kbit/s and would take about an hour to read a complete CD). Around 1997, 12× drives became common. Above 12× speed, there are problems with vibration and heat of CD-ROMs.

CD-ROMs and DVDs were mostly used to distribute software. Previously this was done using floppy disks, but the size of applications made this impractical at some point (Novell NetWare for instance shipped on 25 floppy disks that had to be loaded one-by-one, some needed to be loaded even multiple times for configuration reasons).

Nowadays rewritable DVDs are the standard (storing up to 4 GB of data on a single layer), soon to be replaced by Blu-ray disks, capable of storing 20GB.

The history of Networking

This is a part of chapter "A brief history of IT Infrastructures" of my forthcoming book "Infrastructure Architecture". Please feel free to comment using my email address stated in the right column of this website.    

Networking 

Networking provides online interconnectivity between computer systems using the exchange of data packets.

Computers used to be standalone machines. They performed computing jobs based on input (usually on punched cards or tapes) creating output (usually on printed paper). Computers were large expensive systems mostly in use at universities and large corporations. Since a university or corporation had only one computer, there was little need to have networking capabilities within the university or company. Even with time sharing systems like the early UNIX based systems the user's terminals or teletypes were connected to the central computer through serial (RS-232) lines.

ARPANET and the advert of the Internet

Connecting computers using networks really started with the ARPANET project in 1963. This project, together with the invention of packet switching resulted in a first working network of interconnected computers, which became the basis of the modern Internet.

The ARPANET network consisted of small Interface Message Processors (IMPs) which we would now call network routers. The IMPs performed store-and-forward packet switching functions, and were interconnected using modems that were connected to analogue leased lines. The computers were connected to the IMPs via serial interfaces.

The first four computers in ARPANET all used different operating systems. The system's designers had to come up with a common set of rules (protocols) the network would follow in order for the computers to communicate with each other. ARPANET originally used the Network Control Protocol (NCP), which was replaced by TCP/IP in 1983.

The ARPANET network was expanded over the years, connecting more and more computers exchanging packets.

ARPANET was mostly used to send messages via email. In 1971 the first e- mail was sent. By 1973, the File Transfer Protocol (FTP) specification had been defined and implemented, enabling file transfers over the ARPANET.

In 1973, engineers began to look at ways to connect ARPANET to the packet radio network (PRNET). A packet radio network connects computers through radio transmitters and receivers. Instead of sending data across phone lines, the computers use radio waves. Technicians joined the Satellite Network (SATNET) to the two other networks in 1977. They called the connection between multiple networks inter-networking, or the Internet for short.

Internet grew at an incredible rate in the following years, especially after the development of the World Wide Web (WWW) and the HyperText Transfer Protocol (HTTP).

Local Area Networks

When Personal Computers (PCs) found their way into the office, the need arose for sharing data between office PCs. Local Area Networks (LANs) were designed to allow PCs to connect to each other and to share resources like a file server or a router to the Internet.

In the early years there were no LAN standards, so LANs were built using various technologies. The most used technologies were ARCNET, Token Ring and Ethernet. Today Ethernet is the de-facto standard for LANs.

ARCNET

ARCNET was introduced in 1977 by Datapoint Corporation and was the first widely available networking system, years before the introduction of the IBM PC. ARCNET transmitted packets of data at 2.5 megabit per second (Mb/s). ARCNET was one of the first LAN technologies to makes no assumptions about the type of computers that could be connected, in contrast with for instance SNA or DECnet. ARCNET was less expensive than Token Ring and Ethernet, more reliable, more flexible, and by the late 1980s it had a market share about equal to that of Ethernet. ARCNET used a coaxial BNC cabling system with passive and/or active hubs in a star-wired bus topology, a layout eventually copied by modern twisted pair Ethernet LANs.

The cabling topology made it easy to add and remove nodes without taking the whole network down (compared to the Ethernet shared medium topology), and much easier to diagnose and isolate failures within a complex LAN.

ARCNET used a token-passing protocol where only one station has a token. When the station receives the token, it can either initiate a transmission to another station or it must pass the token to its logical neighbor. All stations are considered peers and no one station can consume all the bandwidth since only one packet can be sent each token pass. This system was also used in other technologies, like Token Ring (that were more expensive).

ARCNET is hardly used anymore in LANs, being surpassed by Ethernet. However, it is still in use today in a variety of embedded networking applications like industrial monitoring and control systems and in shipboard communications, because of its token passing scheme that avoids collisions and makes ARCNET deterministic - an advantage in real-time applications. This makes it possible for designers to accurately predict the time it takes for a particular station to gain access to the network and send a message, a property very practical in for instance robotics.

Token Ring

Token Ring is a LAN technology created by IBM in 1985. Token Ring uses a ring topology whereby the data is sent from one machine to the next and so on around the ring until it ends up back where it started. It uses a special three-byte frame called a token that travels around the ring. It also uses a token passing protocol which means that a machine can only use the network when it has control of the token. This ensures that there are no collisions because only one machine can use the network at any given time.

Token Ring used shielded twisted pair cables, with special rather large hermaphroditic connectors (being both male and female). The cables connected PCs to so-called multi-station access units (MAUs). These units created the physical ring. When a connector was pushed in the MAU, the ring was mechanically opened up for the new station to be part of the ring, creating a small disturbance of the data flow.

Token Ring originally ran at a speed of 4Mb/s, later upgraded to 16Mb/s, which was significantly faster than ARCNET and Ethernet at the time.

Token Ring was a very reliable protocol that allowed for a very high utilization of the available bandwidth (up to 90%), as opposed to Ethernet that could be utilized to a maximum of 60%.

Mainly because of the cost and the complex physical components (including the MAUs) Token Ring networks are now mostly taken over by Ethernet networks.

Ethernet

Ethernet was developed at Xerox PARC between 1973 and 1975 by a team led by Robert Metcalfe. Metcalfe later founded 3Com. 3Com built the first 10 Mbit/s Ethernet adapter in 1981. Ethernet used a shared medium topology, based on coax cable and used a carrier sense multiple access approach. This means that all stations are connected to a shared Ethernet cable (analogue to the Ether in radio technology - hence the name Ethernet). This single Ethernet cable winds around a building or campus to every attached machine.

Any station can start sending packets when the cable is not in use. It uses a sensing circuitry that checks if the cable is not in use. This works very well when the cable is not used very much. Packet collisions appear when two stations start to send a packet at the same time (which is quite likely when more active stations connect to the cable). When a collision happens, it is detected by the stations and they will re-send their packet after a short time. This technology was called Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Ethernet's performance degraded drastically if too many peers attempted to send at the same time, due to the time required to recover from collisions. An Ethernet network could even collapse when is became too busy due to excessive collisions. No data could be transferred anymore at such an event. Throughput on a shared medium Ethernet was limited to between 40% and 60% of bandwidth usage for this reason.

While a simple passive wire was highly reliable for small Ethernets, it was not reliable for large extended networks, where damage to the wire in a single place, or a single bad connector, could make the whole Ethernet segment unusable. Finding such a bad connector could take quite some time (believe me, I have done it more that I wished).

When Ethernet moved from coax cable to twisted pair and a cabling topology based on active hubs, it became much more reliable. Easier cabling, combined with the greater raw speed of Ethernet (10 Mb/s, as compared with 2.5 Mb/s for ARCNET and 4Mb/s for Token Ring), and aggressive marketing helped to increase Ethernet's popularity.

In 1990 the first Ethernet switch was introduced which replaced the CSMA/CD scheme in favor of a switched full duplex system providing higher performance and at a lower cost. Most manufacturers now build the functionality of an Ethernet card directly into PC motherboards, eliminating the need for installation of a separate network card. Nowadays Ethernet is also used for wireless LANs. The wireless “ether” is perfectly suited for the shared medium Ethernet technology.

LAN network protocols

Based on the LAN hardware various network protocols were implemented. These network protocols typically manage the reliable transmission of data packets. The most widely used protocols were SNA, DECnet, IPX/SPX, TCP/IP and UDP/IP.

SNA

SNA stands for Systems Network Architecture. The SNA protocol was created by IBM in 1974. It was used to connect terminals to a mainframe computer in a standardized way. Later PCs using SNA based interface cards could provide terminal access to mainframes, first running in MS-DOS, later running in a window under the Windows operating system. SNA removed link control from all of the application programs and placed it in the network protocol layer as a single instance, greatly reducing the complexity of the applications. Since SNA also took care of routing messages, the burden on applications was reduced even more.

SNA does not define specific protocols for its physical control layer. The physical control layer is assumed to be implemented via other standards. SNA-supported media types include mainframe channels, SDLC, X.25, and Token Ring. Today in most cases SNA has been replaced by TCP/IP.

DECnet

DECnet is a LAN technology developed and supported by Digital Equipment Corporation (DEC). The first version of DECnet was released in 1975 and allowed two directly attached PDP-11 minicomputers to communicate. Later DECnet was the default network protocol for VMS systems. The various versions of DECnet were called "phases", where phase V is the most recent one (published in 1987). Originally DECnet was limited in the amount of nodes it could address. Later versions removed this limitation. Also the original unreliable and limited routing possibilities or DECnet are now improved.

For backwards compatibility DECnet Phase IV and later can operate over IP networks.

IPX/SPX

IPX/SPX stands for Internetwork Packet Exchange/Sequenced Packet Exchange. IPX and SPX are networking protocols used primarily on networks using the Novell NetWare operating systems. Novell NetWare was the dominant file server product in the 1980s and 1990s.

IPX is the network layer protocol (layer 3 of the OSI Model), while SPX is the transport layer protocol (layer 4 of the OSI Model). The SPX layer sits on top of the IPX layer and provides connection-oriented services between two nodes on the network.

IPX/SPX was primarily designed for local area networks (LANs), and is a very efficient protocol for this purpose. In comparison, TCP/IP is designed for Wide Area Networks (WANs) and is less efficient when used in a LAN.

The IPX/SPX protocol stack was implemented in most popular operating systems, including MS-DOS, Windows, MacOS and Linux. It is hardly used anymore since almost all LANs today use TCP/IP as their network protocol, which is suited for both LAN and WAN use.

TCP/IP

TCP/IP stands for Transmission Control Protocol/Internet Protocol. TCP/IP is actually based on two protocols. IP is the network layer protocol (layer 3 of the OSI Model), while TCP is the transport layer protocol (layer 4 of the OSI Model). The TCP layer sits on top of the IP layer and provides connection-oriented services between two nodes on the network.

TCP/IP was created as part of the DARPA project (Defense Advanced Research Projects Agency). The first specification of TCP was published in 1974, while the TCP version 4 that is now in use on the Internet was published in 1978. In 1982, the US Department of Defense declared TCP/IP as the standard for all military computer networking. The migration of the ARPANET to TCP/IP was officially completed in 1983, making TCP/IP the standard protocol on the Internet. TCP/IP implementations are based on RFCs. A Request for Comments (RFC) is a memorandum published by the Internet Engineering Task Force (IETF) describing methods, behaviors, research, or innovations applicable to the working of the Internet and Internet-connected systems. RFCs are the design documents for the Internet. RFC 1122 describes the technical architecture of TCP/IP. All RFCs are published freely on the Internet (http://www.rfc-editor.org).

Most computer operating systems in use today include a TCP/IP implementation.

UDP/IP

TCP is a reliable protocol - it ensures that packets reach their destination in order and without errors. UDP (User Datagram Protocol) is the unreliable version of the protocol, also using IP as the underlying layer 3 protocol. Unreliable does not mean that it does not work sometimes, but that delivery of packets is not guaranteed. Compared to TCP the overhead for UDP is much lower, which makes it very suitable for high speed connections that could suffer some packet loss, like video streams.

Numerous key Internet applications use UDP, including: the Domain Name System (DNS), where queries must be fast and only consist of a single request followed by a single reply packet, the Simple Network Management Protocol (SNMP), the Routing Information Protocol (RIP) and the Dynamic Host Configuration Protocol (DHCP).

Wide area Networks

Commercial wide area networks (WANs) started to appear in the 1980s when companies needed to electronically communicate between their subsidiaries and (sometimes) other companies. The first WANs consisted of point-to-point connections using modems and the analogue public telephone system. The modems would dial each other and establish a connection, usually at around 1200 baud (b/s). The modems were connected to routers that routed the packets between the end points.

Later phone companies started to offer dedicated analogue leased lines. These leased lines eliminated the need to setup a call and were considerably cheaper than dial-up lines. When more and more WAN connections appeared phone companies started to offer digital leased lines. These leased lines eliminated the need for modems to convert digital data to analogue frequencies that would fit the characteristics of telephone conversations. Digital leased line technologies were for instance T1/E1 lines and ISDN connections.

Instead of using point-to-point connections WANs could also be built using less costly circuit switching or packet switching technologies. Protocols like Packet over SONET/SDH, MPLS, ATM and Frame Relay were often used by service providers to deliver WAN links. X.25 was an important early WAN protocol, and is often considered to be the "grandfather" of Frame Relay as many of the underlying protocols and functions of X.25 are still in use today (with upgrades) by Frame Relay.

Frame Relay offered a point-of-presence (POP) to companies. A connection was made between the company and the most nearby POP using a leased line. The routing of messages between the POPs was a responsibility of the Frame Relay provider, usually a phone company.

While Frame Relay is still used in many areas, it is superseded now by VPN connections using the Internet. Cheap DSL and cable modem connections to the Internet can now be used open up company's subsidiaries for communication via secured VPN connections.

Line speed

Modem and line speeds have become faster and faster over the years. Nielsen's law states that network connection speeds for high-end home users would increase 50% per year, or double every 21 months. He predicted this in 1983 and it is still very accurate.

Some typical line and modem speeds were:

• 1972: Acoustic coupler 300 baud
• 1977: 1200 baud modems
• 1986: ISDN introduced with two 64 Kbit/s channels (160 Kbit/s gross bit rate)
• 1990: Increasing Modem bandwidth: 2400 / 4800 / 9600 / 19200 bit/s
• 1995: v.34 modems with 28.8 Kbit/s, v.90 modems with 56 Kbit/s downstream
• 1998: 128 Kbit/s ADSL

I remember that when we had to get a 2Mb/s leased line to a frame relay network in 1998 we had to pay $ 8,000 per month! Now (2010) my home Internet subscription costs me $ 30 per month, delivering 20Mb/s; ten times as much for less than one percent of the cost! There is no reason to assume this trend will break in the next years. The bandwidth available for home users should be 150 Mb/s in 2015 and 8.5 Gb/s in 2025, all for a price of about $50 per month.