Documentation for system administrators

Writing and maintaining documentation is something most system administrators don't like. Experience learns that documentation is seldom kept up-to-date after delivery.

In my opinion, this has two reasons:

• System admins don't see the use of documentation, because they experience very little profit from it;
• Documentation is hard to change (mostly because of procedures) and it is unappreciated work.

It is useful to give both issues enough attention. For system administration documentation it is certainly useful if admins can find relevant information easily.

Searching

In Google one can find an answer on a technical question usually within 30 seconds by stating clever chosen search terms. Searching documentation on a network drive within the company, usually organized in a very deep directory tree, takes much longer to finish. This is very frustrating to the system admins, who prefer to use Google to find answers, even when the answers are available somewhere in the system documentation.

Changing 

Changing is not very rewarding. After changing documentation, it is not visible to others easily what changed and who did it. For these reasons much documentation is not updated at all.

Wiki's

Modern technology can solve much of the described problems. Using Wiki's with search capabilities, for instance. The success of wiki's is best seen on http://www.wikipedia.org, where many people voluntarily invest much spare time in updating and creating articles.

People don't have to document articles on Wikipedia, but they want to do it, even for free. The reason is visibility: the whole world can see what you did and if you amend or improve documentation, your name is connected to this forever. Also information ypu created is easily found in Wikipedia.

Murdoc

My open source project Murdoc is a simple, but handy tool for exposing (already existing) documentation via a web interface with search capabilities. It is even possible for computer systems to be documented automatically, so the documentation is always up-to-date.

Murdoc contains a standard directory setup in which system administrators can organize their documentation.

Rootkits

Rootkits are "malicious software", just like viruses and worms. Rootkits are special because they are almost impossible to detect.

The name "Rootkit" was derived from the superuser name in UNIX, called "root". This root user has all rights on a system, just like the Windows Administrator has. By the way, rootkits are a problem of all operating systems, including Windows.

Rootkits and other malicious software can create backdoors in systems. Using this backdoor, hackers can penetrate the system to use it, to damage it (erasing or destroying data) or to use the system for attacks on other systems.

Rootkits are very hard to detect because they not only install malicious software, but they also install software that replace system commands. An example is the UNIX/Linux commando 'ls -l'. This command is used to print a list of files on the screen:

$ ls -l

total 72

drwxr-xr-x 3 slaan slaan 4096 2006-09-14 11:02 BACKUP

drwxr-xr-x 9 slaan slaan 4096 2006-09-16 13:52 google-earth

-rwxrwxrwx 1 slaan slaan 150 2006-10-02 19:50 maliciouscode.exe

drwxr-xr-x 8 slaan slaan 4096 2006-05-05 09:44 Murdoc_development

drwxrwxrwt 7 slaan slaan 4096 2006-09-10 13:54 My Virtual Machines

drwxr-xr-x 2 slaan slaan 4096 2006-09-15 08:45 scripts

drwxr-xr-x 11 slaan slaan 4096 2006-09-25 15:35 uapplications

drwxr-xr-x 2 slaan slaan 4096 2006-09-12 21:42 vmware

A rootkit could install a new version of 'ls'. This way the malicious code becomes invisible:

$ ls -l

total 72

drwxr-xr-x 3 slaan slaan 4096 2006-09-14 11:02 BACKUP

drwxr-xr-x 9 slaan slaan 4096 2006-09-16 13:52 google-earth

drwxr-xr-x 8 slaan slaan 4096 2006-05-05 09:44 Murdoc_development

drwxrwxrwt 7 slaan slaan 4096 2006-09-10 13:54 My Virtual Machines

drwxr-xr-x 2 slaan slaan 4096 2006-09-15 08:45 scripts

drwxr-xr-x 11 slaan slaan 4096 2006-09-25 15:35 uapplications

drwxr-xr-x 2 slaan slaan 4096 2006-09-12 21:42 vmware

To prevent this to be noticed, incorrect information about the 'ls' command itself is showed by the patched 'ls' command. This hides files with incorrect sizes. If necessary, rootkits could even change the kernel to make it show incorrect values!

There are two ways to prevent rootkits from being installed: 

1. Using virus detection, the installation of rootkits can be avoided;
2. Using Host-based-IDS (Intruder Detection Systems) technology changes to the system can be detected.

Both methods are very delicate however: Circumventing virus scanners can be done already and IDS systems can be misguided by rootkits, just like it misguides other applications and commands.

In 2005 rootkits made the news when it was uncovered that record company Sony/BMG installed rootkits, using their music CD's, to secretly install copy protection software. Eventually, it cost Sony much more than it gained. Client's trust was damaged which cost a lot more than a few illegally copied CD's.