Log analysis - Use your logging information

For a client I am implementing a Log analyses solution. All logging of all servers, network components and appliances are sent to one system. From this system, reports can be generated with log information and the logging can be used for analysis.

There are two reasons for using logdata:

  1. Reports can be generated about (security) trends;
  2. Logs can be used to investigate incidents. They could be used for evidence in forensic analysis, when someone broke company rules or the law.

Log analysis solutions are regularly used for compliancy reasons, like SOX, COBIT or BASEL-II demands, or to prove to auditors that the company is compliant to the ISO/IEC 17799:2005 Code of practice for Information Security Management.

While most log analysis solutions can be used for alerting in real-time, for instance when something suspicious happens, or when a threshold is reached, I think log analysis is something fundamentally different than monitoring. Both have different goals.

Monitoring systems, like IDS systems or SNMP based systems are real-time systems. As soon as something happens, an alarm goes off.

Log systems are meant for analyzing events afterwards.

Log analysis solutions are part of SIEM (Security Information and Event Management) solutions. Roughly the market is split into two parts: software running on servers and hardware appliances. Net Report is an example of a software solution, Loglogic is an appliance. Snare Server can deliver both.

Usually log systems deal with large amounts of data (having gigabytes of logging per day is not unusual for a large environment). This means the solutions have to be designed for storage, bandwidth and fast searching in large amounts of data.


Log reports can be used to give a quick overview of events. Some examples are:

  • The amount of port scans a firewall experienced over a given period of time;
  • Which stations performed more than 5 un-successful login attempts;
  • What is the top-10 websites that are visited during work-hours;
  • The amount of emails sent (internally and externally) during a typical business day;
  • What is the account of which the password is changed the most often.

Based on these reports incident investigations can be started.

The reports are made from data in a database, which contains aggregated logdata. Net Report has some examples of generated reports online, check the Report and Dashboard Samples.

Investigating incidents 

Because all logging of all systems are stored centrally, it is easy to get a complete view of all logging events for a certain moment in time.

Log solutions usually contains all raw logs on a large harddisk. Storing this data must be done in a way that gives enough assurance that the data is not altered and original. This way, the logdata can be used for forensic investigations and can be used in court as evidence.

Usually the data is encrypted and hashed, so it has a digital signature.

This entry was posted on Thursday 02 August 2007

Earlier articles

The cloud is as insecure as its configuration

Infrastructure as code

My Book

DevOps for infrastructure

Infrastructure as a Service (IaaS)

(Hyper) Converged Infrastructure

Object storage

Software Defined Networking (SDN) and Network Function Virtualization (NFV)

Software Defined Storage (SDS)

What's the point of using Docker containers?

Identity and Access Management

Using user profiles to determine infrastructure load

Public wireless networks

Supercomputer architecture

Desktop virtualization

Stakeholder management

x86 platform architecture

Midrange systems architecture

Mainframe Architecture

Software Defined Data Center - SDDC

The Virtualization Model

What are concurrent users?

Performance and availability monitoring in levels

UX/UI has no business rules

Technical debt: a time related issue

Solution shaping workshops

Architecture life cycle

Project managers and architects

Using ArchiMate for describing infrastructures

Kruchten’s 4+1 views for solution architecture

The SEI stack of solution architecture frameworks

TOGAF and infrastructure architecture

The Zachman framework

An introduction to architecture frameworks

How to handle a Distributed Denial of Service (DDoS) attack

Architecture Principles

Views and viewpoints explained

Stakeholders and their concerns

Skills of a solution architect architect

Solution architects versus enterprise architects

Definition of IT Architecture

What is Big Data?

How to make your IT "Greener"

What is Cloud computing and IaaS?

Purchasing of IT infrastructure technologies and services

IDS/IPS systems

IP Protocol (IPv4) classes and subnets

Infrastructure Architecture - Course materials

Introduction to Bring Your Own Device (BYOD)

IT Infrastructure Architecture model

Fire prevention in the datacenter

Where to build your datacenter

Availability - Fall-back, hot site, warm site

Reliabilty of infrastructure components

Human factors in availability of systems

Business Continuity Management (BCM) and Disaster Recovery Plan (DRP)

Performance - Design for use

Performance concepts - Load balancing

Performance concepts - Scaling

Performance concept - Caching

Perceived performance

Ethical hacking

The first computers

Open group ITAC /Open CA Certification

Sjaak Laan

Recommended links

Ruth Malan
Gaudi site
Esther Barthel's site on virtualization
Eltjo Poort's site on architecture


XML: RSS Feed 
XML: Atom Feed 


The postings on this site are my opinions and do not necessarily represent CGI’s strategies, views or opinions.


Copyright Sjaak Laan