Log analysis - Use your logging information

For a client I am implementing a Log analyses solution. All logging of all servers, network components and appliances are sent to one system. From this system, reports can be generated with log information and the logging can be used for analysis.

There are two reasons for using logdata:

  1. Reports can be generated about (security) trends;
  2. Logs can be used to investigate incidents. They could be used for evidence in forensic analysis, when someone broke company rules or the law.

Log analysis solutions are regularly used for compliancy reasons, like SOX, COBIT or BASEL-II demands, or to prove to auditors that the company is compliant to the ISO/IEC 17799:2005 Code of practice for Information Security Management.

While most log analysis solutions can be used for alerting in real-time, for instance when something suspicious happens, or when a threshold is reached, I think log analysis is something fundamentally different than monitoring. Both have different goals.

Monitoring systems, like IDS systems or SNMP based systems are real-time systems. As soon as something happens, an alarm goes off.

Log systems are meant for analyzing events afterwards.

Log analysis solutions are part of SIEM (Security Information and Event Management) solutions. Roughly the market is split into two parts: software running on servers and hardware appliances. Net Report is an example of a software solution, Loglogic is an appliance. Snare Server can deliver both.

Usually log systems deal with large amounts of data (having gigabytes of logging per day is not unusual for a large environment). This means the solutions have to be designed for storage, bandwidth and fast searching in large amounts of data.

Reports

Log reports can be used to give a quick overview of events. Some examples are:

  • The amount of port scans a firewall experienced over a given period of time;
  • Which stations performed more than 5 un-successful login attempts;
  • What is the top-10 websites that are visited during work-hours;
  • The amount of emails sent (internally and externally) during a typical business day;
  • What is the account of which the password is changed the most often.

Based on these reports incident investigations can be started.

The reports are made from data in a database, which contains aggregated logdata. Net Report has some examples of generated reports online, check the Report and Dashboard Samples.

Investigating incidents 

Because all logging of all systems are stored centrally, it is easy to get a complete view of all logging events for a certain moment in time.

Log solutions usually contains all raw logs on a large harddisk. Storing this data must be done in a way that gives enough assurance that the data is not altered and original. This way, the logdata can be used for forensic investigations and can be used in court as evidence.

Usually the data is encrypted and hashed, so it has a digital signature.


This entry was posted on Thursday 02 August 2007

Earlier articles

Infrastructure as code

My Book

DevOps for infrastructure

Infrastructure as a Service (IaaS)

(Hyper) Converged Infrastructure

Object storage

Software Defined Networking (SDN) and Network Function Virtualization (NFV)

Software Defined Storage (SDS)

What's the point of using Docker containers?

Identity and Access Management

Using user profiles to determine infrastructure load

Public wireless networks

Supercomputer architecture

Desktop virtualization

Stakeholder management

x86 platform architecture

Midrange systems architecture

Mainframe Architecture

Software Defined Data Center - SDDC

The Virtualization Model

What are concurrent users?

Performance and availability monitoring in levels

UX/UI has no business rules

Technical debt: a time related issue

Solution shaping workshops

Architecture life cycle

Project managers and architects

Using ArchiMate for describing infrastructures

Kruchten’s 4+1 views for solution architecture

The SEI stack of solution architecture frameworks

TOGAF and infrastructure architecture

The Zachman framework

An introduction to architecture frameworks

How to handle a Distributed Denial of Service (DDoS) attack

Architecture Principles

Views and viewpoints explained

Stakeholders and their concerns

Skills of a solution architect architect

Solution architects versus enterprise architects

Definition of IT Architecture

What is Big Data?

How to make your IT "Greener"

What is Cloud computing and IaaS?

Purchasing of IT infrastructure technologies and services

IDS/IPS systems

IP Protocol (IPv4) classes and subnets

Infrastructure Architecture - Course materials

Introduction to Bring Your Own Device (BYOD)

IT Infrastructure Architecture model

Fire prevention in the datacenter

Where to build your datacenter

Availability - Fall-back, hot site, warm site

Reliabilty of infrastructure components

Human factors in availability of systems

Business Continuity Management (BCM) and Disaster Recovery Plan (DRP)

Performance - Design for use

Performance concepts - Load balancing

Performance concepts - Scaling

Performance concept - Caching

Perceived performance

Ethical hacking

Computer crime

Introduction to Cryptography

Introduction to Risk management

The history of UNIX and Linux

The history of Microsoft Windows

The history of Novell NetWare

The history of operating systems - MS-DOS

The history of Storage

The history of Networking

The first computers

History of servers

Tips for getting your ITAC certificate

Studying TOGAF

Is your data safe in the cloud?

Proof of concept

Who needs a consistent backup?

Measuring Enterprise Architecture Maturity

Human factors in security

Master Certified IT Architect

ITAC certification

Open group ITAC /Open CA Certification

Human factors in security

Google outage

SAS 70

TOGAF 9 - What's new?

DYA: Development without architecture

Spam is big business

Why IT projects fail

Power and cooling

Let system administrators participate in projects

The 7 Habits of Highly Effective People

Archimate

A meeting with John Zachman

ITAC - IT Architect certification

Personal Information is Personal Property

The Irresistible Forces Meet the Movable Objects

Hardeningscheck and hack testing for new servers

Knowledge management

Information Lifecycle Management - What is ILM

LEAP: The Redmond trip

LEAP: The last Dutch masterclasses

What do system administrators do?

Is software ever finished?

SCADA systems

LEAP - Halfway through the Dutch masterclasses

Securing data: The Castle versus the Tank

Non-functional requirements

LEAP - Microsoft Lead Enterprise Architect Program

Reasons for making backups

Log analysis - Use your logging information

Archivering data - more than backup

Patterns in IT architecture

Layers in IT security

High performance clusters and grids

Zachman architecture model

High Availability clusters

Monitoring by system administrators

What is VMS?

IT Architecture certifications

Storage Area Networks (SAN)

Documentation for system administrators

Rootkits

Presentations: PowerPoint sheets are not enough

99,999% availability

Linux certification: RHCE and LPI

IT Infrastructure model

Sjaak Laan


Recommended links

Ruth Malan
Gaudi site
Esther Barthel's site on virtualization
Eltjo Poort's site on architecture


Feeds

 
XML: RSS Feed 
XML: Atom Feed 


Disclaimer

The postings on this site are my opinions and do not necessarily represent CGI’s strategies, views or opinions.

 

Copyright Sjaak Laan