Da Integra.
Log Manager Overview
iknoplex Log Manager is the fastest data warehouse infrastructure for high-volume log data currently on the market.
The product acts as a middleware that collects, normalizes, enriches and archives all the log and audit data, making it available to business intelligence and reporting applications.
- Collect
- ILM incrementally collects terabytes of log data in any format from network devices, systems, middleware and applications.
- Normalize
- All log data is organized according to a rich security and audit-oriented event format. The original structure is retained for compliance and forensic purposes.
- Enrich
- Event data is enriched with business context information: e.g., user organization, IT resource organization, policy statement.
- Archive and Query
- All event data is accessible in the warehouse database with standard SQL for the configured retention period. The query engine is so fast that caching outside the database is unnecessary.
To manage extraordinary large data volumes with unparalleled query performance, ILM leverages an innovative distributed computing architecture based on parallel processing, shared-nothing clusters, and a column-oriented database system.
Background
iknoplex Log Manager has been designed with high performance and scalability as primary goals.
Scalability is a primary market requirement that is becoming the key enabler of log management.
The drivers:
- Collection
- Large organizations are collecting more data from more sources to gain a wider purview of security threats and their ramifications on IT infrastructure.
- Usage
- Log data analysis is used beyond security threat management alone. Business managers, IT operations, compliance administrators, and “C-level” executives are increasingly using log data analysis to monitor numerous business and IT metrics.
- Retention
- Firms are archiving more log data for longer periods of time for future analysis or as liability protection in the event of a legal discovery.
More than 40% of large organizations collect at least 1TB (terabyte) of log data on a monthly basis while 11% collect more than 10TB of data each month. Additionally, nearly one-fourth of large organizations collect data from 1,000 or more sources (i.e. security, networking, and IT devices and applications).
More than one-fourth of large organizations expect that the number of sources (i.e. security, networking, and IT devices and applications) from which they collect log file data will “increase substantially” over the next 12 months.
The same growth pattern holds true with respect to total log data capacity. More than one-fourth of large organizations expect their log file data capacity to “increase substantially” over the next 12 months.
While log data growth is occurring across all market sectors, it is especially pronounced among the largest organizations. For example, nearly half of organizations with over 20,000 employees expect the number of log file sources to “increase substantially” over the next 12 months.
- Source: Jon Oltsik, Senior Analyst at Enterprise Strategy Group, October 2007
Key innovations
iknoplex Log Manager offers a unique combination of innovative features that are essential for high-volume enterprise environments.
- Column-oriented DB
- Fast query speed (>50 times faster than traditional DBMS)
- Superior data compression
- Requires less storage (1/10)
- Grid-based computing
- Open low-cost platform, high-availability with automatic recorvery
- Continuous parallel log collection and loading
- Report on most recent data
- Standard interfaces (SQL, JDBC, ODBC)
- Leverage market-proven BI infrastructure
- Unified event model
- Rapid design and deployment
Unified Event Model
iknoplex Log Manager generates a normalized view for all incoming event data that is independent from the original log type. All original event attributed are retained for fine-grained type-specific searches.
| Dimension | Questions answered
|
| Time | When did the event occur?
|
| Operation | What type of event occurred? Was it successful or not?
|
| User | Who originated the event? What role did this person have and with what privileges did he operate?
|
| Platform | Where did the event occur? To what resource group does the system belong?
|
| Object | On what did the user operate? Which resource was accessed and to which resource group does it belong?
|
| Origin | From where did the user operate on the platform? Which remote terminal/system did he use?
|
| Target | Did the event affect a remote system? Where did the user connect to?
|
Supported Event Sources
iknoplex Log Manager integrates a very large number of operating systems, middleware and applications with specialized and constantly updated log adapters that map the native data structure to the Unified Event Model. New log adapters are added at a constant rate.
The current list of supported event sources is here.
Standard Interfaces
iknoplex Log Manager features industry-standard interfaces to allow interoperability with market-proven applications.
Grid-based Computing Architecture
iknoplex Log Manager
- runs on a cluster of autonomous, hot-swappable low-cost servers
- is designed to work as a single instance across the cluster
- leverages all resources (CPU, hard disk storage, network bandwidth) at their full potential
Example: 5-node cluster (hardware cost: 15-20KEUR)
2 x dual core CPU
12GB RAM
4 x 250GB HD
2 x 150GB HD
2 x 1Gbps Eth.
| Log processing rate | 7 TB/month
|
| Retention period | 6 months
|
| Query response time (avg.) | 3-10 seconds
|
Continuous operation and high availability
- Parallel, continuous load
- Other log management products cannot load data while the database is being queried.
- ILM operates with “snapshot isolation” where all committed data can be queried while uncommitted data is being loaded into the database.
- All processing and loading operations are automatically distributed and parallelized across the cluster.
- High availability and painless recovery and growth
- ILM ensures high-availability across the cluster.
- Hardware failure on a single node is handled automatically without service interruption.
- When the defective node is replaced, ILM automatically recovers missing data on the new node and resumes normal operation.
- To increase capacity and performance, new nodes can be added at run-time. The software automatically adjusts to the increased cluster size.
To request information please write to iknoplex@integra-group.it
