compress()

Compress an array of log lines into semantic templates.

Signature

function compress(
  lines: string[],
  options?: CompressOptions
): CompressionResult

Parameters

`lines`

Type: string[]
Required: Yes

Array of log lines to compress. Each line should be a complete log entry.

`options`

Type: CompressOptions
Required: No

Configuration options for compression.

`options.format`

Type: 'summary' | 'detailed' | 'json'
Default: 'summary'

Output format for the formatted field in the result:

'summary' - Compact template list with frequencies and rare events
'detailed' - Full templates with sample variables and all diagnostic metadata
'json' - Machine-readable JSON with version field

`options.maxTemplates`

Type: number
Default: 50

Maximum number of templates to include in the formatted output. Templates are sorted by occurrence count (most frequent first).

`options.depth`

Type: number
Default: 4

Parse tree depth for the Drain algorithm. Higher values create more specific templates.

`options.simThreshold`

Type: number
Default: 0.4
Range: 0.0 to 1.0

Similarity threshold for template matching. Lines must be at least this similar to match an existing template.

Lower values (0.2-0.3): More aggressive grouping, fewer templates
Higher values (0.5-0.6): More conservative grouping, more templates

`options.maxChildren`

Type: number
Default: 100

Maximum children per parse tree node. When exceeded, tokens are replaced with wildcards.

`options.maxClusters`

Type: number
Default: 1000

Maximum total templates allowed. Limits memory usage for very large log sets.

`options.maxSamples`

Type: number
Default: 3

Maximum number of sample variables to store per template.

`options.preprocessing`

Type: ParsingStrategy
Default: Built-in strategy

Custom preprocessing strategy. See Custom Preprocessing for details.

`options.onProgress`

Type: ProgressCallback
Default: undefined

Callback for progress updates during processing:

type ProgressCallback = (event: ProgressEvent) => void;

interface ProgressEvent {
  processedLines: number;
  totalLines?: number;
  currentPhase: 'parsing' | 'clustering' | 'finalizing';
  percentComplete?: number;
}

Return Value

Returns a CompressionResult object:

interface CompressionResult {
  templates: Template[];
  stats: {
    inputLines: number;
    uniqueTemplates: number;
    compressionRatio: number;
    estimatedTokenReduction: number;
    processingTimeMs?: number;
  };
  formatted: string;
}

`templates`

Array of extracted templates, sorted by occurrence count (descending).

See Template interface for details.

`stats`

Compression statistics:

inputLines - Number of input log lines
uniqueTemplates - Number of unique templates discovered
compressionRatio - Ratio of templates to input lines (0.0 to 1.0)
estimatedTokenReduction - Estimated token reduction percentage (0-100)
processingTimeMs - Processing time in milliseconds

`formatted`

String representation in the requested format.

Examples

Basic Usage

import { compress } from 'logpare';

const logs = [
  'ERROR Connection to 192.168.1.100 failed',
  'ERROR Connection to 192.168.1.101 failed',
  'INFO Request abc123 completed',
  'INFO Request xyz789 completed',
];

const result = compress(logs);

console.log(result.formatted);
// === Log Compression Summary ===
// Input: 4 lines → 2 templates (50.0% reduction)
//
// Top templates by frequency:
// 1. [2x] ERROR Connection to <*> failed
// 2. [2x] INFO Request <*> completed

With Options

const result = compress(logs, {
  format: 'detailed',
  depth: 5,
  simThreshold: 0.5,
  maxTemplates: 100,
});

Progress Tracking

const result = compress(logs, {
  onProgress: (event) => {
    console.log(`Phase: ${event.currentPhase}`);
    console.log(`Processed: ${event.processedLines} lines`);
    if (event.percentComplete !== undefined) {
      console.log(`Progress: ${event.percentComplete.toFixed(1)}%`);
    }
  }
});

Accessing Templates

const result = compress(logs, { format: 'json' });

// Filter error templates
const errors = result.templates.filter(t => t.severity === 'error');

// Get most frequent template
const mostFrequent = result.templates[0];
console.log(`Most common: ${mostFrequent.pattern} (${mostFrequent.occurrences}x)`);

// Extract all URLs
const allUrls = result.templates.flatMap(t => t.urlSamples);

Reading from File

import fs from 'fs';
import { compress } from 'logpare';

const logContent = fs.readFileSync('app.log', 'utf-8');
const lines = logContent.split('\n').filter(Boolean);

const result = compress(lines, {
  format: 'detailed',
  maxTemplates: 20,
});

console.log(result.formatted);