Custom Preprocessing

Mask domain-specific IDs, retokenize non-whitespace log formats, and vary the similarity threshold by depth.

Learn how to create custom preprocessing strategies for domain-specific log formats.

Overview

Preprocessing transforms raw log lines before template extraction. It's crucial for:

Masking variable data (IDs, tokens, values)
Normalizing inconsistent formatting
Handling domain-specific patterns
Improving compression quality

Default Preprocessing

logpare includes built-in patterns for common variables:

import { DEFAULT_PATTERNS } from 'logpare';

console.log(Object.keys(DEFAULT_PATTERNS));
// [
//   'isoTimestamp', 'clockTime', 'uuid', 'unixTimestamp', 'url',
//   'ipv4', 'ipv6', 'port', 'hexId', 'blockId', 'filePath',
//   'numericId', 'numbers'
// ]

DEFAULT_PATTERNS is a Record<string, RegExp>, applied in insertion order — more specific patterns run first so broader ones do not fragment their matches. Each match is replaced with <*>, and adjacent wildcards are then collapsed into one, so 10.251.31.5:50010 becomes a single <*> rather than two.

The last pattern, numbers, masks every bare number. Short numbers such as an HTTP 404 or line:123 are therefore not preserved by the defaults. If you need them, build a strategy whose preprocess applies only the patterns you want.

See the full key table for what each pattern matches.

Creating Custom Strategies

Use defineStrategy() to create a custom preprocessing strategy:

import { defineStrategy } from 'logpare';

const customStrategy = defineStrategy({
  preprocess(line: string): string {
    // Transform the line
    return line;
  },

  tokenize(line: string): string[] {
    // Split line into tokens
    return line.split(/\s+/).filter(Boolean);
  },

  getSimThreshold(depth: number): number {
    // Return similarity threshold for this depth
    return 0.4;
  }
});

All three methods are optional — anything you leave out falls back to the default strategy.

There is also a shortcut for the common case of "defaults plus a few of my own patterns". Pass patterns instead of writing preprocess yourself, and they are merged over DEFAULT_PATTERNS once at definition time:

import { defineStrategy } from 'logpare';

const strategy = defineStrategy({
  patterns: {
    orderId: /order-[A-Z0-9]{8}/g,
    userId: /user_\d+/g,
    sessionId: /session-[a-f0-9]{32}/gi,
  },
});

Supplying your own preprocess takes precedence and ignores patterns.

Common Patterns

Adding Custom ID Patterns

Mask application-specific identifiers:

import { compress, defineStrategy, DEFAULT_PATTERNS, WILDCARD } from 'logpare';

// Preferred: let defineStrategy merge your patterns over the defaults
const strategy = defineStrategy({
  patterns: {
    orderId: /order-[A-Z0-9]{8}/g,
    userId: /user_\d+/g,
    sessionId: /session-[a-f0-9]{32}/gi,
    requestId: /REQ-\d{10}/g,
  },
});

// Or, if you need full control of the ordering, write preprocess yourself.
// Note DEFAULT_PATTERNS are applied in insertion order, so iterate values in order.
const explicit = defineStrategy({
  preprocess(line: string): string {
    let result = line;

    for (const pattern of Object.values(DEFAULT_PATTERNS)) {
      result = result.replace(pattern, WILDCARD);
    }

    result = result.replace(/order-[A-Z0-9]{8}/g, WILDCARD);
    result = result.replace(/user_\d+/g, WILDCARD);

    return result;
  }
});

compress(logs, { drain: { preprocessing: strategy } });

E-commerce Logs

const ecommerceStrategy = defineStrategy({
  preprocess(line: string): string {
    return line
      // Order IDs
      .replace(/order-[A-Z0-9]{8}/g, '<*>')
      // Product SKUs
      .replace(/SKU-\d{6}/g, '<*>')
      // Prices
      .replace(/\$\d+\.\d{2}/g, '<*>')
      // Customer IDs
      .replace(/cust_[a-z0-9]{16}/g, '<*>')
      // Cart IDs
      .replace(/cart-[A-Z0-9]{12}/g, '<*>')
      // Apply defaults
      .replace(/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/g, '<*>');
  }
});

Multi-tenant SaaS Logs

const saasStrategy = defineStrategy({
  preprocess(line: string): string {
    return line
      // Tenant IDs
      .replace(/tenant-[a-z0-9]{16}/g, '<*>')
      // Organization IDs
      .replace(/org_[A-Z0-9]{12}/g, '<*>')
      // Workspace IDs
      .replace(/workspace_\d+/g, '<*>')
      // API keys (partial)
      .replace(/sk_live_[A-Za-z0-9]{24}/g, '<*>')
      .replace(/pk_live_[A-Za-z0-9]{24}/g, '<*>')
      // User emails
      .replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g, '<*>');
  }
});

Kubernetes/Container Logs

const k8sStrategy = defineStrategy({
  preprocess(line: string): string {
    return line
      // Pod names
      .replace(/\b[a-z0-9-]+-[a-z0-9]{8,10}-[a-z0-9]{5}\b/g, '<*>')
      // Container IDs
      .replace(/[0-9a-f]{64}/g, '<*>')
      // Deployment revision
      .replace(/revision=\d+/g, 'revision=<*>')
      // Resource versions
      .replace(/resourceVersion:\s*"\d+"/g, 'resourceVersion:"<*>"')
      // Image tags
      .replace(/(:\s*v?\d+\.\d+\.\d+(-[\w\.]+)?)/g, ':<*>');
  }
});

Custom Tokenization

CSV Logs

Split on commas instead of whitespace:

const csvStrategy = defineStrategy({
  tokenize(line: string): string[] {
    return line.split(',').map(token => token.trim());
  }
});

Tab-Separated Logs

const tsvStrategy = defineStrategy({
  tokenize(line: string): string[] {
    return line.split('\t').filter(Boolean);
  }
});

JSON Logs

Extract specific fields for tokenization:

const jsonStrategy = defineStrategy({
  preprocess(line: string): string {
    try {
      const parsed = JSON.parse(line);
      // Create a normalized format
      return `${parsed.level} ${parsed.message || ''} ${parsed.context || ''}`;
    } catch {
      // Fallback for non-JSON lines
      return line;
    }
  },

  tokenize(line: string): string[] {
    return line.split(/\s+/).filter(Boolean);
  }
});

Depth-Based Similarity Thresholds

Adjust matching strictness by tree depth:

const adaptiveStrategy = defineStrategy({
  getSimThreshold(depth: number): number {
    // More lenient for shallow depths (first few tokens)
    if (depth <= 2) return 0.3;

    // Default for middle depths
    if (depth <= 4) return 0.4;

    // Stricter for deeper levels
    return 0.5;
  }
});

Use case: When initial tokens are highly variable but later tokens are consistent.

Testing Custom Strategies

Verify your strategy works as expected:

import { compress, defineStrategy, WILDCARD } from 'logpare';

const strategy = defineStrategy({
  preprocess(line: string): string {
    return line.replace(/order-[A-Z0-9]{8}/g, WILDCARD);
  }
});

// Test preprocessing
const input = 'Processing order-ABC12345 for user 123';
const output = strategy.preprocess(input);

console.log(output);
// "Processing <*> for user 123"
// Note: this strategy overrides preprocess entirely, so the default patterns
// (which would also mask "123") do not run.

// Test with compress
const result = compress([
  'Processing order-ABC12345 for user 123',
  'Processing order-XYZ98765 for user 456',
], { drain: { preprocessing: strategy } });

console.log(result.templates[0].pattern);
// Expected: "Processing <*> for user <*>"

Best Practices

Apply defaults first - Start with DEFAULT_PATTERNS then add custom patterns
Test incrementally - Add patterns one at a time and verify results
Be specific - Use precise regex to avoid over-matching
Cache patterns - Compile regex once, reuse many times
Document patterns - Comment what each pattern matches
Validate input - Handle malformed logs gracefully
Monitor performance - Complex regex can slow processing

Debugging Tips

Inspect Preprocessing Output

const strategy = defineStrategy({
  preprocess(line: string): string {
    const result = line.replace(/custom-pattern/g, '<*>');
    console.log(`Before: ${line}`);
    console.log(`After: ${result}`);
    return result;
  }
});

Check Pattern Matches

const testPattern = /order-[A-Z0-9]{8}/g;
const testLine = 'Processing order-ABC12345';

const matches = testLine.match(testPattern);
console.log('Matches:', matches);
// Output: ["order-ABC12345"]

Compare Results

// Without custom preprocessing
const result1 = compress(logs);
console.log(`Templates: ${result1.stats.uniqueTemplates}`);

// With custom preprocessing
const result2 = compress(logs, { drain: { preprocessing: customStrategy } });
console.log(`Templates: ${result2.stats.uniqueTemplates}`);
console.log(`Improvement: ${result1.stats.uniqueTemplates - result2.stats.uniqueTemplates}`);