Skip to main content

Log Anomaly Detector

The idea was born when triaging different failures from regressions, especially when a bug was introduced in a sanity suit. Instead of spending engineer's time in manually analyzing the log files, the tool automates this process by approximately comparing the log files with a reference log files. A lot of persistant warning or error messages that can be misleading to (especially fresher) engineers can be handled diligently by the tool as it compares the log files with a reference log files, which likely have similar messages.

Idea​

If a message is present in a passing and failing log, its most likely not contributing to the failure. But a message that is present in a failing log but not in a passing log, is likely contributing to the failure. Such a unique pattern is called as anomaly.

Technical Approach​

A simple diff won't work due to noise (timestamps, file paths), unordered lines, parallel jobs, and template messages with variable parts. The system uses a multi-stage ML pipeline to handle this complexity.

Pipeline Architecture​

  • Preprocessing: Noise filtering, lemmatization (NLTK WordNet), stop word removal, duplicate elimination, custom rules for timestamps/pathnames
  • Vectorization: Hashing Vectorizer with Trigram N-grams (Unigram→0.92 score, Trigram→1.0 score)
  • Anomaly Detection: Nearest Neighbor (KD-Tree) with Euclidean distance.
  • Classification: Agglomerative Clustering (no cluster count required) groups similar anomalies
  • HTML Interface: Interactive regression analysis with embedded feedback collection to grade the algorithm.

Status​

Deployed for a while in production at Synopsys. Continued development — including LLM integration and MCP-based triage — is documented in the work project page.

Duration: Nov 2021 — Feb 2022 (Final Semester)
Role: Developer (Master's Thesis Project)
Institution: BITS Pilani
Degree: M.Tech in Data Science & Engineering