Cascading Failure Prediction
- TYPE
- Spatio-Temporal GNN
- STACK
- 10 COMPONENTS
- STATUS
- SOURCE AVAILABLE
- FEATURED
- —
- LINKS
- SOURCE ↗
01 / OVERVIEW
A dynamic spatio-temporal graph neural network trained on Google's Borg cluster traces that predicts which data-center machines are about to fail — and how that failure would cascade to their neighbors.
Built as a 4-person team project for the Interdisciplinary Deep Learning on Graphs course at PES University, this is an end-to-end pipeline — preprocessing, training, and evaluation — that models a cloud data center as a dynamic graph and predicts machine failures roughly 15 minutes ahead of time. It runs on Google's 2019 Borg cluster trace: ~4,900 machines across ~8,900 five-minute windows, with graph edges rebuilt every window from live cluster/collection membership instead of one fixed topology. A V6 debugging pass found that the richest columns — average, maximum, and sampled CPU/memory usage — were stored as dict-like strings (e.g. "{'cpus': 0.0046, 'memory': 0.0123}") that a blanket pd.to_numeric() call had been silently zeroing out, which is why AUROC had been stuck at exactly 0.50; parsing them into real numeric features finally gave the model genuine signal to learn from. The label design also means roughly 70% of (machine, window) pairs with real feature data are tagged 'about to fail' — the opposite of the usual rare-failure problem — so the focal loss (alpha=0.3) is tuned to protect the minority NORMAL class instead. The final 2-layer GraphSAGE + GRU + attention model reaches 0.745 AUROC, with its decision threshold deliberately tuned toward recall — catching close to 100% of failing machines at the cost of a low 0.09 precision, a trade-off the evaluation script exposes explicitly via a full threshold sweep. Every part of the pipeline is engineered around an 8GB laptop GPU: hard-capped batch size and hidden dimension, sparse packed feature/edge storage instead of a 41GB dense tensor, and scheduled thermal cooldowns between epochs.
02/ STACK & ARCHITECTURE
BACKEND
- Python
- PyTorch
- PyTorch Geometric
- Pandas
- NumPy
- scikit-learn
- NetworkX
- Matplotlib
- PyArrow (Parquet)
- PyYAML
03 / WHAT WAS ACTUALLY HARD
- 01
V6 fix: parsing dict-string usage columns unstuck AUROC from 0.50
average_usage, maximum_usage, and random_sample_usage arrived as dict-like strings such as "{'cpus': 0.0046, 'memory': 0.0123}". Earlier preprocessing ran pd.to_numeric(errors='coerce') over them, silently zeroing out the six most informative columns and leaving the model with no real signal — AUROC sat at exactly 0.50 because of it. Rewriting the parser to pull cpus/memory out of each string into avg_cpu, avg_memory, max_usage_cpu, max_usage_memory, sample_cpu, and sample_memory columns is what finally gave the model working features to learn from.
- 02
Per-timestep dynamic graph topology
Instead of one fixed adjacency matrix, edges are rebuilt for every 5-minute window from live cluster and job-collection membership — a machine that joins a busy collection gains new neighbors, one that leaves loses them. The GraphSAGE forward pass takes a list of six edge_index tensors, one per timestep in the sequence, so the model watches the topology itself evolve rather than reading node features on a graph frozen in time.
- 03
Leak-proof ghost injection for masked training
Only a small slice of the ~4,900 possible node slots are active in any given window (~22 typically have real feature data), and even after the mask is expanded to graph neighbors, some failing nodes still fall outside it. The fix force-includes those missing failing nodes, plus ten decoy negative 'ghost' nodes per missing positive, so simply being force-included into the mask can't become a shortcut the model learns as 'this node is failing.'
- 04
Sparse packed storage for a graph too large to keep dense
A dense [windows, nodes, features] tensor for this dataset would need roughly 41.5GB. DynamicGraphLoader instead pre-computes every feature, label, and edge tensor once into a sparse packed CPU shared-memory format (~10MB total) and reconstructs per-window dense slices on the fly — on-GPU when available — keeping the whole pipeline inside 16GB RAM / 8GB VRAM.
- 05
Thermal-aware training loop on an 8GB laptop GPU
Trained natively on Windows 11 on an RTX 4060 Mobile with hard limits enforced directly in code — batch size capped at 2, hidden dimension capped at 32 regardless of what config.yaml requests, num_workers forced to 0 to dodge Windows' Error 1455 — plus scheduled 1-second pauses every 50 batches and a 15-second pause between epochs that held the GPU around 80-87°C instead of thermal-throttling at 89°C.
- 06
Gradient-based interpretability: critical nodes, uncertainty, and contagion scoring
Beyond prediction, evaluate.py backpropagates through failure sequences to rank which machines and which specific edges carry the most failure signal (edge_score = |grad(src)| × |grad(dst)|, min-max scaled to 1.0-5.0 for a contagion-network plot), and runs 50-pass MC Dropout to quantify per-node prediction uncertainty — giving operators both a 'what's about to fail' signal and a 'which dependency to sever' recommendation.