Feature Engineering Patterns for Real-World ML
Model quality on tabular data is often decided by features, not model family. A disciplined feature process can turn an average model into a strong production system.
Why Feature Engineering Still Matters
Even with modern deep learning, many business systems are structured-data problems. In these settings, feature quality controls:
- signal strength
- data efficiency
- model stability
- explainability
A weak feature pipeline cannot be fixed by tuning a bigger model.
Core Feature Types
- numeric raw features: amount, count, duration
- categorical features: plan type, channel, region
- temporal features: hour of day, day of week, seasonality markers
- aggregate features: rolling counts, averages, max/min windows
- interaction features: combinations that express domain logic
Start simple, then add features that encode concrete business hypotheses.
High-Value Patterns
1) Windowed Aggregations
Examples:
- purchases in last 7/30 days
- average ticket value in last 90 days
- support interactions in last 14 days
Windowed features capture short-term behavior shifts and are often predictive of churn, fraud, and demand.
2) Ratio Features
Examples:
- successful payments / attempted payments
- active days / account age
- returned orders / total orders
Ratios normalize across users/entities with different scales.
3) Recency Features
Examples:
- days since last purchase
- hours since last login
Recency is usually one of the strongest predictors in customer behavior models.
4) Frequency Encoding
For high-cardinality categories, frequency/count encoding can outperform naive one-hot in memory and speed.
Leakage Prevention Is Non-Negotiable
Most feature disasters are leakage disasters.
Common leakage mistakes:
- feature includes future events
- aggregations use data beyond prediction timestamp
- random split for time-dependent behavior
Leakage-proof design:
- define prediction timestamp contract
- generate features only from data available before that timestamp
- validate with time-aware splits
If leakage exists, offline metrics are fiction.
Categorical Encoding Strategy
Choose encoding by cardinality and model type.
- low cardinality: one-hot
- medium/high cardinality: target encoding with strict leakage controls
- tree models: often tolerate label/frequency encoding better
Target encoding must be done with out-of-fold strategy to avoid label leakage.
Missing Values as Signal
Missingness is often informative. Do not blindly impute without tracking missing indicators.
Pattern:
- create
is_missing_feature_x - impute numeric with median
- impute categorical with explicit “unknown”
This preserves information about data collection behavior.
Feature Selection Workflow
Use a layered approach:
- domain sanity filtering
- variance and redundancy checks
- model-based importance (permutation/SHAP)
- ablation experiments
Keep only features that improve validation metrics and operational reliability.
Training-Serving Consistency
A feature is production-ready only if it can be computed online or in batch exactly as trained.
Checklist:
- same transformation code path for training/inference
- versioned feature definitions
- schema validation
- backfill reproducibility
Training-serving skew silently kills model performance.
Common Mistakes
- feature creation before defining prediction timestamp
- too many weak features without ablation
- one-hot encoding very high-cardinality columns
- no feature versioning or lineage
- no data quality checks on feature distributions
Key Takeaways
- Feature engineering is often the highest-leverage work in practical ML.
- Leakage control must be built into feature generation logic.
- Strong feature pipelines are versioned, reproducible, and serving-compatible.
- Feature quality plus disciplined validation beats random model complexity.