0.25
0.5
0.75
1.25
1.5
1.75
2
Model Compression
Published on Feb 25, 20074549 Views
Decision trees are intelligible, but do they perform well enough that you should use them? Have SVMs replaced neural nets, or are neural nets still best for regression, and SVMs best for classificatio
Chapter list
Model Compression00:00
Outline00:39
Supervised Learning01:57
Normalized Scores for ES03:40
Ensemble Selection Works, But Is It Worth It?23:06
Computational Cost24:22
Ensemble Selection25:16
Best Ensembles are Big & Ugly!25:38
Best Ensembles are Big & Slow!26:38
Can’t we make the ensembles smaller, faster, and easier to use by eliminating some base-level models?28:25
What Models are Used in Ensembles?28:37
What Models are Used in Ensembles?30:25
Summary of Models Used by ES32:14
Motivation: Model Compression33:10
Solution: Model Compression34:53
Why Mimic with Neural Nets?37:01
Unlabeled Data?39:21
Synthetic Data: True Distribution40:23
Synthetic Data: Small Sample40:43
Synthetic Data: Random40:49
Synthetic Data: Random41:04
Synthetic Data: Random41:33
Synthetic Data: NBE42:44
These don’t work well enough. Had to develop a new, better method.44:09
These don’t work well enough. Had to develop a new, better method. Munging [1. To imperfectly transform information. 2. To modify data in a way that cannot be described succinctly.]44:10
Munging44:47
Munging44:56
Munging49:27
Munging49:42
Synthetic Data: Munge49:51
Synthetic Data: Munge50:16
Synthetic Data: Munge51:51
Synthetic Data54:29
Now That We Have a Method to Generate Data, Let’s Do Some Compression54:44
Experimental Setup: Datasets55:00
Experimental Setup55:10
Average Results by Size55:30
Average Results by Size55:58
Average Results by Size57:23
Average Results by Size58:27
Letter.P1 Results59:00
Hs Results59:18
Average Results by HU01:00:01
Letter.P1 Results01:00:55
Letter.P2 Results01:01:06
Letter Results01:01:14
It Doesn’t Always Work As Well As We’d Like, Yet!01:03:55
Covtype Results01:03:59
Covtype Results01:04:28
Covtype Results01:05:28
Covtype Results01:07:36
Adult Results01:07:50
Adult Results01:08:17
Adult Results01:08:26
RMSE Results – 400K, 256 HU01:15:40
We’re Retaining 97% of Accuracy of Target Model, but How Are We Doing on Compression? 01:16:37
Size of Models (MB)01:16:42
Execution Time of Models01:17:20
Summary of Compression Results01:17:39
Related Work01:17:50
Related Work01:18:21
Related Work01:19:46
Related Work01:21:55
What Still Needs to Be Done?01:22:18
Future Work: Other Mimic Models01:22:20
Future Work: Other Target Metrics01:22:30
Future Work: Model Complexity01:23:43
Future Work: Munge01:24:16
Future Work: Active Learning01:28:03
Thank You. Questions?01:30:00