|
1 | 1 | # pytorch-hessian-eigenthings |
2 | 2 |
|
3 | | -The `hessian-eigenthings` module provides an efficient way to compute the eigendecomposition of the Hessian for an arbitrary PyTorch model. It uses PyTorch's Hessian-vector product and stochastic power iteration with deflation to compute the top eigenvalues and eigenvectors of the Hessian. |
| 3 | +The `hessian-eigenthings` module provides an efficient (and scalable!) way to compute the eigendecomposition of the Hessian for an arbitrary PyTorch model. It uses PyTorch's Hessian-vector product and your choice of (a) the Lanczos method or (b) stochastic power iteration with deflation in order to compute the top eigenvalues and eigenvectors of the Hessian. |
4 | 4 |
|
5 | 5 | ## Installation |
6 | 6 |
|
@@ -29,12 +29,23 @@ eigenvals, eigenvecs = compute_hessian_eigenthings(model, dataloader, |
29 | 29 |
|
30 | 30 | This also includes a more general power iteration with deflation implementation in `power_iter.py`. |
31 | 31 |
|
| 32 | +## Citing this work |
| 33 | +If you find this repo useful and would like to cite it in a publication, here is a BibTeX entry: |
| 34 | + |
| 35 | + @misc{hessian-eigenthings, |
| 36 | + author = {Noah Golmant, Zhewei Yao, Amir Gholami, Michael Mahoney, Joseph Gonzalez}, |
| 37 | + title = {{pytorch-hessian-eigentings: efficient PyTorch Hessian eigendecomposition}}, |
| 38 | + month = oct, |
| 39 | + year = 2018, |
| 40 | + version = {1.0}, |
| 41 | + url = {https://github.com/noahgolmant/pytorch-hessian-eigenthings} |
| 42 | + } |
| 43 | + |
| 44 | + |
32 | 45 | ## Acknowledgements |
33 | 46 |
|
34 | | -This code was written in collaboration with Zhewei Yao, Amir Gholami, and Michael Mahoney in UC Berkeley's [RISELab](https://rise.cs.berkeley.edu). |
| 47 | +This code was written in collaboration with Zhewei Yao, Amir Gholami, Michael Mahoney, and Joseph Gonzalez in UC Berkeley's [RISELab](https://rise.cs.berkeley.edu). |
35 | 48 |
|
36 | 49 | The deflated power iteration routine is based on code in the [HessianFlow](https://github.com/amirgholami/HessianFlow) repository recently described in the following paper: Z. Yao, A. Gholami, Q. Lei, K. Keutzer, M. Mahoney. "Hessian-based Analysis of Large Batch Training and Robustness to Adversaries", *NIPS'18* ([arXiv:1802.08241](https://arxiv.org/abs/1802.08241)) |
37 | 50 |
|
38 | 51 | Stochastic power iteration with acceleration is based on the following paper: C. De Sa, B. He, I. Mitliagkas, C. Ré, P. Xu. "Accelerated Stochastic Power Iteration", *PMLR-21* ([arXiv:1707.02670](https://arxiv.org/abs/1707.02670)) |
39 | | - |
40 | | - |
|
0 commit comments