Download as PDF, PPTX
Uploaded byRoelof Pieters
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
The document discusses deep learning and its significance in image understanding through convolutional neural networks, highlighting its ability to learn features automatically as opposed to manually designed features. It outlines the deep learning process, including data preprocessing, model architecture selection, training, optimization, and debugging techniques, while emphasizing the importance of regularization methods like dropout and batch normalization. Furthermore, it provides resources for implementing deep learning in Python and references various research studies related to the field.
In this document
Powered by AI
Introduction to deep learning with convolutional neural networks by Roelof Pieters, a PhD candidate and data science consultant.
Overview and definition of deep learning, emphasizing its algorithmic nature and learning capabilities across various levels of abstraction.
Deep learning eliminates handcrafted features, using learned features adaptable to visual and linguistic information, structured as hierarchical models.
Historical advancements in audio and image recognition via deep learning (e.g., ImageNet classification by Krizhevsky in 2010).
Explaining the inner workings of deep learning models, including activation functions and optimization strategies.
Available Python libraries for deep learning, including Theano, Caffe, Keras, and Lasagne.
Training a neural network involves preprocessing data, selecting architecture, optimizing, and regularization.
Steps for effective data preprocessing, including normalization and PCA, essential for improving model performance.
Selecting the appropriate neural network architecture such as DBN, CNN, and RNN for specific tasks.
Framework for training neural networks focusing on data processing, architecture selection, training, and optimization.
Visualizing and interpreting loss curves, accuracy, and weights to debug and improve training processes.
Strategies for optimizing hyperparameters and employing techniques like dropout and data augmentation to reduce overfitting.
Use of ensemble techniques to enhance model predictions by aggregating outputs of multiple models.
Discussion on vulnerabilities in machine learning systems, including saliency maps, and resources for further learning.
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
- 1. for Image Understanding: Deep Learningwith Convolutional Neural Nets Roelof Pieters PhD candidate at KTH & Data Science consultant at Graph Technologies @graphific London 2015roelof@graph-technologies.com
- 2.
- 3. “Deep learning isa set of algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction.” (much debated definition) A Definition
- 4. • A hostof statistical machine learning techniques • Enables the automatic learning of feature hierarchies • Generally based on artificial neural networks A typology Deep learning is:
- 5. Manually designed featuresare often over-specified, incomplete and take a long time to design and validate Learned features are easy to adapt, fast to learn Deep learning provides a very flexible, (possibly?) universal, learnable framework for representing world, visual and linguistic information. Deep learning can learn unsupervised (from raw text/ audio/images/whatever content) and supervised (with specific labels like positive/negative) Summary by Richard Socher. Old vs new school?
- 6. No More HandcraftedFeatures ! 6
- 7.
- 8. input layer output layer hiddenlayers Feature Hierarchies: Vision
- 9. input layer output layer hiddenlayers Feature Hierarchies: Audio
- 10. cars elephants chairs FeatureHierarchies: And so on…
- 11. History: audio recognition chartby Clarifai
- 12. History: image recognition chartby Clarifai
- 13. History: image recognition Krizhevskyet al. ImageNet Classification with Deep Convolutional Neural Networks, ILSVRC2010
- 16. 16 Karpathy, A., FeiFei, L. (2015) Deep Visual-Semantic Alignments for Generating Image Descriptions Image-Text: Joint Visual Semantic embeddings
- 17.
- 19.
- 20.
- 22.
- 25.
- 26.
- 27.
- 28.
- 29. So what’s therefor Python?
- 30. python has awide range of deep learning-related libraries available Deep Learning with Python Low level High level (efficient gpu-powered math) (computer-vision oriented DL framework, model-zoo, prototxt model definitions) pythonification ongoing! (wrapper for theano, yaml, experiment-oriented) (theano-wrapper, models in python code, abstracts theano away) (theano-extension, models in python code, theano not hidden) and of course:
- 31. python has awide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org deeplearning.net/software/pylearn2 keras.io lasagne.readthedocs.org/en/latest and of course:
- 32. python has awide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org deeplearning.net/software/pylearn2 keras.io lasagne.readthedocs.org/en/latest and of course: we will use lasagne in our examples
- 33.
- 34. 1. Preprocess thedata 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
- 35. 1. Preprocess thedata 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
- 36. • Mean subtraction •Normalization • PCA and Whitening 1. Preprocess the data
- 37. • Mean subtraction •Normalization • PCA and Whitening 1. Preprocess the data: Normalization (mean image visualised of cifar-10)
- 38. • Mean subtraction •Normalization • PCA and Whitening 1. Preprocess the data: PCA & Whitening
- 39. 1. Preprocess thedata: PCA & Whitening
- 41. 1. Preprocess thedata, the right way Warning: • compute preprocessing statistics on training data • apply on all (training / validation / test) data
- 42. 1. Preprocess thedata 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
- 43. • Deep BeliefNetwork (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
- 44. • Deep BeliefNetwork (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
- 45. • Deep BeliefNetwork (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
- 46. • Deep BeliefNetwork (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
- 47.
- 48.
- 51. DrawCNN: visualizing theunits' connections Agrawal, et al. Analyzing the performance of multilayer neural networks for object recognition. ECCV, 2014 Szegedy, et al. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013 Zeiler, M. et al. Visualizing and Understanding Convolutional Networks, ECCV 2014
- 52. 1. Preprocess thedata 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
- 53. 1. Preprocess thedata 2. Choose architecture 3. Train (Code Finally!) 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
- 54. Training a (deep)Neural Network (…) layer definitions layer parameters
- 55. Training a (deep)Neural Network hyper parameters
- 56. Training a (deep)Neural Network Lift off!
- 57. Training a (deep)Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Debug
- 58. Debug Training: VisualizeLoss Curve
- 59. Debug Training: VisualizeLoss Curve
- 60. Debug Training: VisualizeLoss Curve This looks (too) linear: lower your learning rate! This looks (too) wide: increase your batch size!
- 61. Debug Training: VisualizeAccuracy big gap: overfitting: regularize! no gap: underfitting (increase model size)
- 62. Debug Training: VisualizeWeights (usually: first layer) complete mess, doesn't get past the random initialisation
- 63. Debug Training: VisualizeWeights (usually: first layer) better, but still “noisy” weights mostly solvable by stronger regularisation
- 64. Debug Training: VisualizeWeights (usually: first layer) good: now operates as “edge detector”
- 65. Training a (deep)Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Debug
- 66. • Tweak Hyperparameters/ Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
- 67. • Tweak Hyperparameters/ Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
- 68. • Grid searchwon't work on your millions + parameters • Random Search? Mwah… • Bayesian Optimization: Yeh baby! • Spearmint: https://github.com/HIPS/Spearmint • Hypergrad: https://github.com/HIPS/hypergrad Choosing Hyperparameters
- 69. • Tweak Hyperparameters/ Architecture • Data Augmentation • Dropout • Batch Normalization Overfitting
- 70.
- 71. Data Augmentation http://benanne.github.io/2015/03/17/plankton.html (realtime dataaugmentation at Kaggle’s #1 National Data Science Bowl ≋ Deep Sea ≋ team) rotation: random with angle between 0° and 360° (uniform) translation: random with shift between -10 and 10 pixels (uniform) rescaling: random with scale factor between 1/1.6 and 1.6 (log- uniform) flipping: yes or no (bernoulli) shearing: random with angle between -20° and 20° (uniform) stretching: random with stretch factor between 1/1.3 and 1.3 (log- uniform)
- 72. • Tweak Hyperparameters/ Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
- 73. Dropout as Regularization (naivelytrained net) Overfits !
- 74. Dropout as Regularization (naivelytrained net) Overfits ! Dropout!
- 75. Dropout as Regularization (naivelytrained net) Overfits ! Dropout!
- 76. Dropout as Regularization (naivelytrained net) (net with dropout) less strongly overfitted & can run for more epochs higher accuracy
- 77. Overfitting • Tweak Hyperparameters/ Architecture • Data Augmentation • Dropout • Batch Normalization
- 78. •Normalize the activationsin each layer within a minibatch •Learn the mean and variance of each layer as parameters Batch Normalization as regularization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift S Ioffe and C Szegedy (2015) 85% 50% 15%
- 79. Training a (deep)Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Further Tips & Tricks to improve Model Accuracy Debug
- 80. • Ensembles • Finetuningpre-trained/earlier-trained net • Sticking extracted layer features in another classifier (ie SVM) Other “Tricks”
- 81. • Ensembles • Finetuningpre-trained/earlier-trained net • Sticking extracted layer features in another classifier (ie SVM) Other “Tricks”
- 82. Ensembles • majority votewhen hard predictions (ie classes) • average vote for soft predictions (continious scale) • make sure classifiers are uncorrelated • cross validate ensemble weights (by grid search, or rank average) • stacked • blending
- 83. Ensembles (10 similar netswith varying hyperparameters on same tiny-imagenet dataset)
- 84. avg: 0.3647 Ensembles (10 similarnets with varying hyperparameters on same tiny-imagenet dataset)
- 85. predict by meanof all: 0.4244 avg: 0.3647 Ensembles (10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
- 86. (10 similar netswith varying hyperparameters on same tiny-imagenet dataset) predict by mean of all: 0.4244 leave out model9: 0.4259 avg: 0.3647 Ensembles
- 87. (10 similar netswith varying hyperparameters on same tiny-imagenet dataset) 0.4259 0.4244 0.3647 Ensembles
- 88. danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/ (ensembling specialist netsby Daniel Nouri, Kaggle facial keypoint tutorial) Ensembles “Specialists” Ensemble
- 89. https://www.kaggle.com/c/facial-keypoints-detection/ try it yourself:) 3 similar nets trained on the same data but with different hyper parameters. RMSE’s: •2,08449 •2,04575 •2.01565 together: 1.93397 disclaimer: Kaggle is not real life, people…
- 90. Machine learning systems caneasily be fooled but beware… / no free lunch:
- 91. Saliency Maps K. Simonyan,A. Vedaldi, A. Zisserman , "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps", ICLR Workshop 2014 first we predict on a pixel level
- 92. Fooling ConvNets Szegedy, Christian,et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images." arXiv preprint then we do our “magic”
- 93. Fooling ConvNets Szegedy, Christian,et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images." arXiv preprint then we do our “magic”
- 94.
- 95. Failing ConvNets “Suddenly, aleopard print sofa appears”, rocknrollnerd.github.io
- 96. Failing ConvNets “Suddenly, aleopard print sofa appears”, rocknrollnerd.github.io
- 97. Failing ConvNets “Suddenly, aleopard print sofa appears”, rocknrollnerd.github.io
- 98.
- 99. thanks for listening;) questions? or find me @graphific
- 100. • Computer Vision: Fei-Fei Li & Andrej Karpathy, Stanford course “Convolutional Neural Networks for Visual Recognition” http://vision.stanford.edu/teaching/cs231n • Natural Language Processing: Richard Socher, Stanford course “Deep Learning for Natural Language Processing”, http://cs224d.stanford.edu/ • Neural Nets: Geoffrey Hinton, Coursera/Toronto, “Neural Networks for Machine Learning" https://www.coursera.org/course/neuralnets Wanna Play? • Bunch of tutorials: http://deeplearning.net/tutorial/ • Book: Yoshua Bengio, et al, “Deep Learning” http://www.iro.umontreal.ca/~bengioy/dlbook/ • UFLDL Tutorial http://deeplearning.stanford.edu/tutorial/ • Reading Lists: http://deeplearning.net/reading-list/ http://memkite.com/deep-learning-bibliography/ • Podcast Talking Machines, http://www.thetalkingmachines.com/