You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,6 @@ For learning, visualizing and understanding the optimization techniques and algo
4
4
#
5
5
#
6
6
7
-
8
7
Visualizations and in depth concepts of the Machine Learning optimization algorithms are discussed and shown here with different functions as examples and understanding the differences by comparing them.
9
8
## Over all types of Line and direction search algorithms in Gradient Descent discussed here: :shipit:
10
9
- Line search & direction search in Gradient Descent:
@@ -44,7 +43,7 @@ Here we go,
44
43
First some visualizations of functions will definitely make you curious to know more about the optimization, so,
45
44
look at the functions think how to find the minimum starting from a arbitrary point,
We will work with simple univariate and bivariate functions for understanding,:yawning_face: One convex and one non-convex function is shown below,
@@ -113,8 +112,8 @@ As we have added noise to perform a SGD, it becomes tough now to reach the minim
113
112
114
113
#
115
114
## Comparison of different methods of dynamic step size:
116
-
Here polynomially decreases step size is used but you can use exponential functions to handle eta, or you may keep it constant or step wise decreasing, a plot showing comparison among these different methods are shown here,,, these different methods behave differently in different function, so be careful. But in most cases Polynomially decreasing $\eta$ is doing better control. The below methods are discussed inside the ipynb file.
117
-
-[x] Constant Learning Rate ,i.e value of $\eta$ is constant with iteration. As a result step length will decrease with decrease in gradient value
115
+
Here polynomially decreases step size is used but you can use exponential functions to handle eta, or you may keep it constant or step wise decreasing, a plot showing comparison among these different methods are shown here,,, these different methods behave differently in different function, so be careful. But in most cases Polynomially decreasing <imgsrc="https://render.githubusercontent.com/render/math?math=\eta"> is doing better control. The below methods are discussed inside the ipynb file.
116
+
-[x] Constant Learning Rate ,i.e value of <imgsrc="https://render.githubusercontent.com/render/math?math=\eta"> is constant with iteration. As a result step length will decrease with decrease in gradient value
118
117
-[x] Step Wise decreasing LR , i.e, LR is reduced by a fraction when function balue in any iteration is increased
119
118
-[x] Exponentially decresing LR with iteration
120
119
-[x] Polynomially decreasing LR
@@ -123,7 +122,7 @@ Here polynomially decreases step size is used but you can use exponential functi
123
122
124
123
#
125
124
## For the non-convex Surface:
126
-
Finally we will see how tough this is for a non-convex surface ( the function used here is shown above $\large f(x_1, x_2) = x_1^2 - 2 x_2^2$ ),
125
+
Finally we will see how tough this is for a non-convex surface ( the function used here is shown above <imgsrc="https://render.githubusercontent.com/render/math?math=f(x_1, x_2) = x_1^2 - 2 x_2^2" > ),
127
126
128
127

129
128

@@ -153,3 +152,5 @@ Thats it. Go to the ipynb files now.
153
152
Also I think, it will be better if anyone want to help me by just making the ipynb files more understandable by separation the topics.
154
153
If you feel hard anywhere, contact me in mahendranandi.0608@gmail.com
0 commit comments