DEV Community

Amit
Amit

Posted on

Dev Log D-07 to D-09. My Backprop is/will sucking/suck.

Don't learn here. All code is implementing based on Noob Imagination and poor understanding of Articles.

import mpmath as mpMath mpMath.mp.dps = 5 class Neuron: def __init__(self,W): # Weight is an vector.  self.W = W self.B = np.random.uniform(-1, 1) def forward(self,X): self.wX = X.dot(self.W) self.f_out = self.wX + self.B return self.f_out def tanh(self): self.out_tanh = mpMath.tanh(self.f_out) return self.out_tanh def derv(self,w_index,step_v,X): u = self.W.copy() print(f'Printing old Weight \n{self.W[w_index]}') u[w_index] += step_v print(f'Printing update weight \n {u[w_index]}') _d = (self.f_out - X.dot(u))/step_v return _d neuron = Neuron( np.random.uniform(-1.,1.,size=(784,)).astype(np.float16))
Enter fullscreen mode Exit fullscreen mode 
print(neuron.forward(train_data[0])) print(neuron.derv(2,0.001,train_data[0]))
Enter fullscreen mode Exit fullscreen mode

Slope when step is 0.00001

-707.2305512337132 Printing old Weight 0.496826171875 Printing update weight 0.496826171875 39243.93562717796
Enter fullscreen mode Exit fullscreen mode

As you can see output below when I update weight 0.001 there is 951 slope and If I go too small slope is even large. I don't understand.

-5.366328888592351 Printing old Weight -0.47216796875 Printing update weight -0.47119140625 951.3314431337232
Enter fullscreen mode Exit fullscreen mode

Turns out this noob brain of mine forgot to add bais.

 _d = (self.f_out - (X.dot(u)+self.B))/step_v)
Enter fullscreen mode Exit fullscreen mode

so moving on.

Below is the code snippet how I am finding out how my model is predicting.

 def predict(self,X): self.inputL_out = self.inputL.forward(X) # print(f"Input Layer Output---> \n \t {self.inputL_out}")  # self.inputL.display_output()  for l in range(0,len(self.neuronsLayers)): if l == 0: self.neuronsLayers[l].forward(self.inputL_out) else: self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out) print(f'Forward output of layer {l} is :--> {self.neuronsLayers[l].f_out}') # self.neuronsLayers[l].display_output1()  if l == len(self.hidden_layer_dim_arr)-1: self.outL_f_out = self.outL.forward(self.neuronsLayers[l].f_out) print(f'output of outL \n \t{self.outL_f_out}') print("Predicting--->") max_index = np.argmax(self.outL_f_out) print(f"index of max confidente {max_index} and probability is {self.outL_f_out[max_index]}")
Enter fullscreen mode Exit fullscreen mode

So, My last layer have 10 neurons. each index represent each digit.
like index 2 represent 2.

Now I am stuck on another bug.
I works drev method works for 0.001 but it is 0.0 when I change setup to 0.0001.
I think I should learn basic Numpy and Python. But I will not it is fun doing things this way. Getting stuck on silly bugs.

Top comments (0)