Vectorization Of Gradient Descent in python

Vectorization Of Gradient Descent in python

Vectorization is a powerful technique for writing efficient numerical computations in Python, particularly with libraries such as NumPy. By using vectorized operations, you can leverage highly optimized C and Fortran libraries without having to write slow Python loops.

When it comes to implementing gradient descent, vectorization is crucial for efficiency, especially with large datasets. Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient.

Below is a vectorized implementation of gradient descent using NumPy in Python for a simple linear regression problem:

import numpy as np # Example function: mean squared error cost function for linear regression def compute_cost(X, y, theta): m = len(y) predictions = X.dot(theta) square_errors = (predictions - y) ** 2 return 1 / (2 * m) * np.sum(square_errors) # Gradient descent function def gradient_descent(X, y, theta, alpha, num_iters): m = len(y) J_history = [] # Use a list to save cost in every iteration for i in range(num_iters): predictions = X.dot(theta) errors = np.dot(X.transpose(), (predictions - y)) theta -= (alpha / m) * errors J_history.append(compute_cost(X, y, theta)) return theta, J_history # Sample data # Add a column of ones to X for the intercept term X = np.array([[1, 2], [1, 3], [1, 4], [1, 5]]) y = np.array([[7], [6], [5], [4]]) theta = np.array([[0.1], [0.2]]) # Hyperparameters for the gradient descent alpha = 0.01 # Learning rate iterations = 1000 # Run gradient descent theta, J_history = gradient_descent(X, y, theta, alpha, iterations) # Print the optimized parameters print("Theta found by gradient descent:", theta) # Print the cost history print("Cost History:", J_history[-10:]) # Last 10 cost entries for brevity

In this implementation:

  • compute_cost() is the function that calculates the cost (mean squared error in this case) given X (input features), y (output values), and theta (parameters).

  • gradient_descent() is the function that performs gradient descent. It updates the parameters theta using the gradient of the cost function.

  • The X matrix includes the input features with an additional first column of ones to accommodate the theta_0 intercept term in linear regression.

  • The theta vector is initialized with random values and will be updated to minimize the cost function.

  • alpha is the learning rate, and iterations is the number of times gradient descent will update theta.

This code assumes you have NumPy installed:

pip install numpy

This vectorized approach is efficient and concise because it avoids explicit loops over the dataset when calculating the predictions and the gradient, utilizing NumPy's optimized operations that run at C-level speed under the hood.

More Tags

bulk xorg spp android-cardview git-filter-branch cuda hybrid-mobile-app intentfilter coffeescript price

More Programming Guides

Other Guides

More Programming Examples