Table of Contents

Theory

Regression in machine learning is a type of supervised learning algorithm that is used to predict a continuous outcome variable (also known as the dependent or target variable) based on one or more predictor variables (also known as independent variables or features). The goal of regression is to find the best-fit line (or curve, in the case of polynomial regression and other nonlinear techniques) that can accurately predict the output values within a range.

The simplest form of regression is linear regression, which aims to find the line that best fits the data points. Mathematically, this line can be represented by an equation $y=mx+c$, where y is the dependent variable, x is the independent variable, m is the slope of the line, and c is the y-intercept.

More advanced types of regression include:

• Multiple Linear Regression: Multiple predictor variables are used to predict the outcome.
• Polynomial Regression: The relationship between the independent and dependent variables is modeled as an nth degree polynomial.
• Ridge and Lasso Regression: These are regularization methods to help prevent overfitting.
• Logistic Regression: Despite its name, it is used for binary classification problems but uses the concept of the logistic function to model the probabilities.

Why Sklearn's Linear Regression Has No Hyperparameters?

• Stochastic Gradient Descent VS OLS method

Evaluation Metrics

In regression problems, evaluating the performance of a model is crucial for understanding how well the model is performing and where it can be improved. Below are some commonly used metrics for evaluating regression models:

Mean Absolute Error (MAE)

This is the simplest and most straightforward metric, calculated as the average of the absolute differences between the predicted and actual values.

$$ \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i| $$

Mean Squared Error (MSE)