Machine Learning

03 Regression: Fundamentals, Implementation, and Evaluation

Imron Rosyadi

00. Introduction of Regression

Regression

Regression Introduction Image

Has anyone seen a crime show or heard of an investigation where they used a footprint to determine a suspect’s height? It’s a tactic frequently mentioned in connection to forensics, but does it actually work? Let’s try it out!

Exercise (15 minutes): Use your own shoe size and height as the suspect’s, and tell students to keep in mind that US men’s size equals US women’s size - 2. Don’t tell them the suspect’s height, but tell them the suspect’s shoe size and tell them that their task will be to guess the suspect’s height.

Split the students into groups of ~6 each and give each group a sheet of graph paper.

Ask the groups to plot each group member’s shoe size on the x-axis and height on the y-axis. What do they think is the suspect’s height based on the suspect’s shoe size?

Then have the groups share data, so each has a plot of the whole class’s information. Make another guess per group. Does anyone come close?

They should theoretically have better guesses with more data, but shoe size might not actually be well correlated to height, so they might not.

Mathematical Model

Mathematical Model for Linear Regression

Linear regression has a simple goal: to find a straight line that best fits a set of data. This model is foundational in many ECE applications, such as predicting sensor outputs or analyzing circuit performance based on input parameters.

Features Go In, Targets Come Out

Features and Targets in Machine Learning

Recall that the equation for a line is y = m * x + b, where x denotes our input variable and y is our output. In the case of machine learning, x represents input features and y represents target outputs. For example, if we were trying to forecast the power consumption of a device from its operating frequency, the operating frequency would be the input feature, and the power consumption would be the target output.

What is the Machine “Learning?”

Weights and Biases in Linear Regression

Using the data, simple linear regression “learns” two values. The first is m, which you may have called “slope” and which we’ll refer to as a “weight / coefficient.” This represents how much a change in the feature value (x) should affect our prediction (y). In other words, a 1 unit increase in x yields an m unit change in y.

The second is b, which you may have called an “intercept” and which we’ll refer to as a “bias.” The bias represents the prediction we would make if our input features are all zero. For example, you may expect an electronic component to draw some baseline current even with zero input signal; this baseline would be analogous to the bias.

Multiple Features

Multiple Features in Regression

Realistically, the performance of an ECE system might depend on several factors (e.g., current drawn depends on voltage, temperature, and load resistance). Now, our model needs to learn three weights (one for each input feature) and one bias.

The concept of weights and biases is important to most machine learning models, even complex neural networks. The model uses data to learn how each input feature affects the output and it learns a bias to linearly shift its predictions to fit the data. This is like shifting a y-intercept.

Machine Learning Process

1. Infer/Predict/Forecast
2. Calculate Error/Loss/Cost
3. Train/Learn (Update parameters)
4. Iterate/Repeat (until some stopping condition)

Note

This iterative cycle is fundamental to how most machine learning algorithms “learn” from data.

But how do we “learn” the weights and biases? Typically, in machine learning we use the following iterative process.

• Given an input value, we forecast (or guess) the potential target value.
• We calculate the error (or difference) between the actual target value and the target we guessed.
• We update the weights and biases to produce a guess that is closer to the actual target.
• We iterate. That is, we repeat these steps until some stopping condition. (The stopping condition could be a small enough error, or that the error is no longer changing between iterations.)

Machine Learning Process Flow

graph TD
    A[Start] --> B{Data Input};
    B --> C[Predict Target];
    C --> D[Compare to Actual Target];
    D --> E[Calculate Error/Loss];
    E --> F["Update Model Parameters <br> (Weights & Bias)"];
    F --> G{Stopping Condition Met?};
    G -- No --> C;
    G -- Yes --> H[Model Converged];

This flowchart visually represents the iterative machine learning process. Data is fed in, a prediction is made, the error is calculated, parameters are updated, and the cycle repeats until the model achieves a satisfactory level of accuracy or reaches a predefined limit.

Predict the Selling Price of a House

House Price Data

Here are four data points. The feature (x-value) is square footage of a house (in thousands of square feet), and the target (y-value) is the price of the house (in thousands of dollars). We will use this simple example to trace the machine learning process.

Predict the Price of a House Using the Machine Learning Process

House Price Scatter Plot

Here is a different depiction of the same four data points. It is a simple scatter plot, where the x-axis is the size of a house (our feature), and on the y-axis we have the price of the house (our target).

Predict the Price of a House Using the Machine Learning Process

Initial Guess for Line

To begin the iterative machine learning process, we make an initial guess at the weights and biases. In this case, we have one weight, m, and one bias, b. Glancing at the data (but not agonizing too hard), we make a simple guess: b = 160 and m = 1.

Predict the Price of a House Using the Machine Learning Process

Line from Initial Guess

We now have an initial guess for our model’s parameters, and we use them to create the linear equation y = mx + b. This line represents our initial prediction model.

Predict the Price of a House Using the Machine Learning Process

Predicted Values Example

Now, we use this line to forecast predicted output values. For each point in our training data set (x_k, y_k), we calculate y_pred = m(x_k) + b. These are the predictions the model makes given its current parameters.

Predict the Price of a House Using the Machine Learning Process

Actual vs Forecasted Values

Here we have the actual target outputs (blue dots) and the forecasted outputs that came from our model (purple points on the line). We’ve completed the infer/predict/forecast step. Notice the vertical distance between the blue dots and the purple points; these represent the errors.

Predict the Price of a House Using the Machine Learning Process

Error Calculation Illustration

Now we move onto step 2, which is to compute error/loss/cost. We calculate the error between the actual target values, and the forecasted values. The metric we use to calculate this error can be simple Euclidean distance, but there are other measures as well. We will talk about error/cost functions in a minute, but for now it’s okay to think of the vertical distance between the actual value and the forecasted value.

Predict the Price of a House Using the Machine Learning Process

Updated Line Illustration

Finally, we update the weight and bias such that we reduce the error. The machine learning algorithm adjusts m and b in a way that minimizes the overall error. This leads to a new, better-fitting line. Now we have new m and b values, and we start at step 1 using these new parameters (predict, calculate error, update, repeat). This iterative refinement continues until the model converges or a stopping condition is met.

Interactive: Tuning Our Regression Model

Adjust the weight (slope) and bias (intercept) to fit the data and minimize the Mean Squared Error (MSE).

viewof weight_m = Inputs.range([-10, 10], {value: 1, step: 0.1, label: "Weight (m)"});
viewof bias_b = Inputs.range([100, 200], {value: 160, step: 1, label: "Bias (b)"});

This interactive plot demonstrates how drastically changing the weight (m) and bias (b) affects the regression line and, consequently, the Mean Squared Error. Try to find the optimal m and b that result in the lowest MSE. For this simple dataset, the optimal values are m=2 and b=159, which yield an MSE of 0.00. This exercise visually reinforces the goal of regression: minimizing the error between predicted and actual values by adjusting model parameters.

Error/Loss/Cost Functions

Common Loss Functions

Now let’s look at a few common loss/cost functions. Remember we use these functions to determine the error that results from a particular set of weights and biases. These are not the only loss functions, but they are very common.

• L1 Loss (Least Absolute Deviations or LAE): L1 is resistant to outliers in the data (i.e. robust). If your data has outliers that can be ignored, then L1 is a good choice. If it is important to pay attention to any and all outliers, the method of least squares is a better choice.

• L2 Loss (Least Squares): Generally, L2 loss is preferred to L1, but when outliers are present in the data, then L2 may not perform well. The reason for this is because we are squaring the difference between the actual target and the predicted target. So if the error is large (in the case of an extreme outlier), then the error function will overcompensate.

• Mean Squared Error (MSE): MSE is the average of the squared differences between predicted targets and actual targets. Due to squaring, predictions which are far away from actual values are penalized heavily in comparison to less deviated predictions (similar to L2). MSE also has nice mathematical properties which make it easier to calculate gradients, which are used to update the model parameters (weights and biases). It’s widely used in ECE for tasks like signal reconstruction error or system identification.

Housing Example

Housing Example Data Table

Let’s practice calculating each of these loss functions for the data in the housing example, with the initial model y = 70x + 100000.

Exercise (15 minutes): Have students work in small groups to calculate the loss functions based on the data in the table. It may be helpful to write the loss functions on the board at this point or flip back to the slide with the loss functions and allow students to write them down.

Housing Example (L1 Loss)

L1 Loss Calculation

The L1 Loss, also known as Mean Absolute Error (MAE), for our initial model y = 7x + 100000 for the housing data is 1.0. This is the sum of the absolute differences between the actual and predicted values.

Housing Example (L2 Loss)

L2 Loss Calculation

The L2 Loss for our initial model y = 7x + 100000 for the housing data is …. This is the sum of the squared differences between the actual and predicted values. Notice how squaring penalizes larger errors more significantly.

Housing Example (MSE)

MSE Calculation

The Mean Squared Error (MSE) for our initial model y = 7x + 100000 for the housing data is …. This is the average of the squared differences. This metric is a cornerstone for optimizing many ECE-related ML models due to its smooth, differentiable nature.

Computer vs. Human Jobs

The computer’s job:

• Start with an arbitrary guess of parameters.
• Tweak these parameters to reduce loss.
• The less the loss is changing, the less the value should be tweaked.

The human’s job:

• Choose the learning rate, a constant value which scales how far we tweak the value during each iteration.

Important

Learning rate is a hyperparameter - not a parameter in the actual model.

A hyperparameter is not a parameter in the model. In other words, it’s not a weight or bias. It is a value that is chosen by the machine learning specialist that controls how the algorithm “learns” the model parameters. This is a subtle but important distinction. In ECE terms, it’s like choosing the gain in a control loop: it’s part of the controller’s design, not a variable being optimized by the system itself during operation.

Gradient Descent

Gradient Descent Illustration

How does the model “iteratively” update its parameters? We can think of our goal as an optimization problem, where we’d like to optimize (minimize) a loss function. Machine learning models then use an “optimizer,” an algorithm to perform that optimization.

The most common optimizer is gradient descent, where the model starts by picking random values for each parameter. It then changes each in the direction that reduces loss the most. On each iteration or “step,” the model should get closer to the minimal loss until it “converges,” or reaches a point where the loss isn’t changing much between steps. (Usually this is based on some threshold, like the loss function changing by less than 0.001 between steps.) Since this isn’t a closed-form solution, gradient descent isn’t guaranteed to converge to the absolute lowest loss possible. There are more sophisticated optimizers that can sometimes do better.

You can control gradient descent by choosing the learning rate, which determines how much you tweak each parameter on each step. We call this a hyperparameter: a value you can change to change model performance, but one that isn’t “learned” by the model. This is analogous to tuning PID controller gains in an ECE system to optimize stability and response time.

Linear Algebra Notation for \(y=mx+b\)

Linear Algebra Notation for Single Feature

Here \(\theta_0\) is the bias and \(\theta_1\) is the weight (i.e. \(\theta_0 = b\) and \(\theta_1 = m\)). This notation is helpful in ECE when dealing with system identification or control, allowing for a compact representation of input-output relationships.

Linear Algebra Notation for \(y=mx+b\)

Compact Linear Algebra Notation

Using matrix/vector notation we can rewrite the equation of the line more compactly as \(\theta^T X\). This compact form is efficient for computations, especially when dealing with large datasets or multiple features, and simplifies theoretical analysis.

Multiple Regression (i.e. Multiple Features)

Multiple Features in Multiple Regression

This notational convenience can be extended to regression with multiple features. Recall our example from before where energy is a function of coffee, time of day, and temperature. In ECE, this could be predicting system latency based on processor clock speed, memory usage, and ambient temperature.

Multiple Regression Notation

Multiple Regression Compact Notation

Again, we can use \(\theta^T X\) to represent the regression equation, even with multiple features. Here, \(X\) becomes a vector of features for each data point, and \(\theta\) becomes a vector of corresponding weights (coefficients) and bias. This generalizes the single-feature case efficiently.

Closed Form, Exact Solution

\[ \theta = (X^{T} \cdot X)^{-1} \cdot X^{T} \cdot y \]

• Good for small datasets
• Finds optimal solution
• Can be computationally expensive
• Requires an invertible matrix

How does the model actually “learn” those values? Through linear algebra, there is an exact (closed form) solution. All you need to do is plug in your \(X\) (features) and \(y\) (targets) values, and calculate to get your weight and bias values (\(\theta\)).

\(X\) is an \(m \times n\) matrix. \(X^T X\) is invertible if and only if \(m \leq n\) and rank(X) = m. This method is an elegant mathematical solution, typically used when computational resources aren’t a concern or for smaller, well-behaved datasets. In ECE, direct matrix methods are common in signal processing and control for static or well-defined systems.

Batched Data

Break data into smaller batches.

• We’ll use a new batch on each learning step.
• New hyperparameter batch size controls how much data is used for each learning step.

Batched Data for Training

Another important hyperparameter is batch size. While you could perform gradient descent based on your full dataset every step, it may require too much memory, and take longer to converge. To combat both, we split the data into smaller batches. On each step, we’ll use a new batch to update parameters. You can control how large these batches are. This is particularly relevant in ECE for real-time processing or embedded systems with limited memory, where processing data in chunks is necessary.

Hyperparameters We Care About

Hyperparameter Tuning Guidelines

After setting up a model, you may find you need to perform “hyperparameter tuning” to achieve better results. Different problems work well with different combinations of hyperparameter values. You’ll often need to experiment or “tune” those combinations. Here are some rough guidelines for potential problems with learning rate and batch size that might suggest increasing or decreasing their values. This tuning process is similar to empirically optimizing parameters in hardware or control systems to meet specific performance requirements.

01. Introduction to scikit-learn

scikit-Learn

Scikit-learn.org is the primary website for the scikit-learn project. Here you will find information pertaining to scikit-learn, including instructions on installation, documentation, and even the project source code.

Let’s take a few moments to look around the project website.

Exercise (10 minutes): Either navigate to scikit-learn.org on your own computer and present your computer screen to the students (recommended), or ask them to open their laptops to scikit-learn.org. Take the time to point out the following elements on the website: * The classification, regression, clustering, dimensionality reduction, model selection, and preprocessing sections on the main page. These represent core groupings of features provided by scikit-learn. * The top-page navigation with links on how to install the toolkit, documentation, and examples. * The banner on the upper right corner that says “Fork me on GitHub.” This leads to the source code. * When you click the ‘Documentation’ drop-down in the upper navigation, it tells you the current stable version and has a link to ‘All available versions.’ Tell students to be sure to check the version of scikit-learn they’re working with once they start the lab. * The ‘Examples’ linked in the top navigation are not just API usage examples; they also contain some interesting machine learning insights.

Datasets

Scikit-learn comes with support for acquiring and generating datasets. The library even comes packaged with some datasets that are commonly used for exploring new models. Let’s look at some of the ways you can acquire data with scikit-learn: loading built-in datasets, fetching external ones, and generating synthetic data. This versatility makes it ideal for testing ECE algorithms against various data contexts without needing complex data acquisition setups initially.

Loading

Scikit-learn has a few datasets that are installed alongside the library. To access these datasets, you can rely on load functions like the load_iris function shown in this example. These are small, classic datasets ideal for quick demos and code verification.

Fetching

Some common datasets aren’t installed alongside scikit-learn, but the library does know how to access them. For these datasets, we use fetch functions, which pull the dataset down from the internet if necessary. This allows access to larger, more realistic datasets relevant to diverse engineering problems.

Generating

Finally, sometimes it makes more sense to generate a dataset from scratch. This is particularly useful in ECE for simulating system responses, creating test cases for control algorithms, or understanding model behavior under idealized conditions. For this, we can use one of the many generator functions provided by scikit-learn, such as make_regression for linear regression tasks.

Bunches

Bunch objects are scikit-learn objects that are often used to store datasets. If you find yourself using a load or fetch method, you’ll likely encounter a Bunch object. The lab for this lesson provides more details on Bunch objects and explores the data stored within them. You’ll encounter data that is composed of named features, as well as target values paired with sets of features.

For the most part, we will convert scikit-learn Bunch objects into Pandas DataFrame objects or TensorFlow dataset objects. The aforementioned objects are more easily integrated with the methods and frameworks we will cover in this course.

Estimators

Most of the models in scikit-learn are considered estimators. An estimator is expected to implement two methods: fit and predict.

• fit is used to train the model. At a minimum, it is passed the feature data used to train the model. In supervised models, it is also passed the target data.
• predict is used to get predictions from the model. This method is passed features and returns target predictions.

This consistent API design simplifies the process of swapping out different models for a given ECE problem, allowing engineers to quickly experiment with various ML approaches.

Transformers

In practice, it is rare that you will get perfectly clean data that is ready to feed into your model for training. Most of the time you will need to perform some type of cleaning or preprocessing on the data first. This is especially true for ECE datasets, which might come from various sensors requiring normalization, filtering, or feature scaling.

Transformers implement fit and transform methods. The fit method calculates parameters necessary to perform the data transformation (e.g., min/max values for scaling). transform actually applies the transformation. There is a convenience fit_transform method that performs both fitting and transformation in one method call.

Pipelines

It isn’t a coincidence that transformers have fit and transform methods and that models have fit methods. The common interface across classes allows scikit-learn to create pipelines for data processing and model building.

A pipeline is simply a series of transformers, often with an estimator at the end. This allows for a streamlined, organized workflow, ensuring consistent data preprocessing steps are applied before training and prediction, which is critical for reproducible results in ECE experiments.

Metrics

Scikit-learn also comes with many functions for measuring model performance, located in the metrics package.

In this case, we are calculating the mean squared error. (In the Introduction to Regression lesson, you saw L1, L2, and MSE). These metrics are crucial for quantifying the success of an ML model in ECE applications, e.g., evaluating the accuracy of a power prediction model or the precision of a control algorithm.

Your Turn

It may be helpful to scroll through the lab associated with this unit and point out to the students the key ideas we covered here. Now, let’s apply these scikit-learn concepts in a practical lab.

02. Linear Regression with scikit-learn

We’ve learned about linear regression, and we’ve learned about scikit-learn. In this unit we’re going to perform a linear regression using the scikit-learn toolkit, focusing on practical implementation for ECE problems.

Linear Regression

Linear Regression Fit Animation

Remember that linear regression involves fitting a straight line to a dataset. Most of the time, the line doesn’t fit perfectly for all data points. You can see in this illustration, the blue data points, the regression line, and then the red lines between the data points and the regression line. The red lines indicate the “error.” There are many ways to measure this error that we’ll talk about in detail in a future unit. This GIF helps visualize the goal of minimizing these errors.

scikit-learn: Using Linear Algebra

Note

This is not a learning algorithm that iteratively optimizes. It’s a direct, closed-form solution.

To perform linear regression in scikit-learn without iterative learning, we use the LinearRegression class from the linear_model package. As you can see in this example, performing the regression is as simple as instantiating the class and then calling the fit() method. The model then directly calculates the coefficient and intercept for the linear equation using the closed-form solution we discussed earlier. This is efficient for smaller datasets or when GDRegressor convergence is not desired / needed.

Optimizers

1. (Batch) Gradient Descent
2. Stochastic Gradient Descent (SGD)
3. Mini-Batch Gradient Descent

Recall that our overall goal is to learn parameters that minimize a particular cost/loss function. There are many ways to perform this optimization, but gradient descent is a very popular choice. At a high level, we use the gradient (i.e. the derivative/slope) of the cost function to determine the direction to adjust the parameters. In other words, if we want to get to the bottom of the hill, we walk in the direction of the steepest downward slope.

• Batch Gradient Descent: The entire dataset is used to calculate the gradient during each iteration of training. This can be computationally expensive for large datasets.
• Stochastic Gradient Descent (SGD): We randomly choose one data point from our training set to compute the gradient at each iteration (i.e. we use a batch-size of 1). This introduces more noise but can be much faster per iteration.
• Mini-Batch Gradient Descent: A middle ground between batch and stochastic gradient descent, using a fixed number of training samples (greater than 1, but less than the entire dataset) to compute the gradient during each iteration. This combines the speed of SGD with some stability.

These various methods mirror iterative optimization techniques used in ECE for adaptive systems or circuit design (e.g., tuning filters).

scikit-learn: Stochastic Gradient Descent

Using stochastic gradient descent looks strikingly similar to using LinearRegression. This is no accident. Scikit-learn’s API is very consistent, simplifying learning and application.

In this example, we load the data into memory, perform SGD (fit method), and then print out the coefficient and intercept.

Note that this might not be the absolute optimal coefficient and intercept; it’s the best one that the SGD algorithm found after running through its epochs and iterations, within its convergence criteria. This is a common characteristic of iterative optimization in ECE.

scikit-learn: SGD Hyperparameters

There aren’t really any hyperparameters to tune for LinearRegression (the closed-form solution). SGDRegressor, however, has many hyperparameters that can be tuned. You can see some of those hyperparameters in use here:

• max_iter: The maximum number of passes over the training data (epochs). Sometimes training more can improve performance.
• n_iter_no_change: Manages “early stopping.” If loss doesn’t meaningfully improve for this many iterations, training stops.
• tol: The tolerance for loss improvement to trigger early stopping.
• learning_rate: Affects how much the weights are adjusted during each step. ‘adaptive’ means it adjusts dynamically. Different learning rate schedules are crucial in ECE control systems for stability or convergence speed.

There are many more hyperparameters that can be found in the SGDRegressor documentation, allowing for fine-grained control over the optimization process, similar to tuning intricate analog circuits.

scikit-learn: SGD partial_fit

from sklearn.linear_model import SGDRegressor

sgd_reg = SGDRegressor()
sgd_reg.partial_fit(X_1, y_1)
sgd_reg.partial_fit(X_2, y_2)
...
sgd_reg.partial_fit(X_n, y_n)
sgd_reg.coef_, sgd_reg.intercept_

Another capability of the SGDRegressor is the ability to partially train the model. This can be useful if your data doesn’t fit into memory, or if you’re working with streaming data. You can continually call partial_fit with subsets of the full dataset.

This is highly relevant in ECE for embedded systems, real-time control, or sensor networks where data arrives continuously and cannot be stored entirely. partial_fit allows the model to learn incrementally.

Loss

Mean Squared Error

\[ MSE = \frac{1}{n} \sum_{n=1}^{n}(y_{i} - \hat{y_{i}})^{2} \]

We’ll get into loss and different ways to measure it in later units. For this unit we’ll calculate loss using the mean squared error. The mean squared error is the measure of the values that our model predicts vs. what the values actually are. The differences are calculated, squared to get rid of negatives, and summed so that the average squared error can be found. This measure is crucial for quantifying the performance of ECE systems, from signal reconstruction errors to predictive modeling accuracy.

Train/Validate, Test

Train, Validate, Test Split

This lab will also be the first time we’ll need to split our data for model training.

When we train a model, we could use all of the data that we have. When we do that, however, we risk overfitting the model to our data, and we lose the ability to test our model on “new” data it hasn’t seen. The model might become really good at making predictions that look like the data that it has already seen, but really bad at generalizing.

For this reason we typically hold out some of the data and don’t use it to train the model at all. We keep this “test set” of data and use it only to evaluate the model after training has completed. We pass the trained models the features in the test set, get the predictions from the model, and then calculate the difference between the predictions and the actual values.

The concept of a validation set is also important. The validation set is used during training to let the optimizer evaluate the model. The loss calculated with the validation set directly affects decisions the model makes. This practice is akin to rigorous testing of an ECE prototype against a test bench and then against real-world scenarios before deployment.

Train/Validate, Test, Validate

Double Validation Process

The holdout data story gets more complicated when hyperparameter tuning is involved. When you tune hyperparameters, you’ll still have the same training and validating data available during model fitting. Then you’ll use your test data to see how well the model generalizes. With that said, if you then change hyperparameters and test again, you risk over-tuning hyperparameters to the test data set.

In order to prevent this, many data scientists also keep another holdout dataset called the validation dataset (often called a “holdout validation set” or “dev set” to distinguish from the training-time validation set). This dataset is used for one final check after you have selected hyperparameters and before final deployment. This multi-layered validation ensures the model’s robustness, similar to multiple stages of verification and validation in ECE product lifecycle management.

Your Turn

Let’s now build a few different linear regression models using scikit-learn in our lab exercise. You’ll apply these concepts to practical ECE datasets.

03. Regression Quality

So far in this course, we have spent some time building and testing regression models. But how can we measure how good these models are? In this unit we will examine a few of the ways we can measure and graph the results of a regression model in order to better understand the quality of the model, which is paramount in ECE for ensuring system reliability and performance.

Coefficient of Determination (\(R^2\))

\[ SS_{res} = \sum_{i}(y_i - \hat{y_i})^2 \] \[ \bar{y} = \frac{1}{n}\sum_{i=1}^{n}y_{i} \] \[ SS_{tot} = \sum_{i}(y_{i}-\bar{y})^2 \] \[ R^{2} = 1 - \frac{SS_{res}}{SS_{tot}} \]

The coefficient of determination, denoted \(R^2\), is one of the most important metrics in regression. It tells us how much of the data is “explained” by the model. This is very useful in ECE to understand how well a model captures the underlying physics or behavior of an electronic component or system.

Before we can define the metric itself, we need to define a few other key terms: * Residual sum of squares (\(SS_{res}\)): The summation of the square of every difference between the target value \(y_i\) and the predicted value \(\hat{y_i}\). * Total sum of squares (\(SS_{tot}\)): The sum of the squared differences between each value \(y_i\) and their mean \(\bar{y}\).

Given these, we can calculate \(R^2\). The \(R^2\) score measures how well the actual variance from \(x\)-values to \(y\)-values is represented in the variance between the \(x\)-values and the predicted \(\hat{y}\)-values.

Typically, this score ranges from 0 to 1, where 0 is bad and 1 is a perfect mapping. However, the score can also be negative. This happens if a simple horizontal line through the mean \(\bar{y}\) performs better than your regression, meaning your model is truly terrible and needs serious revision. For values in the range 0-1, interpreting the \(R^2\) is more subjective. The closer to 0, the worse your model is at fitting the data. And generally, the closer to 1, the better. But you also don’t want to overfit, which we’ll discuss later.

Mean Squared Error (MSE)

\[ MSE = \frac{1}{n} \sum_{n=1}^{n}(y_{i} - \hat{y_{i}})^{2} \]

The mean squared error is the measure of the values that our model predicts vs. what the values actually are. The differences are calculated, squared to get rid of negatives, and summed so that the average squared error can be found.

What is a good MSE value? The answer really depends how perfect you want your model to be. The values of MSE are a little difficult to interpret, though. Since they are the square of the error, the units don’t match the units of the target in our model (e.g., if predicting voltage in volts, MSE is in volts-squared). This is fine for machines training the model, but it’s nearly impossible for an engineer to intuitively reason about afterward.

There is a solution, though.

Root Mean Squared Error (RMSE)

\[ RMSE = \sqrt{\frac{1}{n} \sum_{n=1}^{n}(y_{i} - \hat{y_{i}})^{2}} \]

Note

RMSE returns the error in the original units of the target variable for easier interpretation.

The root mean squared error is simply the square root of the mean squared error. It adjusts the units of the error back to the units of the target, which makes model quality much easier to reason about directly.

For instance, if you have a model that predicted the output voltage of a power supply, and it had an RMSE of 0.1V, that would tell an ECE engineer that, on average, the predictions are off by 0.1 volts. If the target voltage was 5V, this error might be acceptable, but if it was 0.5V, it might be too high. This direct interpretability is why RMSE is often preferred when discussing model accuracy in engineering contexts.

Mean Absolute Error (MAE)

\[ MAE = \frac{1}{n} \sum_{n=1}^{n}(|y_{i} - \hat{y_{i}}|) \]

Mean absolute error is calculated similarly to mean squared error. Instead of using squaring to remove negative values, the absolute value of the difference in actual and predicted values is taken.

A benefit of MAE is that the units remain the same as that of the target, similar to RMSE.

The primary difference in MAE and MSE/RMSE is that MSE/RMSE squares the error, which gives larger differences in value and much higher error scores, effectively giving extra penalty to really bad predictions (outliers). MAE treats all errors linearly. In ECE, choosing between MAE and RMSE often depends on whether large errors should be heavily penalized (RMSE) or if all errors are considered equally important magnitude-wise (MAE).

Predicted vs. Actual Plots

Good Predicted vs Actual Plot

There are numerous ways to visualize regression predictions, but one of the most basic and intuitive is the “predicted vs. actual” plot. Here, we plot the actual target values on one axis and the model’s predicted values for those targets on the other. A perfect model would have all points lying directly on the y=x line.

In this case, the data points scatter pretty evenly around the prediction-to-actual line (i.e., the line y=x, where the actual and prediction are equal). This indicates a good fit without significant bias.

So what does a bad plot look like?

Predicted vs. Actual Plots (Positive Bias)

Predicted vs Actual Plot with Positive Bias

Now we have a situation where there is an obvious bias. All predictions are consistently higher than the actual values (lying below the y=x line, meaning Predicted > Actual). This suggests the model needs to be adjusted to make smaller predictions across the board. Such biases are critical to identify in ECE, as they could indicate systematic errors in sensor calibration or system modeling.

Residual Plots

Good Residual Plot

Another helpful visualization tool is plotting the regression residuals. As a reminder, residuals are the difference between the actual values and the predicted values (y - y_pred).

We plot residuals on the y-axis against the predicted values on the x-axis and draw a horizontal line through y=0. Ideally, residuals should be randomly scattered around zero, indicating no systematic pattern in the errors, and that the model has captured the underlying relationship well.

Residual Plots (Bias Example)

Residual Plot with Bias

Cases where our predictions were too low appear above the zero line (positive residuals). Cases where our predictions were too high appear below the zero line (negative residuals).

In the “predicted vs. actual” section above, we plotted a case where there was a large positive bias in our predictions (model consistently overpredicts). Plotting the same biased data on a residual plot shows all of the residuals falling below the zero line. This clear pattern in residuals indicates a systemic issue with the model, which an ECE engineer would seek to diagnose and correct.

Your Turn

Let’s now move on to the lab portion of the unit. In this lab, you’ll create and interpret various measures and visualizations of regression model quality. This is a practical skill for any ECE professional working with data-driven systems.

04. Polynomial Regression & Overfitting

Outline

1. Recall Linear Regression
2. Polynomial Regression: What is it and how is it different (or not so different)?
3. Caution! Dangers of polynomial regression!

Tip

Understanding when and how to use polynomial regression is key to modeling complex system behaviors in ECE.

Linear Regression Fit Animation

We have seen linear regression and have implemented it using scikit-learn. We found the line of best fit using optimizers such as gradient descent. Though linear regression can be a powerful predictive tool, it isn’t appropriate for all types of regression problems, especially when the underlying relationship isn’t truly linear, as often happens with non-linear components or systems in ECE.

Dataset for Polynomial Regression

Take a look at this dataset for a few seconds. See if you can find a good spot to place a straight line. Does a linear relationship seem to adequately describe these points? This kind of data might arise from, for example, the efficiency curve of a power amplifier.

Linear Fit on Non-linear Data

This is the line that the closed-form solution of linear regression would create. It doesn’t look like a very good fit, does it? The R-squared score for this line is actually 0.228, indicating a poor explanation of variance. Clearly, a straight line is insufficient here.

Polynomial Fit on Non-linear Data

We can see that if we introduce a polynomial line (in this case, a 2nd degree polynomial, a quadratic curve), we get a much better looking fit. The R-squared score is now 0.790, which is significantly better. This demonstrates how polynomial regression can capture non-linear relationships, commonly encountered in the real-world performance characteristics of ECE devices.

Polynomial Equations

Examples of Polynomial Equations

Here are a few examples of polynomial equations. The topmost is the linear equation we are used to (degree 1). The next is commonly called a quadratic equation (degree 2). The third is a cubic equation (degree 3). The number of factors (or the ‘degree’ of the polynomial) you can add to the equation is theoretically unbounded, though you’ll pay a computational expense for polynomials of higher degree and also increase the risk of overfitting, which we’ll discuss soon.

What is the Original Curve?

Exercise (3 minutes): Ask students to turn to a partner next to them and discuss the two questions on the slide: “What kind of curve do you think generated this data?” and “Did the polynomial regression model find the ‘true’ curve?”

Original Polynomial Curve

This dataset was actually randomly generated based on a polynomial equation with some random noise. You can see the original polynomial line drawn in green on the chart. Our predicted line in red isn’t perfect by any means. You can see a much sharper slope at both ends, but it is still much better than a straight-line fit. This highlights that models aim to approximate, not necessarily perfectly replicate, the true underlying function.

Polynomial Regression Process

If we understand the process behind polynomial regression, we can see that it is not so different from linear regression. Ultimately, in order to reduce error in our model, we are still attempting to find the most accurate weights and biases we can find. The underlying optimization principles remain the same; we are just working with a richer set of features.

Polynomial Regression = Linear Regression

Turn the original polynomial regression problem into a polynomial regression problem with multiple features.

To find the weights and biases for a polynomial regression model, we recast the problem as a multivariate linear regression problem. This is a key insight: We are not changing the linear regression algorithm itself. Instead, we are changing the input features.

For example, if we want a order-2 polynomial \(y = w_0 + w_1 x + w_2 x^2\), we take our original feature x and create new features: x_1 = x and x_2 = x^2. Then, the problem becomes a linear regression y = w_0 + w_1 x_1 + w_2 x_2. The machine learning algorithm still learns linear weights for these new, transformed features. This technique of feature engineering is very important in ECE for transforming raw sensor data into more meaningful inputs for ML models.

Overfitting

We will now explore the potential dangers of using higher-order polynomial fits. While they can capture non-linear relationships, they pose a significant risk of ‘overfitting’ the data.

Dataset for Overfitting Example

What could possibly go wrong if we use a polynomial model to fit the following data? Brainstorm some ideas. Consider how the curve behaves between the observed data points, and imagine how it might generalize to new, unseen data points. This is a common pitfall in ECE when modeling complex systems with too many parameters.

Overfitting Demonstration

Imagine we introduce one more data point to the right (in a region where curve B does not pass through, perhaps above the second valley). It is clear that curve B is not able to generalize to new data points. It fits the existing points almost perfectly, but its behavior in between and outside these points is erratic. This is a classic example of overfitting.

Overfitting Analogy: Clothing Fit

Well-fitting shirt

Let’s think of overfitting by looking into clothing. Here, we have a person wearing a reasonably well-fitting shirt. It’s comfortable, covers well, and fits a typical body shape, allowing for some variation without being loose. This is like a well-generalized model.

Overfitted clothing

Clothing can be a little more form-fitting, or even custom-tailored. This reduces the number of people that it will fit perfectly. This is an analogy for overfitting. You can think of it like a custom-tailored suit of armor that fits only one person perfectly; it’s extremely specific to the current data but fails to generalize.

Underfitted clothing

There is the other extreme, where you make clothing so loose that just about anyone could wear it. This would be underfitting. The model is too simple to capture the underlying patterns, like a garment that’s too generic to even properly fit its intended wearer.

Just right fit

Most of the time, you probably just want a simple, midsized, unisex t-shirt of a given style. It’s not perfectly tailored but fits a large range of people reasonably well. This represents a model that generalizes well: it captures the essential patterns without being overly specific to the training data.

Illustration of Overfitting Regression

So how does this apply to real data? Here is an illustration of overfitting a regression model. You can see how the regression line perfectly fits almost every data point on the training data, but notice its wild oscillations between points and at the edges. This model will likely have very poor predictive performance for new, unseen data points, even if they closely resemble the training distribution.

How Do We Avoid This?

Given the problem of a polynomial fitting data too closely (overfitting), how would you avoid it? Give students some time to throw out some ideas. This is a crucial design challenge in ECE when trying to build robust and generalizable ML systems for signal processing, control, or embedded intelligence.

Avoiding Overfitting

• Simpler polynomial
• More training data
• Dropping out some training data (e.g., regularization)
• Overfitting penalties (regularization)

Important

Regularization is a key technique to manage overfitting by adding penalties to model complexity.

Here are some of the most common ways to avoid overfitting:

• Using a polynomial function with fewer degrees: If your model is introducing enough curvature to cross most training data points, then use fewer degrees in your polynomial. This reduces model complexity.
• More training data: As your dataset grows in size, it will likely also grow in diversity and create a model that is less overfitted because it sees more variations. This can be challenging for ECE systems where data collection is expensive.
• Dropping out some training data (or features): Sometimes removing less informative features or artificially simplifying the input can help.
• Overfitting penalties (regularization): There are strategies for adding penalties to the model that make even a high-degree polynomial less likely to overfit. Some common strategies are called Lasso, Ridge, and ElasticNet. We’ll look at each of these more closely, and you’ll experiment with them in your lab exercise. Regularization can be seen as introducing a “cost” for excessive model complexity.

Regularization

Regularization is a method of “shrinking” the coefficients (weights) in the learned equation. This process discourages overly complex models that fit the noise in the training data rather than the underlying signal. There are many types of regularizers, but we will look at the most common ones here: Lasso, Ridge, and ElasticNet. These methods are indispensable in ECE for developing robust models, especially in areas like predictive maintenance or system fault diagnosis where generalization is critical.

Recall: Mean Squared Error

Mean Squared Error Formula Breakdown

As a reminder, this is the equation of a common loss function, the mean squared error. Let’s break down its components in the context of our linear algebra notation:

• Line 1: (y_true - y_pred) is the error between the true target and the predicted target from the model.
• Line 2: Recall that y_pred came from a linear regression equation, which can be written in matrix notation as X * theta.
• Line 3: Completing the matrix multiplication and writing the multivariate regression formula using detailed notation:
  • n = number of rows in the training data
  • p = number of coefficients in the equation (also number of features + bias term)
  • y_true = true target value
  • theta_0 = intercept (bias)
  • theta_j = coefficients (weights) of the polynomial equation
  • x_i = feature values

This foundation helps us understand how regularization will modify this loss function.

Lasso (L1) Regularization

Lasso (L1) Regularization Formula

What does it mean to shrink coefficients? It effectively means to increase the value of the loss function as the coefficients get larger.

Lasso is L1 regression. This means that it adds the sum of the absolute values of the coefficients (multiplied by a regularization parameter \(\lambda\)) to the original MSE loss function. We can see that by adding lambda * |theta_j| (a positive number), the cost function is always slightly larger than the regular MSE loss function for any non-zero theta_j. This regularization term forces the model to make theta_j values smaller.

Smaller coefficients make the model “more linear” or less sensitive to specific features. For example, in a high-degree polynomial, if theta_2 and theta_3 become very small, their terms have less impact on the curve.

LAGSO is an acronym for “Least Absolute Shrinkage and Selection Operator.”

Due to the absolute value function, L1 regularization has a unique property: it can drive some coefficients exactly to zero. This means L1 regularization can perform feature selection, effectively identifying and ignoring less important features. This is highly useful in ECE for reducing model complexity and identifying critical system parameters.

Ridge (L2) Regularization

Ridge (L2) Regularization Formula

Ridge regularization looks similar to Lasso, but instead of appending the sum of absolute values of coefficients to the loss function, it appends the sum of their squares. This is also multiplied by a regularization parameter \(\lambda\).

Ridge regularization also shrinks coefficients towards zero, but it tends to shrink them proportionally. Unlike Lasso, L2 regularization rarely drives coefficients exactly to zero; it prefers to reduce their magnitude without fully eliminating them.

Since the L2 norm (sum of squares) is differentiable, problems using this method can be solved efficiently by gradient descent, making it computationally attractive. In ECE, Ridge regression helps stabilize models in the presence of multicollinearity (highly correlated features), which is common in sensor data fusion.

ElasticNet (L1 + L2)

ElasticNet is a regularization technique that combines both L1 (Lasso) and L2 (Ridge) penalties. It includes both the sum of the absolute values of coefficients and the sum of their squares in the loss function.

This combination allows ElasticNet to benefit from both methods: it can perform feature selection (like Lasso) by shrinking some coefficients to zero, while also handling groups of correlated features gracefully (like Ridge). This makes ElasticNet a robust choice in many real-world ECE applications where datasets might have many features, some of which are correlated, and model sparsity is desired.

Which Regularization Is Best?

Caution

The choice of regularization often depends on the specific dataset and problem. Experimentation is key!

It depends on the specific characteristics of your data and your modeling goals:

• L1 regularization (Lasso):
  • Can drive coefficients to zero, and it tends to produce a sparse model (i.e., many coefficients are zero).
  • Its unique ability to eliminate irrelevant features makes it useful for feature selection.
• L2 regularization (Ridge):
  • Is less likely to drive coefficients exactly to zero; it shrinks them proportionally.
  • Instead, it tends to produce a more dense model (most coefficients are small but non-zero).
  • Effective at handling multicollinearity (highly correlated features).
• ElasticNet:
  • As a compromise between L1 and L2, it often works well in situations where there are multiple correlated features or when feature selection is important.

It is probably worth experimenting with each method to see which works best for your particular model, considering performance, interpretability, and computational resources, especially in diverse ECE problem domains.

Your Turn

Now that we’ve covered polynomial regression and regularization techniques, it’s time to apply these concepts in a hands-on lab. You’ll experiment with fitting complex curves and battling overfitting, essential skills for robust system modeling in ECE.