Machine Learning

04 Classification: Introduction to Classification

Imron Rosyadi

00. Introduction to Classification

Welcome to this lecture on Classification for Electrical and Computer Engineering undergraduates. This topic is fundamental to many real-world engineering applications, especially in areas like signal processing, image recognition, control systems, and embedded machine learning. Today, we’ll dive into what classification is, how it differs from regression, common models, and crucial evaluation metrics.

Classification vs. Regression: A Quick Review

Regression: Predicting Continuous Values

• Predicts a numeric, continuous output.
• Examples: House prices, temperature, signal strength.
• Evaluation: Measures like Mean Squared Error (MSE).

Recall that regression attempts to predict a continuous value. In this illustration, you can see a linear regression fitting a line to a dataset. We judge the quality of our regression by measuring the distance of the actual data from our prediction line. Metrics like Mean Squared Error and Root Mean Squared Error are common. In ECE, regression might be used for predicting sensor readings or component lifetimes.

Classification: Predicting Categories

• Predicts a categorical, discrete output.
• Examples: Spam/Not Spam, Object presence (cat/dog), Fault detection.
• Evaluation: Focuses on correct vs. incorrect assignments.

Classification models don’t predict a continuous value, but instead attempt to predict the “class” or category of a data point. Classification algorithms can distinguish between two states (binary classification), like spam or not. They can also determine the probability that a data point belongs to one of many classes (multi-class classification), such as identifying different types of modulation in a communication signal. The illustration shows separating red and blue data points using a clear boundary. In ECE, this could be identifying a defect on a PCB, classifying speech commands, or detecting anomalies in sensor data.

What Does It Mean to Classify?

Classification model results are often returned as a list of confidences for each class. The model predicts the probability a data point belongs to each class.

Understanding Classification Confidence

Example Output:

• Tiger: 0.96
• Lion: 0.75
• Cougar: 0.68

Tip

In ECE, such confidence levels are critical in systems like autonomous vehicles (identifying pedestrians with high certainty), medical image diagnosis, or anomaly detection in power grids.

Classification models often return a list of confidences, indicating the predicted probability that a given data point belongs to each possible class. For example, when analyzing an image, the model might assign a 98% confidence to “tiger,” 1% to “lion,” and 1% to “cheetah.” These confidences are not always true probabilities without further transformation, like applying a softmax function. Your code then needs to interpret these predictions to make a final decision, perhaps by selecting the class with the highest confidence or applying a specific threshold.

Ambiguous Cases

• Orange: 0.97
• Grapefruit: 0.96
• Sun: 0.45

Here’s an example of a model returning confusing predictions for an image. The model is highly confident that the image contains either an orange or a grapefruit, with very similar confidence scores. What would you do in this situation? This highlights that sometimes ML models provide ambiguous results, and the system or human operator needs to handle such cases. For instance, in an automated fruit sorting system, this might lead to a re-scan or manual inspection.

Common Classification Models

• Logistic Regression:
  • A variation of linear regression, uses a sigmoid function for binary outcomes. Simple and interpretable.
• Nearest Neighbors:
  • Classifies based on the majority class among its closest data points. Intuitive, but sensitive to local data structure.
• Decision Trees:
  • Tree-like structure where each node tests a feature, leading to a class decision. Good for interpretability.
• Random Forests:
  • An ensemble of many decision trees, combining their predictions for robustness and better accuracy. Often very powerful.
• Naive Bayes:
  • Based on Bayes’ theorem, assumes feature independence. Useful for text classification and spam detection.
• Deep Learning (Neural Networks):
  • Multi-layered networks capable of learning complex patterns. Highly effective for image, speech, and sensor data classification.

There are numerous models that can be used for classification, each with different underlying mathematical principles and strengths. Some of the more common ones we’ll encounter are listed here. In ECE, the choice of model often depends on the type of data, computational constraints (especially for embedded systems), and real-time requirements. For example, simpler models might be preferred for low-power edge devices, while deep learning excels in complex image or signal processing tasks.

The Machine Learning Classification Workflow

graph TD
    A["Raw Sensor/System Data"] --> B{"Data Preprocessing"}
    B --> C["Feature Engineering/Extraction"]
    C --> D{"Split Data <br> (Training & Testing Sets)"}
    D -- Training Data --> E[Choose & Train ML Model]
    E -- & Evaluation --> F[Model Evaluation & Tuning]
    D -- Testing Data --> F
    F -- Performance OK? --> G[Deploy Model to ECE System]
    F -- Needs Improvement --> E
    G --> H[New Live Data Input]
    H --> I["Real-time Prediction / Classification"]
    I --> J["Action/Decision <br> (e.g., Control Signal, Alert)"]

    style A fill:#f9f,stroke:#333,stroke-width:2px;
    style G fill:#bbf,stroke:#333,stroke-width:2px;
    style J fill:#fcf,stroke:#333,stroke-width:2px;
    style E fill:#ccf,stroke:#333,stroke-width:2px;
    style F fill:#dfd,stroke:#333,stroke-width:2px;

This flowchart illustrates a typical machine learning classification workflow. Notice the continuous loop for model training and evaluation, highlighting that model development is an iterative process. For ECE applications, “Raw Sensor/System Data” could be anything from acceleration data in a drone to voltage readings in a power grid. “Feature Engineering” could involve extracting frequency components, statistical moments, or energy levels from time-series signals. “Deploy Model to ECE System” represents integrating the trained model into hardware, software, or an embedded system, where it performs “Real-time Prediction” and drives “Actions,” such as adjusting a motor’s speed or triggering a safety shutdown.

Classification Model Performance

Unlike regression, we can’t measure continuous “distance” to evaluate classification. Instead, we count correct vs. incorrect predictions. These counts form the basis for various performance metrics.

As mentioned, determining the performance of a regression model involves measuring the distance between continuous values. In classification, we’re dealing with discrete categories, so we can’t use the same “distance” concept. Instead, we focus on how many predictions the model got right versus wrong. These counts allow us to define metrics that robustly evaluate model quality. Understanding these metrics is crucial in ECE, especially when a false positive or false negative can have significant consequences.

The Confusion Matrix

• True Positive (TP): Model predicted positive, was actually positive.
• False Positive (FP): Model predicted positive, was actually negative (Type I error).
• False Negative (FN): Model predicted negative, was actually positive (Type II error).
• True Negative (TN): Model predicted negative, was actually negative.

Most of the performance measures we’ll discuss are derived from the confusion matrix. For simplicity, we often analyze binary classification or evaluate a single class as “positive” against all others. For instance, in a system detecting defects (positive), a True Positive means the system correctly identified a defect. A False Positive means it flagged a non-defect as a defect (a false alarm). A False Negative means it missed a defect (a critical failure). A True Negative means it correctly identified a non-defect. The implications of FP vs. FN can vary greatly across ECE applications; e.g., in medical diagnosis, FN is often more critical than FP, while in spam detection, FP might be more annoying than FN.

Accuracy

• The fraction of all predictions that a classification model got right.
• Simply the sum of True Positives and True Negatives, divided by the total.

\[ \text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} \]

Accuracy seems like a straightforward measure, but it can be misleading, especially with imbalanced datasets. Imagine a system trying to detect a rare fault in an industrial machine that occurs only 1% of the time. If our model simply predicts “no fault” all the time, it would achieve 99% accuracy, but it would completely fail to detect any actual faults. In ECE, relying solely on accuracy for rare event detection, like critical system failures or cybersecurity intrusions, can be very dangerous. This is why we need more nuanced metrics.

Motivation for Precision

When the model predicted positive, how often was it correct?

What is the probability that a detected anomaly in our sensor data is an actual* anomaly, given that our model flagged it?*

In many ECE scenarios, precision is crucial. For example, in a robotic arm that picks up defective items, high precision means fewer good items are accidentally discarded (fewer false positives). While accuracy might be high, if the FP rate is unacceptably high, it leads to waste or unnecessary actions. Precision directly answers the question: “Of all the times the model said ‘yes,’ how many times was it actually ‘yes’?”

Precision

• The fraction of correct positive predictions out of all positive predictions made by the model.

\[ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} \]

Precision focuses on the accuracy of positive predictions. A high precision indicates a low rate of false positives. In ECE, high precision is desirable in applications where false alarms are costly or disruptive. Consider a system for chip manufacturing QA: discarding a good chip because it was misclassified as defective is costly. Here, precision helps minimize such errors. Precision of 1 means every time the model said “positive,” it was truly positive, but it doesn’t tell us if it missed any actual positives.

Motivation for Recall

Out of all the actual positive cases, how many did the model correctly identify?

What is the probability that our model will detect a ‘critical’ electromagnetic interference event, given that it actually occurred?

Recall addresses a different but equally important question: “Of all the actual ‘yes’ cases, how many did the model find?” In safety-critical ECE applications, missing an actual positive (a false negative) can be devastating. This is where recall shines.

Recall

• The fraction of correct positive predictions out of all actual positive cases.

\[ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} \]

Recall, also known as sensitivity or true positive rate, is about how complete the model’s positive detections are. A high recall means a low rate of false negatives. In ECE, high recall is crucial for applications where missing a positive event is more detrimental than a false alarm. Examples include detecting equipment failures, security breaches, or health monitoring systems where failing to detect a critical condition is unacceptable. A recall of 1 means the model caught every single positive instance, but it doesn’t say anything about how many false positives it had along the way.

Precision vs. Recall: A Trade-Off

• Increasing one often decreases the other.
• The optimal balance depends on the application’s cost of FP vs. FN.

Balancing precision and recall is often a “tug-of-war.” If you want to increase recall (catch more positives), you might lower your decision threshold, potentially increasing false positives and thus lowering precision. Conversely, if you want to increase precision (be more certain of your positives), you’d raise the threshold, which might cause you to miss some actual positives, lowering recall. Finding the optimal point where these two metrics are acceptable for your specific ECE application is critical. For instance, in a fire detection system, high recall is paramount, even if it means some false alarms (lower precision).

F1 Score

• The harmonic mean of precision and recall.
• High F1 indicates both precision and recall are high.

\[ F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \]

The F1 score provides a single metric that balances both precision and recall. It’s particularly useful when you have an uneven class distribution, as it penalizes models that favor one metric over the other. The harmonic mean gives more weight to lower values, meaning a model needs both good precision and good recall to achieve a high F1 score. It’s a common metric when you need a general measure of a model’s effectiveness in ECE applications where both missing events and false alarms are important to manage.

F1 Score: Simplified

The F1 formula can be reduced to:

\[ F_1 = \frac{2 \cdot \text{TP}}{2 \cdot \text{TP} + \text{FP} + \text{FN}} \]

This simplified form of the F1 score can be directly calculated from the True Positives, False Positives, and False Negatives, making it convenient for quick computations.

Interactive Metric Calculator

Adjust the True Positives, False Positives, False Negatives, and True Negatives to see how Accuracy, Precision, Recall, and F1 Score change.

viewof tp_val = Inputs.range([0, 100], {value: 1, step: 1, label: "True Positives (TP)"});
viewof fp_val = Inputs.range([0, 100], {value: 1, step: 1, label: "False Positives (FP)"});
viewof fn_val = Inputs.range([0, 100], {value: 8, step: 1, label: "False Negatives (FN)"});
viewof tn_val = Inputs.range([0, 100], {value: 90, step: 1, label: "True Negatives (TN)"});

This interactive tool allows you to directly manipulate the components of a confusion matrix and observe their impact on the key classification metrics. By adjusting the sliders, you can simulate different model behaviors and gain an intuitive understanding of how True Positives, False Positives, False Negatives, and True Negatives affect Accuracy, Precision, Recall, and the F1 Score. This hands-on experience is crucial for ECE students to grasp the trade-offs involved in designing and evaluating ML systems.

Which Metric Do I Use?

Note

The answer is always: it depends on your specific ECE application!

• Accuracy: Rarely a sufficient standalone metric, especially with imbalanced classes.
• Precision: Crucial when False Positives are costly (e.g., discarding good products in QA, false alarms in security).
• Recall: Critical when False Negatives are costly (e.g., missing a fault in critical infrastructure, failing to detect a disease).
• F1 Score: A good general measure when you need to balance both precision and recall, particularly with imbalanced datasets.

Just remember, there’s no single “best” metric. In ECE, the choice depends heavily on the cost associated with different types of errors. In some contexts, a false positive might just be an annoyance, while in others (like a medical diagnosis or a safety system), it could lead to significant financial loss or danger. Conversely, a false negative in a critical fault detection system could lead to catastrophic equipment failure. Always consider the real-world consequences of your model’s mistakes when choosing which metrics to prioritize. Later, we’ll discuss more advanced measures and graphical tools that help provide a more complete picture.

Confusion Matrix Example

Scenario: A model predicts if a tumor is malignant.

(Positive Class: Malignant, Negative Class: Benign)

Let’s practice calculating these metrics using a concrete example. Focus on a model designed to predict if a tumor is malignant. A “positive” outcome from the model means it predicts the tumor is malignant. We will walk through the components of the confusion matrix again with this scenario in mind. Remember:

• True Positive: Model said malignant, and it was malignant.
• True Negative: Model said benign, and it was benign.
• False Positive: Model said malignant, but it was benign (Type I error - false alarm, patient undergoes unnecessary procedure).
• False Negative: Model said benign, but it was malignant (Type II error - dangerous miss, patient doesn’t get treatment).

Confusion Matrix Example: Data

Model to predict if a tumor is malignant

Given these values:

• TP = 1
• FP = 1
• FN = 8
• TN = 90

Solution: Accuracy

\[\text{Accuracy} = \frac{1 + 90}{1 + 1 + 8 + 90} = \frac{91}{100} = 0.91\]

As you can see, the accuracy is 91%. This looks quite good on its own. However, let’s see how the other metrics inform our understanding of this model. Remember our earlier discussion about accuracy with imbalanced datasets and the potential for false confidence.

Solution: Precision

\[\text{Precision} = \frac{1}{1 + 1} = \frac{1}{2} = 0.50\]

The precision is 0.50, or 50%. This means that when the model predicts a tumor is malignant, it’s only correct half the time. While the overall accuracy was high, the model frequently gives false alarms for malignancy. This would lead to many patients undergoing potentially stressful and invasive follow-up tests unnecessarily.

Solution: Recall

\[\text{Recall} = \frac{1}{1 + 8} = \frac{1}{9} \approx 0.11\]

The recall is approximately 0.11, or 11%. This is a very low recall score. It means the model only correctly identifies about 11% of the actual malignant tumors. In this medical context, missing 89% of malignant tumors (false negatives) is a severe problem. This highlights a critical failure of the model despite its seemingly high accuracy. Without surgical intervention, the consequences of a malignant tumor would be dire.

Solution: F1 Score

\[ F_1 = \frac{2 \cdot 0.50 \cdot 0.11}{0.50 + 0.11} = \frac{0.11}{0.61} \approx 0.18 \]

Or, using the simplified formula:

\[ F_1 = \frac{2 \cdot 1}{2 \cdot 1 + 1 + 8} = \frac{2}{11} \approx 0.18 \]

Solution: F1 Score

The F1 score is very low, approximately 0.18. This low F1 score correctly reflects the poor performance of the model by penalizing the low recall, despite the relatively high precision (0.50) and high accuracy (0.91). This example vividly demonstrates why looking at all metrics, especially F1, is crucial and why relying solely on accuracy can be misleading in applications where class imbalance or the cost of different error types is significant.

Weather Prediction

Scenario: Predict “Rain” or “No Rain”.

Create a Confusion Matrix from this data.

Here’s another chance to practice. Given this weather prediction scenario for a week, where ‘R’ denotes Rain and ‘NR’ denotes No Rain, first, construct the confusion matrix based on the model’s predictions versus the actual weather.

Your Turn: Calculate Metrics

Now that you have constructed the confusion matrix for the weather prediction:

• Accuracy = ?
• Precision = ?
• Recall = ?
• F1 Score = ?

Once you’ve built the confusion matrix, calculate the Accuracy, Precision, Recall, and F1 Score. Take a few moments to work through these.

Solution: Weather Prediction

Solution: Weather Prediction

Confusion Matrix:

• TP (Actual Rain, Predicted Rain): 2
• FP (Actual No Rain, Predicted Rain): 2
• FN (Actual Rain, Predicted No Rain): 2
• TN (Actual No Rain, Predicted No Rain): 1

Metrics:

• Accuracy: (2+1) / (2+2+2+1) = 3/7
• Precision: 2 / (2+2) = 2/4
• Recall: 2 / (2+2) = 2/4
• F1 Score: 2/4

Here are the solutions. This example shows a more balanced performance compared to the tumor prediction model, where all metrics are relatively consistent around two-thirds. This indicates a more even-handed performance from the classifier.

Graphical Measurements for Classification

Beyond single scalar metrics, graphical tools offer deeper insights into model performance across different decision thresholds.

Precision vs. Recall Curve

• Plots Precision against Recall for different threshold values.
• Helps select an optimal operating point based on FP/FN costs.

The Precision-Recall curve visualizes the trade-off. Each point on the curve represents the precision and recall achievable at a specific classification threshold. For example, if your ECE system needs extremely high precision (e.g., avoiding false alarms for a critical alert), you might choose an operating point far to the right on the curve, even if it means lower recall. Conversely, if high recall is paramount (e.g., never missing a faulty component), you would choose a point further to the left, accepting more false positives. This curve is especially useful when one class is significantly rarer than the other, and helps you tune your ML model for the specific operational constraints of your ECE application.

Receiver Operating Characteristic (ROC) Curve

• Plots True Positive Rate (Recall) against False Positive Rate.
• Helps compare models across all possible thresholds.

Another powerful graphical tool is the ROC curve. It plots the True Positive Rate, which is identical to Recall, against the False Positive Rate for various sensitivity thresholds. The False Positive Rate quantifies how many negative instances the model incorrectly classified as positive. A perfect classifier would reach the top-left corner (100% TPR, 0% FPR). The dotted line represents a random classifier.

ROC Curve: True Positive Rate (TPR) / Recall

\[ \text{TPR (Recall)} = \frac{\text{TP}}{\text{TP} + \text{FN}} \]

The Y-axis of the ROC curve is the True Positive Rate, which is simply our familiar Recall. It tells us, out of all actual positive cases, what fraction the model correctly identified.

ROC Curve: False Positive Rate (FPR)

\[ \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} \]

• FPR is 1 minus the True Negative Rate (TNR, Specificity).
• Measures how many actual negative examples were falsely predicted as positive.

The X-axis of the ROC curve is the False Positive Rate. This is 1 minus Specificity, where Specificity is the True Negative Rate. FPR indicates the proportion of actual negative cases that were incorrectly classified as positive. So, if we have a system for detecting anomalies, a high FPR means we get a lot of false alarms from normal operations.

Interpreting the ROC Curve

• TPR (Y-axis): Proportion of actual positives correctly identified.
• FPR (X-axis): Proportion of actual negatives incorrectly identified as positive.
• Dotted Line: Represents a random classifier (AUC = 0.5).
• Area Under Curve (AUC): Single scalar metric to summarize the curve.
  • AUC near 1.0 indicates excellent discriminative power.
  • AUC near 0.5 suggests poor or random classification.

An ideal ROC curve hugs the top-left corner, signifying high TPR and low FPR across various thresholds. The Area Under the Curve (AUC) provides a single value summary of the model’s performance across all possible classification thresholds. An AUC of 1.0 is a perfect classifier, while 0.5 is no better than random guessing. AUC is a robust metric for comparing different models, especially when the class distribution is balanced or when the cost of FP and FN is similar. In ECE, AUC is commonly used for evaluating diagnostic systems, anomaly detection, and classification problems where overall discriminative ability is key, regardless of a specific operating point initially.

01. Binary Classification

Now that we’ve covered the basics of classification and its evaluation, let’s dive into the simplest form: binary classification. This is fundamental and forms the building block for more complex multi-class problems.

Binary Classification: Two Outcomes

Yes or No

• Predicts one of two discrete values or states.
• Commonly encoded as 0 or 1.
• Examples:
  • Spam / Not Spam
  • Fault / No Fault
  • Signal Present / Signal Absent
  • Pass / Fail for product testing

Binary classification problems involve predicting one of only two possible outcomes. These outcomes are often represented as a positive class (e.g., 1) and a negative class (e.g., 0). In ECE, binary classification is ubiquitous. Think of a sensor determining if a component’s temperature is “over threshold” or “normal,” a communications system detecting if a packet contains “error” or “no error,” or an embedded system classifying a touch input as “press” or “release.”

Binary Classification: Common Models

• Logistic Regression: Transforms linear regression output into a probability (0-1).
• Decision Trees & Random Forests: Can naturally split data into two categories.
• Support Vector Machines (SVM): Finds an optimal hyperplane to separate classes with the largest margin.
• Bayesian Networks: Probabilistic graphical models used for classification.
• Neural Networks: Highly versatile, learn complex non-linear boundaries.

Many different machine learning models can be adapted for binary classification. Each has its own strengths and weaknesses regarding computational complexity, interpretability, and ability to handle different data distributions. We will explore several of these throughout the course. For an ECE context, the choice often depends on the available computational resources on a chip or embedded system, the latency requirements, and the complexity of the decision boundary needed.

Binary Classification: Logistic Regression Example

Classification

• Finds a logistic function to separate two classes.
• Outputs a probability value (0-1) which is then thresholded for classification.
• Relatively easy to interpret.

In the upcoming lab, we will build a logistic regression model. This algorithm is particularly useful for binary classification. It applies a sigmoid function to the output of a linear model, squeezing the result into a range between 0 and 1, which can be interpreted as a probability. You can then set a threshold (e.g., 0.5) to classify points. While straightforward, it performs best when the data is linearly separable or can be made so with feature engineering. This is a good starting point for many ECE tasks due to its computational efficiency.

Lab Preview: Fruit Classification

Fruit Classification

• Objective: Differentiate between oranges and grapefruit.
• Dataset: Contains features like weight, size, and color.
• Model: We will build a logistic regression model.

In our hands-on lab, you’ll tackle a binary classification problem: distinguishing between oranges and grapefruits. We’ll utilize a dataset rich with physical characteristics like weight, size, and color. This is a practical example of how classification can be used in areas like automated sorting systems in agriculture or quality control in food processing, which are relevant to ECE.

Lab Preview: Hyperparameter Tuning with Grid Search

search = GridSearchCV(model, {
  'learning_rate': [1e-3, 1e-4],
  'max_iter': [10000, 15000],
  'C': 1,
})

• Grid Search: Systematically explores a combination of hyperparameters.
• Helps find the best parameter settings for your model.
• Can be computationally intensive, especially with many parameters.

In the lab, you’ll also encounter GridSearchCV, a powerful technique for hyperparameter tuning. Instead of manually trying different learning_rate or max_iter values, Grid Search automates this process. It tries every possible combination of the provided hyperparameter values, trains a model for each combination, and evaluates its performance to find the optimal set. While incredibly useful for robust model development, remember that the number of combinations grows factorially, which can make it time-consuming, especially for complex ECE models running on limited embedded hardware.

Lab Preview: Confusion Matrix Generation

Confusion Matrix

• You will generate your first confusion matrix.
• Visualizing TP, FP, FN, TN for your fruit classifier.

Finally, the lab will guide you through creating your first confusion matrix, just like we discussed earlier. This will allow you to see the real-world performance of your logistic regression model on the fruit classification task and apply the metrics we’ve learned today.

Your Turn: Binary Classification Lab

Let’s apply these concepts and build a binary classifier!

Now it’s time to put what you’ve learned into practice! Head over to the lab environment to begin working on the binary classification exercise.

03. Multiclass Classification

Moving beyond two categories, we now explore scenarios where models must distinguish between more than two distinct classes—a common occurrence in many advanced ECE applications.

Multiclass Classification: Many Outcomes

• Classification problems with more than two classes.
• Examples:
  • Digit recognition (0-9)
  • Speech command recognition (e.g., “activate,” “mute,” “volume up”)
  • Modulation scheme identification (e.g., BPSK, QPSK, 16-QAM)
  • Component type classification

“Multiclass classification” refers to problems where a data point needs to be assigned to one of several possible categories. This is distinct from binary classification, which handles only two categories. In ECE, multi-class problems are prevalent: classifying different types of electrical faults, recognizing multiple speech commands for a microcontroller, identifying different types of cybersecurity attacks, or classifying different materials using sensor data.

Multiclass Strategies: One-vs-All (OvA) & One-vs-One (OvO)

• One-vs-All (OvA):
  • Trains k binary classifiers for k classes.
  • Each classifier distinguishes one class from all others.
  • Final prediction is the class with the highest confidence.
• One-vs-One (OvO):
  • Trains k * (k-1) / 2 binary classifiers.
  • Each classifier distinguishes one class from another specific class.
  • Final prediction is derived by a voting scheme among classifiers.

Many binary classification algorithms can be extended to handle multi-class problems using strategies like One-vs-All (OvA) or One-vs-One (OvO). OvA is simpler: for each class, you train a binary classifier to decide if an instance belongs to that specific class or any other class. OvO is more complex, training a binary classifier for every unique pair of classes, and then aggregating their votes. While often hidden from the user in modern ML libraries, understanding these strategies helps in debugging and selecting appropriate models, especially in resource-constrained ECE systems where computational efficiency matters. Some models, like Decision Trees, naturally handle multiple classes without needing these decomposition strategies.

Lab Preview: The Iris Dataset

• Classic ML Dataset: Widely used for multiclass classification.
• Features: Sepal length, sepal width, petal length, petal width.
• Target: Three species of Iris flowers (Setosa, Versicolor, Virginica).

For our multiclass classification lab, we’ll use the famous Iris dataset. This dataset is a cornerstone in machine learning and provides a perfect example for practicing multi-class problems. The features are physical measurements of different parts of the flower, and the goal is to classify the species. This type of problem is analogous to classifying different types of electronic components based on their physical or electrical characteristics, which could be relevant in automated manufacturing and quality control in ECE.

Lab Preview: Cross-Fold Validation

1. Shuffle data.
2. Split into k groups (folds).
3. Iterate k times:
   • Use one fold as test set.
   • Use remaining k-1 folds as training set.
   • Train model on training data.
   • Evaluate on test data and record performance.
4. Average performance metrics across k iterations.

In this lab, we’ll also introduce cross-fold validation. This technique is crucial for robust model evaluation, especially with smaller datasets, as it ensures that your model’s performance isn’t just an artifact of a particular train-test split. By repeatedly partitioning your data and evaluating the model, you get a much more reliable estimate of its generalization performance. For ECE applications, this means you can be more confident that a model trained in the lab will perform similarly when deployed in real-world conditions, preventing unexpected behavior in critical systems.

Lab Preview: Wine Producer Identification

• Challenge: Identify wine producers based on chemical properties.
• Dataset: Chemical analysis of different wines.
• Your Task: Apply your ML skills with minimal guidance.

For your final exercise in the lab, you’ll tackle another multiclass problem: identifying the producer of a wine based on its chemical composition. This exercise will provide less guidance, challenging you to independently apply the machine learning skills you’ve acquired. This scenario mirrors many real-world ECE problems where sensor data (e.g., from spectroscopy or chemical analysis) is used for classification tasks like material identification, quality control, or authentication.

Your Turn: Multiclass Classification Lab

Time to apply your knowledge to solve multiclass problems!

You’ve now covered the theoretical foundations of multiclass classification and learned about powerful evaluation techniques like cross-fold validation. The lab awaits, where you’ll get hands-on experience with these concepts, culminating in a challenge to really test your machine learning prowess. Good luck!

Classification with TensorFlow

We have performed binary and multiclass classification with scikit-learn. We’ll now use the TensorFlow toolkit to create a deep neural network that can perform classification. TensorFlow is a powerful open-source library developed by Google, widely used in ECE for advanced applications like image recognition, natural language processing, and embedded AI due to its flexibility and performance.

Dataset: UCI Heart Disease

Predicting the presence of heart disease

Note

This will be a binary classification problem: 0 = does not have heart disease 1 = has heart disease

The dataset we’ll use is the UCI Heart Disease dataset. This dataset contains health information about patients, as well as a “presence of heart disease” indicator. This indicator is a 1 for “has heart disease” and 0 for “does not have heart disease.” As you can probably guess, the model that we will build will be a binary classification model. This medical application highlights the ethical and high-stakes nature of such predictions, relevant to bio-medical engineering aspects of ECE.

Dataset: Features

Feature	Description
age	Age in years.
sex	Sex (0 = female, 1 = male).
cp	Chest pain type (1 = typical angina, 2 = atypical angina, 3 = non-anginal pain, 4 = asymptomatic).

The dataset contains 13 features. ‘age’ is an integer value representing the patient’s age in years. ‘sex’ is a categorical column with zero representing female and one representing male. ‘cp’ stands for chest pain. It is a categorical column with the four values shown. These features are typical inputs for many data-driven ECE systems that process various sensor or clinical data.

Dataset: Features (continued)

Feature	Description
trestbps	Resting blood pressure in Hg.
chol	Serum cholesterol in mg/dl.
fbs	Is fasting blood sugar > 120 mg/dl (0 = false, 1 = true).
restecg	Results of a resting electrocardiograph (0 = normal, 1 = ST-T wave abnormality, 2 = left ventricular hypertrophy).

trestbps is the resting blood pressure of the patient upon admission to the hospital. chol is a variable representing cholesterol. fbs is a measure of fasting blood sugar, but it is represented as a categorical column that measures if blood sugar is over a threshold. restecg is a categorical column with the three values shown. These are physiological signals, often acquired through sensor arrays, that ECE engineers would be responsible for designing and processing.

Dataset: Features (continued)

Feature	Description
thalach	Max heart rate.
exang	Exercise induced angina (0 = no, 1 = yes).
oldpeak	Measurement of an abnormal ST depression.
slope	Slope of peak of exercise ST segment (1 = upslope, 2 = flat, 3 = downslope).

The next two columns have to do with an exercise stress test the patients completed. thalach is the maximum heart rate the patient achieved during the exercise session. exang is a categorical variable that lets us know if the exercise caused angina. oldpeak is a variable that measures ST depression. ST depression is a curve on an electrocardiogram graph where the ST segment line is very low when compared to a baseline. slope is a categorical variable that lets you know which direction the line was going at the peak exercise ST segment. These stress test parameters are derived from complex biomedical signal processing, a core area of ECE.

Dataset: Features (continued)

Feature	Description
ca	Count of major blood vessels colored by fluoroscopy (0, 1, 2, 3, or 4).
thal	Presence heart condition (0 = unknown, 1 = normal, 2 = fixed defect, 3 = reversible defect).

ca is a count of major blood vessels colored by fluoroscopy. The values are 0, 1, 2, or 3, and are limited by biology, though occasionally 4 appears due to data quirks. thal relates to a heart defect, describing if it exists and its nature. You might notice the values on the slides for some of these columns differ from the documentation. For instance, the documentation for ‘ca’ states that the values range from 0-3, but there are 4s in the data. And the documentation for ‘thal’ says that the values are 3, 6, and 7, but the actual values in the data are 0, 1, 2, and 3. The takeaway from this is that you should always read the documentation, but you should also always look at the data and verify that the documentation is accurate. When there are questions, you should do research. If you are in contact with the source of the data, ask for clarification. Though documentation is great and can really help in data science, the dataset itself is the actual ground truth. This is a critical lesson for ECE engineers handling data from new sensors or instruments.

The Model: Output Layer Activation

    tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)

• For binary classification, the final layer typically has 1 neuron.
• Uses a sigmoid activation function.
• Output range [0.0, 1.0] interpreted as prediction confidence.
• Threshold determines final class.

The model in this lab won’t look too different from the TensorFlow Keras models we built for regression analysis. The primary difference is the final layer in the model. We want to create a binary prediction that will let us know if a patient has heart disease or not. If we stick with a ReLU activation function for the output, then there is no bound for the maximum output value, so it would be impossible to understand what the prediction confidence is. Instead, we’ll use an activation function that limits the output value. In this particular lab, we use a sigmoid function, so the output is limited to the range of 0.0 to 1.0. This output is then a measure of confidence that a patient has heart disease (since “has heart disease” is the 1.0 value). We can then decide how much confidence it takes to classify the patient as having heart disease. The choice of threshold is very important for model performance, and remember that we can measure performance at different thresholds with an ROC curve. This is directly applicable to digital circuit design for ML acceleration, where the choice of activation impacts hardware implementation efficiency.

The Model: Loss Function & Optimizer

    model.compile(
        loss='binary_crossentropy',
        optimizer='Adam',
        metrics=['accuracy']
    )

• Loss Function: binary_crossentropy is standard for binary classification.
• Optimizer: Adam is an adaptive learning rate optimization algorithm.
  • Adjusts learning rate for each parameter, often faster convergence.

How we measure loss is also very important. For binary classification problems, we need to use binary cross-entropy, which is specifically designed to quantify the difference between two probability distributions. Although we’ve talked a lot about using gradient descent for optimization, there are other methods as well. Adam is one of these methods. Adam uses an adaptive learning rate. That is, it uses a different learning rate for each of the different parameters in the model. This differs from stochastic gradient descent which uses a single learning rate for all parameters. While Adam often offers faster convergence, selecting the right optimizer is a non-trivial part of ECE machine learning, influencing training time and model accuracy on specialized hardware. A lot of research is being done to understand the conditions under which different optimizers perform better.

The Model: Early Stopping

    tf.keras.callbacks.EarlyStopping(
        monitor='loss',
        min_delta=1e-3,
        patience=5,
    )

• Purpose: Prevents overfitting and reduces training time.
• Mechanism: Stops training when a monitored metric (e.g., loss) stops improving significantly.
  • monitor='loss': Watches the validation loss.
  • min_delta=1e-3: Minimum change in the monitored quantity to qualify as an improvement.
  • patience=5: Number of epochs with no improvement after which training will be stopped.

We’ll also visit early stopping in this lab. Early stopping is a model-fitting strategy where you monitor some metric - say, loss - and stop training when that metric doesn’t change enough across a number of epochs. In this example we monitor the validation loss and stop early if the loss hasn’t changed by at least 0.001 during any of the last five epochs. This technique is critical in practical ECE ML deployments to save computational resources and ensure the model generalizes well to unseen data, preventing wasted energy and time on embedded or cloud-based training platforms.

Your Turn: TensorFlow Lab

Now, it’s your turn to perform binary classification using TensorFlow Keras and deep neural networks!

This lab will connect the theoretical concepts of classification with the powerful, industry-standard tools for deep learning. You’ll gain practical experience in building, training, and evaluating a neural network for a real-world ECE-relevant problem.

04. Classification Project: Predicting Titanic Survivors

In this project you will apply what you have learned about classification and TensorFlow to complete a project from Kaggle. This is a chance to consolidate your understanding and explore a full end-to-end ML pipeline.

The Titanic Shipwreck Challenge

• Goal: Achieve a high accuracy score in predicting passenger survival.
• Application: A canonical challenge for applying binary classification.

The challenge is to achieve a high accuracy score while trying to predict which passengers survived the Titanic shipwreck. This is a classic dataset for introductory machine learning due to its mix of numerical and categorical features, missing data, and clear binary classification target. It’s an excellent way to practice data preprocessing and feature engineering, skills directly transferable to ECE applications involving sensor data or system logs.

Review: Types of Classification

What types of classification have we learned about?

Let’s take a moment to review before diving into the project. What are the key categories of classification problems we’ve covered? (Expected answers: Binary classification, Multiclass classification). Can you give examples of when each is appropriate, perhaps from an ECE context? For instance, binary for fault detection (fault/no fault) and multiclass for identifying different types of signal modulation (e.g., PSK, FSK, QAM).

Review: ML Tools for Classification

What tools have we learned about for classification?

We have learned many different models and tools for performing classification. What are some of those models and tools? (Expected answers: Scikit-learn (e.g., Logistic Regression), TensorFlow/Keras for Deep Neural Networks, concepts like OvO/OvA for multiclass conversion). Briefly explain a bit about each of the tools and when you might choose one over the other in an ECE scenario, considering factors like complexity, speed, and interpretability.

Review: Evaluation Metrics

What metrics have we learned for evaluating classification models?

Now, let’s talk about how we measure success. What are some of the evaluation metrics we’ve discussed for classification models? (Expected answers: Confusion Matrix, Accuracy, Precision, Recall, F1 Score, ROC Curve, AUC). Have students discuss what each metric measures and how we interpret them, emphasizing the importance of choosing the right metric based on the application’s cost function for FP and FN errors in ECE.

Review: Other Useful Techniques

What other useful techniques have we learned, and what are they used for?

We’ve covered a few other techniques that can be useful for model training and testing. Can you name a few and describe their purpose? (Expected answers: Hyperparameter tuning (e.g., Grid Search), Cross-validation, Early Stopping, Data preprocessing (normalization, encoding categorical features)). Discuss how these techniques contribute to building robust and generalized ML models for real-world ECE systems.

Classification Project: The Data

Column	Type	Description
Survived	number	1 or 0 ( target )
Name	string	Passenger name
Pclass	number	Ticket class
Sex	string	male or female
Age	number	Passenger age
SibSp	number	# of siblings/spouses on board
Embarked	string	Port of Embarkation

The dataset we’re using comes from Kaggle. Here are just a few of the columns of data you’ll be working with. As you can see, we have both numbers and strings. The target column is Survived, and it is a number that is either 0 or 1. This mix of data types is very common in ECE applications, where you might deal with numerical sensor readings, categorical status codes, and potentially textual log data. You’ll need to think about how to preprocess these different types of features.

Classification Project: Kaggle Competition

Titanic: Machine Learning from Disaster

• Engage with a global community of ML practitioners.
• Upload your results to compare your model’s performance.

Kaggle hosts several competitions that are open to users. It’s an exciting way to engage with the broader machine learning community and learn new things. At the end of this lab, you will upload your results to the Kaggle competition and see how your model compares to the over 17,000 other models people have created! This provides a real-world context for competition and benchmarking, akin to engineering design challenges.

Classification Project: Your Turn

1. Exploratory Data Analysis (EDA):
   • Understand the data, identify obvious problems, and perform initial cleaning.
   • Consider pros/cons of using ML for this problem.
2. Model Building & Evaluation:
   • Choose your model (scikit-learn or TensorFlow).
   • Train and evaluate your model, discussing chosen metrics.
3. Make Predictions & Upload to Kaggle:
   • Generate predictions for the test dataset.
   • Submit your predictions to the Kaggle competition.
4. Iterate on Your Model:
   • Tweak hyperparameters, try different models, explore new features.
   • Discuss your methodical approach to improvement.
   • Research and compare with other solutions for deeper insights.

It is now your turn to perform a classification from end-to-end. The lab you are about to be given is divided into four primary parts. In the first section, you’ll acquire and explore the data. Here we expect you to write code and prose about the data. Does the data have obvious problems? Do any model-independent changes need to be made to the data? EDA is the place to reason about and perform these tasks. This is also a good time to think about the pros and cons of using machine learning to solve this problem. In the next section, you will build and evaluate your model. You may choose to use scikit-learn or Tensorflow. You may even try multiple approaches and compare your results. Here you should also evaluate your model and discuss your particular evaluation metrics, including why you chose them and what they say. Finally, you will make predictions on the features found in the test.csv file and upload them to Kaggle using the Kaggle API. Your lab should discuss your predictions as well as your Kaggle results. Last but not least, iterate on your model. Tweak hyperparameters, and see if you can improve your model. Discuss your method for changing specific hyperparameters. Be thoughtful and methodical; don’t just do it at random! Since this is a popular Kaggle dataset and competition, research other users’ solutions. Try looking at solutions that both do and don’t use ML, and discuss their relative merits. Take your time. Experiment. Don’t be afraid to throw away some work along the way. This iterative process is crucial for ECE engineers developing and refining complex systems.