Supervised Machine Learning
Mathematical foundations of predictive modeling: cost functions (MSE), learning rates, gradient descent parameter updates, Bayes theorem, and Naive Bayes text classification.
What is Machine Learning?
Machine Learning is a subfield of artificial intelligence where computer systems learn mathematical models and predictive rules directly from data, rather than being explicitly hardcoded.
A computational system that iteratively learns predictive models h(x) from training data instances without manual rule programming.
Supervised learning relies on labeled training pairs (x, y) to predict outputs for unseen instances. Unsupervised learning analyzes unlabeled data to uncover natural latent clusters.
Supervised Learning Pipeline
In supervised learning, a training set of input-output pairs is fed into a learning algorithm to produce a hypothesis function h that maps new feature vectors x to predicted outputs y.
Input vector x = [Size ft², Bedrooms, Floors, Age] maps to Target Output Y = Price ($1000). For continuous targets, this is Regression; for discrete labels (Yes/No, Spam/Ham), this is Classification.
- Instance x(1) = [2104, 5, 1, 45] -> Target Y = $460k
- Instance x(2) = [1416, 3, 2, 40] -> Target Y = $232k
- Instance x(3) = [1534, 3, 2, 30] -> Target Y = $315k
Feature Vectors & Notation
A feature vector x = [x1, x2, ..., xn] is a structured array of measurable attributes describing a specific sample in the dataset.
Dataset Representation
The entire purpose of the learning algorithm is to find the function h that minimizes prediction error on unseen x.
Bayes Theorem — Mathematical Foundations
Bayes Theorem allows us to invert conditional probabilities, calculating the posterior probability of a class given observed features.
Bayes Theorem
Because the denominator P(F) is identical for all classes during a classification query, it can be dropped when finding the argmax class.
The Naïve Bayes Classifier & Independence Assumption
Naïve Bayes makes the simplifying assumption that all input features are conditionally independent given the class label.
Naïve Bayes Optimization Objective
The Conditional Independence Assumption: P(F1, F2, ..., Fn | C) = P(F1|C) * P(F2|C) * ... * P(Fn|C). Although features are rarely 100% independent in real life, Naïve Bayes works remarkably well in practice, especially for text and spam classification.
1-Feature Worked Example: 'Drew' Gender Classification
Consider a dataset of 8 individuals (3 Male, 5 Female). A new person arrives with Name = 'Drew'. What is their most probable gender?
Training data: Male = 3, Female = 5. Drew appears 1 time under Male and 2 times under Female.
- P(Male) = 3/8 = 0.375 | P(Female) = 5/8 = 0.625
- Likelihood P(Drew | Male) = 1/3 ≈ 0.333
- Likelihood P(Drew | Female) = 2/5 = 0.400
- Posterior P(Male | Drew) ∝ 1/3 * 3/8 = 1/8 = 0.125 (Normalized: 33.3%)
- Posterior P(Female | Drew) ∝ 2/5 * 5/8 = 2/8 = 0.250 (Normalized: 66.7%)
- Conclusion: P(Female | Drew) > P(Male | Drew) -> Classified as Female.
Full Worked Example: Play Tennis Dataset ⭐ (Exam Classic)
Given 14 days of weather conditions, classify whether tennis will be played on a day with: Outlook=Sunny, Temp=Cool, Humidity=High, Wind=Strong.
Tennis Posterior Probability Comparison
Advantages & Limitations of Naïve Bayes
Understanding the strengths and trade-offs of the algorithm for real-world engineering deployments.
Interactive Quiz: Naïve Bayes Predictor
What is the primary assumption made by the Naïve Bayes Classifier?
Python Implementation from Scratch (Tennis Dataset)
1# Naive Bayes Classifier - Play Tennis Example2P_yes = 9 / 143P_no = 5 / 1445# Likelihoods for Query: (Outlook=Sunny, Temp=Cool, Hum=High, Wind=Strong)6# Yes likelihoods:7P_sunny_yes = 2 / 98P_cool_yes = 3 / 99P_high_yes = 3 / 910P_strong_yes = 3 / 91112# No likelihoods:13P_sunny_no = 3 / 514P_cool_no = 1 / 515P_high_no = 4 / 516P_strong_no = 3 / 51718score_yes = P_yes * P_sunny_yes * P_cool_yes * P_high_yes * P_strong_yes19score_no = P_no * P_sunny_no * P_cool_no * P_high_no * P_strong_no2021total = score_yes + score_no22p_yes_norm = score_yes / total23p_no_norm = score_no / total2425print(f"Score Yes: {score_yes:.6f} -> Normalized: {p_yes_norm * 100:.1f}%")26print(f"Score No: {score_no:.6f} -> Normalized: {p_no_norm * 100:.1f}%")27print(f"Final Decision: PlayTennis = {'Yes' if score_yes > score_no else 'No'}")
Score Yes: 0.005291 -> Normalized: 20.5% Score No: 0.020571 -> Normalized: 79.5% Final Decision: PlayTennis = No