Machine Learning Fundamentals: Concepts and Techniques

Posted on Jul 13, 2026 in Mathematics and Computer Science

Core Machine Learning Concepts

  • ML: Building systems that learn patterns from data rather than using hand-coded rules.
  • Supervised Classification: Labels represent classes or categories.
  • Regression: Predicts a numeric target value.
  • Unsupervised Learning: Examples include clustering blog visitors or visualization algorithms (2D/3D output).
  • Logistic Regression: Primarily used for classification tasks.
  • Learning Strategies:
    • Batch Learning: Trained offline on a full dataset, then deployed.
    • Online Learning: Useful for continuous data streams where the system adapts.
    • Out-of-core Learning: Used when data exceeds RAM capacity, processed in chunks.
  • Instance-based Learning: Generalizes by comparing new points to stored examples (e.g., k-NN).
  • Model Parameters vs. Hyperparameters: Parameters are learned during training; hyperparameters (e.g., regularization strength) are set beforehand.
  • Generalization Issues:
    • High Variance: Overfitting.
    • High Bias: Underfitting.
    • Sampling Bias: Nonrepresentative training sets lead to poor generalization.
  • Feature Engineering: Irrelevant features are addressed via selection or engineering.
  • Model Evaluation:
    • Test Set: Estimates generalization error on unseen data.
    • Validation Set: Used to compare models and tune hyperparameters.
  • Reinforcement Learning (RL): An agent maximizes cumulative reward (e.g., a robot learning to walk).
  • Data Importance: The “unreasonable effectiveness of data” suggests that more or better data often outweighs algorithm choice.

Regression and Data Preprocessing

  • Objective: The first step in any ML project is understanding the objective and how the output will be used.
  • Cost Measures:
    • RMSE: Standard for regression; weights large errors more heavily.
    • MAE: Preferred when there are many outliers.
    • l2 norm: Equivalent to RMSE.
    • l1 norm: Equivalent to MAE.
  • Data Splitting: Create test sets early. Use stratified sampling (e.g., train_test_split(stratify=)) to ensure the test set mirrors category proportions.
  • Pipelines: Use Sklearn Pipeline to chain preprocessing with an estimator. All steps except the last must be transformers.
  • Feature Scaling: StandardScaler is vital for normalizing feature scales.
  • Categorical Data: Convert text to numbers using OneHotEncoder.
  • Missing Values: Use SimpleImputer (e.g., median strategy) for numeric data.
  • Transformations: Use log transforms to fix long right-tail features.
  • Cross-Validation: k-fold CV averages scores across held-out folds.

Classification Metrics and Performance

  • Confusion Matrix: Diagonal elements represent correct predictions (TP & TN).
  • Accuracy: (TP+TN) / All.
  • Precision: Of predicted positives, how many are correct.
  • Recall (Sensitivity/TPR): Of actual positives, how many were caught.
  • F1 Score: Harmonic mean of precision and recall.
  • Thresholding: Raising the threshold increases precision but decreases recall.
  • ROC Curve: Plots TPR vs. FPR (1−specificity).
  • AUC: Threshold-independent metric; useful for comparing models. Prefer PR curves over ROC when the positive class is rare.

Linear Models and Gradient Descent

  • Linear Regression: Predicts via a weighted sum of features plus bias.
  • Normal Equation: Solves linear regression in closed form.
  • Gradient Descent (GD): Minimizes cost by iterating against the gradient.
  • Learning Rate: Controls step size; too small leads to slow convergence, too large may cause divergence.
  • GD Variants:
    • Batch GD: Uses the whole training set.
    • Stochastic GD (SGD): Uses one random instance per step; faster but noisier.
    • Mini-batch GD: Benefits from GPU hardware acceleration.
  • Regularization:
    • Ridge: Penalizes l2 norm of weights.
    • Lasso: Produces sparse models (weights = 0).
    • Elastic Net: Combines Ridge and Lasso.

Support Vector Machines (SVM)

  • Core Concept: Fits the widest possible “street” (margin) between classes.
  • Support Vectors: Instances on the margin edge; removing non-support-vector instances does not change the boundary.
  • Scaling: Feature scaling is mandatory as large-scale features dominate the margin.
  • C Hyperparameter: Smaller C creates a wider margin (more tolerance); larger C creates a narrower margin.
  • Kernels:
    • Polynomial Kernel: Adds nonlinearity via feature powers.
    • Gaussian RBF: Measures similarity to landmarks; decays toward 0 with distance.
    • Kernel Trick: Computes high-dimensional dot products implicitly without explicit mapping.
  • SVM Regression: Uses an ε-insensitive tube; points inside the tube do not affect the fit.