Machine Learning Foundations & Definitions

Machine Learning focuses on algorithms that allow computers to learn patterns directly from data without being explicitly programmed. [1]

* Tom Mitchell's Definition: A computer program is said to learn from Experience ($E$) with respect to some class of Tasks ($T$) and Performance measure ($P$), if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

* Data: The raw foundation, which can be structured, semi-structured, or unstructured.

* Model: A mathematical representation of a real-world process derived from data.

* Loss Function: A mathematical metric quantifying how much a model's prediction deviates from the true target value. [1, 2, 3]

------------------------------

## Machine Learning Frameworks & Paradigm Comparison

[ Machine Learning Paradigms ]

│

┌─────────────────────────────┼─────────────────────────────┐

▼ ▼ ▼

[ Supervised Learning ] [ Unsupervised Learning ] [ Reinforcement Learning ]

- Labeled Data - Unlabeled Data - State/Action Feedback

- Predict Target Outputs - Discover Hidden Patterns - Reward/Penalty Signal

## Types of Learning

* Supervised Learning: The algorithm learns a mapping function from input variables ($X$) to labeled output variables ($Y$).

* Unsupervised Learning: The algorithm uncovers hidden structures, relationships, or clusters within completely unlabeled data ($X$).

* Ensemble Learning: A meta-approach that combines predictions from multiple base models to produce a single, more robust output.

* Reinforcement Learning: An autonomous agent interacts with a dynamic environment via trial and error to maximize cumulative rewards. [4, 5, 6, 7, 8]

## Comparative Summary

|---|---|---|---|---|

------------------------------

## Designing a Learning System

Building an operational learning system requires walking through an explicit series of architectural steps. [9]

[ Choose Target Function ] ──> [ Choose Representation ] ──> [ Choose Learning Algorithm ]

1. Choosing the Training Experience: Determining the type, volume, and quality of data feedback available (e.g., direct feedback vs. indirect feedback). [1]

2. Choosing the Target Function: Defining exactly what type of knowledge is to be learned (e.g., a function $V: State \rightarrow \mathbb{R}$ that evaluates system states).

3. Choosing a Representation for the Target Function: Deciding how to approximate the function mathematically (e.g., a linear combination of features, a polynomial, or a network of nodes). [9]

4. Choosing a Learning Algorithm: Selecting a robust optimization procedure (e.g., Gradient Descent) to adjust internal weights to accurately fit the target function. [9, 10, 11]

## Key Perspectives and Operational Issues

* Data Quality: Unrepresentative, biased, or noisy input data severely limits learning capabilities.

* Exploration vs. Exploitation: Balancing the choice between tracking known high-yield actions and seeking out new, unvisited states.

* Computational Scalability: Ensuring optimization strategies remain efficient as data scales exponentially up. [1, 7, 10, 11, 12]

------------------------------

## Hypothesis Space & Inductive Bias## Hypothesis Space ($H$)

The Hypothesis Space ($H$) is the complete set of all possible target functions or configurations that an algorithm can evaluate to fit the data. Selecting a model architecture restricts what your system can learn. For instance, a simple linear model cannot represent complex, non-linear boundaries. [5, 9, 13]

## Inductive Bias

Inductive Bias is the set of explicit assumptions a machine learning algorithm uses to predict outputs for unobserved, future examples without depending solely on raw data. Without an inductive bias, an algorithm has no logical basis to generalize beyond observed training points. [5, 11]

* Restrictive Bias (Language Bias): Restricts the absolute search space to a specific subset of functions.

* Example: Linear Regression assumes the true relationship is strictly linear. [14]

* Preferential Bias (Search Bias): Prefers certain hypotheses within the search space over others based on specific rules.

* Example: Decision Trees prefer shorter, simpler trees over highly complex ones (adhering to Occam's Razor). [14]

------------------------------

## Algorithms: Linear Regression & Decision Trees## Linear Regression

Linear Regression models the predictive relationship between continuous independent features and a continuous target scalar variable. [6, 8]

* Mathematical Formula:

$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_n X_n + \epsilon$$

Where $\beta_0$ is the intercept, $\beta_i$ represent feature coefficients, and $\epsilon$ represents random residual error.

* Optimization Goal: Minimize the Ordinary Least Squares (OLS) objective, which calculates the Sum of Squared Errors:

$$SSE = \sum_{i=1}^{m} (y_i - \hat{y}_i)^2$$

## Decision Trees

Decision Trees partition data into increasingly smaller, homogeneous subsets by applying logical, axis-aligned binary checks. [8]

* Splitting Criteria: Evaluated using information metrics such as Information Gain (relying on Entropy) or the Gini Impurity Index to identify the most discriminative splits.

* Structure: Composed of interior Decision Nodes (representing criteria evaluations) and Terminal Leaf Nodes (containing final class or value predictions). [8]

------------------------------

## Overfitting & System Evaluation## Overfitting

Overfitting happens when a model learns the training data too deeply—including its random noise, anomalies, and background variations. [8, 11]

* The Result: The model shows exceptional accuracy on training data but fails to generalize to new, unseen testing data.

* Causes: Excessively complex models (e.g., highly deep decision trees), noisy datasets, or an overly small training sample. [11]

## Evaluation & Cross-Validation

To accurately evaluate real-world generalization performance, datasets are separated into isolated splits. [10, 15]

Standard K-Fold Cross-Validation:

Dataset ──> Split into [ K ] Equal Folds

Iteration 1: [ Test ] [ Train ] [ Train ] [ Train ] ──> Score 1

Iteration 2: [ Train ] [ Test ] [ Train ] [ Train ] ──> Score 2

...

Final Performance = Average(Scores)

* Holdout Validation: The data is split into discrete training, validation, and testing sets. [15, 16]

* K-Fold Cross-Validation:

1. The complete dataset is split evenly into $K$ equal, non-overlapping folds.

2. The model trains over $K-1$ folds and evaluates performance on the remaining test fold.

3. This loop repeats $K$ times so every fold serves exactly once as the test set.

4. Final metrics are calculated by averaging the individual performance scores across all runs.

------------------------------

## Feature Engineering

Feature engineering involves applying domain knowledge to transform raw structural data into informative input features that help algorithms optimize efficiently. [17, 18]

* Imputation: Substituting missing or null fields with calculated statistical metrics (e.g., median values or modes).

* One-Hot Encoding: Converting unstructured categorical labels into numeric binary vectors ($0$ or $1$).

* Feature Scaling: Standardizing disparate numerical features into matching scales using Min-Max Normalization or Z-score Standardization.

* Interaction Features: Creating new attributes by combining existing features mathematically (e.g., multiplying $width \times height$ to yield $area$).

------------------------------

## Applications of Machine Learning

* Healthcare: Diagnosing diseases from medical imaging scans and predicting patient readmission rates.

* Finance: Real-time credit card fraud detection, stock price trend prediction, and automated algorithmic trading.

* E-Commerce: Building collaborative filtering recommendation engines and automating customer support using natural language chatbots.

* Autonomous Driving: Processing real-time sensor layouts for lane-tracking, pedestrian detection, and path planning. [7, 16, 18, 19]

Study Material

Machine Learning Foundations & Definitions

Comments

Post a Comment