Machine Learning

/// problem

Naive Bayes assumes feature independence and works natively on categorical data, but the UCI datasets we were given mixed continuous and categorical features, class counts from 2 to 6, and sizes from a few dozen to several hundred rows. We wanted to build the classifier from scratch and see how well the independence assumption holds across that shape variation.

/// approach

Implemented the full pipeline from scratch: equal-width discretization for continuous features (10 bins), Laplace-smoothed prior and conditional probability tables, log-space prediction to avoid underflow, and 10-fold cross-validation. Evaluated on five UCI datasets (Wisconsin breast cancer, '84 House votes, Iris, Soybean, Glass) with precision, recall, and F1 from confusion matrices. Also tested robustness by injecting random noise into 10% of the data.

/// takeaway

Naive Bayes was effective everywhere we tried it, but the gap between dataset shapes was wide. Breast cancer and Soybean reached 0.93 to 1.00 F1 (clean categorical features or lots of attributes); Glass collapsed to roughly 0.6 F1 with six classes and all-continuous features. Equal-width discretization carried the continuous datasets cleanly but is brittle to outliers, which is what we'd expect from a model whose independence assumption tells you upfront which problems it fits.

/// problem

Naive Bayes carried us through categorical data, but we needed methods that handle continuous features natively and work for both classification and regression. We wanted to build three nonparametric algorithms (KNN, Edited KNN, K-Means) from scratch and see how they trade off accuracy, hyperparameter sensitivity, and tuning cost across the two task types.

/// approach

Implemented KNN (Euclidean distance, plurality vote for classification, RBF-kernel weighted average for regression), Edited KNN (drop training points the model misclassifies under k=1, then run KNN on the survivors), and K-Means (assign points to nearest centroid, update, repeat to stability, predict via 1-NN to the cluster centroid). Added stratified k-fold cross-validation, one-hot encoding for categoricals, mean/mode imputation for missing values, and grid search over k, sigma, and epsilon. Evaluated on six UCI datasets, three classification and three regression.

/// takeaway

Plain KNN was the most reliable across the classification datasets (95.9% on breast cancer, 61% on noisy Glass, 100% on tiny Soybean). Edited KNN was a disappointment: roughly the same or slightly worse on classification, and tuning the extra ε hyperparameter on regression problems blew up combinatorially (one dataset would have taken seven hours to tune, so we skipped it). K-Means worked when natural clusters aligned with class labels (Soybean) and broke down when they didn't (breast cancer). Across the board, regression was harder than classification for all three methods, with each showing clear overfitting to the tuning set.

/// problem

Naive Bayes and KNN both depended on strong assumptions about the data. We wanted to build a general-purpose model that can learn arbitrary input-to-output mappings from scratch: a feed-forward neural network flexible enough to handle both classification and regression on the same six UCI datasets we'd been using.

/// approach

Built a configurable FFNN with arbitrary hidden layers and node counts, logistic activations on hidden layers, softmax plus cross-entropy for classification, and linear output plus MSE for regression. Backpropagation with momentum, mini-batch gradient descent, min-max normalization on inputs to mitigate vanishing gradients, and early stopping with a patience hyperparameter. Grid-searched over architecture (zero, one, or two hidden layers), learning rate, momentum, batch size, and patience, then validated with 10-fold stratified cross-validation.

/// takeaway

Surprising result: the best architecture on most datasets was zero hidden layers, meaning the predictions were essentially linear and a deeper network didn't help. The hypothesis that more layers would mean higher accuracy traded off against more overfitting didn't hold; the actual effect varied per dataset, and only Abalone (large, simple features) behaved the way the textbook suggested. Got viscerally familiar with how unforgiving backprop is to linear-algebra bugs: most of the project's time went into debugging the gradient computations, and even with everything working the model performed unevenly across datasets.

/// problem

Backprop trained the network from Paper 3 but the results were uneven, and the math behind it is brittle to bugs. We wanted to try population-based, gradient-free optimization methods to tune the same FFNN architecture and see whether they'd produce more reliable models on the same six datasets.

/// approach

Implemented three population-based optimizers from scratch, each evolving a population of FFNNs (fixed at 2 hidden layers of 5 nodes for fair comparison): genetic algorithm (fitness-weighted selection, uniform crossover, Gaussian mutation, steady-state replacement), differential evolution (DE/each/2/z donor vectors with binomial crossover), and particle swarm optimization (star topology, velocity updates from inertia plus personal-best plus global-best terms). Tuned each algorithm's hyperparameters via grid search and compared against the Paper 3 backprop baseline using 10-fold cross-validation.

/// takeaway

On classification, all three population methods beat backprop convincingly: Breast Cancer's cross-validation loss dropped from 2.39 to between 0.25 and 0.43, and Glass dropped from 0.82 to between 0.19 and 0.28. On regression the story flipped: GA matched backprop on Abalone, but DE and PSO showed clear premature convergence and overfit by an order of magnitude on the trickier datasets. Most of our hypotheses didn't pan out (GA didn't avoid overfitting despite steady-state replacement's lower sampling pressure, DE was the most volatile rather than the safest). Confirmed gradient-free methods are a legitimate alternative on classification but harder to tune well on regression.