Context
I wanted to explore how machine learning can extract meaningful patterns from unstructured text. Unlike traditional numerical datasets, language requires transforming qualitative writing characteristics into measurable features. The Federalist Papers provided a compelling historical case for applying natural language processing, feature engineering and classification to a question where authorship must be inferred from writing style.
Who wrote the disputed Federalist Papers?
Authorship attribution requires identifying subtle and consistent linguistic patterns across documents written by different authors. The Federalist Papers provide a well-established benchmark because several essays have disputed authorship associated with James Madison and Alexander Hamilton. The challenge is to determine whether measurable differences in writing style can reliably distinguish the authors and support attribution of the disputed essays.
Approach
This project combined stylometric analysis with supervised machine learning to distinguish the writing styles of James Madison and Alexander Hamilton. Linguistic characteristics were engineered as interpretable features, while TF-IDF representations captured patterns in word usage. Multiple classification models were then evaluated using known-authorship papers before being applied to the disputed essays.
Data Sources
Federalist Papers Corpus
Publicly available Federalist Papers text collected from Project Gutenberg and structured into individual essays containing paper number, cleaned text, and known author labels. Papers with established authorship were used for model training, while disputed papers 49-58 and 62-63 were held out for authorship attribution.
Solution
Collected and structured Federalist Papers into a labeled document corpus
Cleaned and normalized historical text while preserving meaningful linguistic patterns
Extracted stylometric features including vocabulary size, sentence structure, punctuation usage and function-word frequency
Built TF-IDF feature representations for supervised machine learning classification
Compared Multinomial Naive Bayes, Logistic Regression and Support Vector Machine models
Evaluated models using cross-validation, accuracy, precision, recall, F1-score, ROC-AUC and log-loss metrics
Applied trained models to disputed Federalist Papers and compared attribution consistency across models
Architecture
The pipeline transforms historical documents into two complementary representations of writing style. Stylometric features capture interpretable linguistic characteristics, while TF-IDF represents patterns in word usage. These representations feed supervised classification models trained on known-authorship papers, with the resulting predictions applied to the disputed essays.
- 01Federalist Papers corpus extraction and preprocessing
- 02Text normalization and linguistic feature extraction
- 03Stylometric feature engineering
- 04TF-IDF text vectorization
- 05Classification model comparison
- 06Disputed authorship inference
Outcome
The analysis demonstrated that classical NLP and machine learning models could identify meaningful authorship patterns in the Federalist Papers. Logistic Regression and Support Vector Machine achieved 95% accuracy and ROC-AUC of 1.00 on the known-authorship evaluation set. When applied to the 12 disputed papers, all three models agreed on the attribution of 9 papers to James Madison. The remaining papers produced less consistent predictions, illustrating the uncertainty inherent in authorship attribution. Overall, the results showed increasing alignment between the model predictions and the Madison attribution used as the reference for the disputed papers, providing a useful example of how machine learning can contribute evidence to a historically ambiguous authorship problem.
Stylometric Comparison
Comparison of linguistic characteristics including vocabulary usage, document structure and writing patterns between authors.
Disputed Paper Attribution
Final authorship predictions for historically disputed Federalist Papers.
Function Word Analysis
Function word frequency analysis revealing subtle differences in writing style.
Lessons Learned
Text classification requires careful feature engineering to convert language into measurable signals
Combining interpretable linguistic analysis with predictive models provides stronger insights into authorship patterns
Classical machine learning models remain highly effective for structured NLP problems
Model agreement and confidence analysis are important when applying classification to ambiguous cases
Data preparation and corpus construction are critical components of successful NLP workflows
Future Improvements
Expand the analysis to include additional historical authors
Compare classical models against transformer-based language models
Incorporate word embeddings and contextual language representations
Explore larger historical text collections for broader authorship attribution tasks