Overview
For Georgetown University's Massive Data Fundamentals course, our four-person team investigated climate misinformation across Reddit communities. The research corpus contained approximately 17 million posts and comments from January 2021 through March 2023, accessed through the course's AWS-hosted data.
We explored environmental discussion, sentiment, climate-denial claims, external sources, and patterns of author participation. This brought together data engineering, NLP, supervised classification, and network analysis in a published research report.
Team: Keya Sengupta, Sanha Tahir, Brad Wayne, and Fanni Varhelyi.
My contributions
My hands-on work included preparing data with PySpark and S3, setting up and troubleshooting Spark NLP, persisting sentiment-enriched outputs as Parquet, and adapting pretrained CARDS RoBERTa model handling for Spark.
The wider classification, evaluation, network analysis, and published findings represent the team's combined work. The project strengthened my ability to work through real distributed-processing dependencies and connect data preparation to downstream analysis.
From raw comments to research-ready features
- 01 / PREPARE
S3 + PySpark
Filter communities, clean text, derive metadata, and aggregate trends.
- 02 / ENRICH
Spark NLP
Tokenize, lemmatize, derive dictionary-based sentiment, and write Parquet.
- 03 / CLASSIFY
Pretrained RoBERTa
Apply CARDS inference to sampled comments and combine labels with features.
- 04 / EVALUATE
Spark ML + reporting
Compare binary classifiers, explore networks, and publish a Quarto report.
Team-level stack: Python, Apache Spark / PySpark, AWS S3, SageMaker, Hadoop S3A, Spark NLP, Hugging Face Transformers, PyTorch inference, Spark ML, pandas, NumPy, Matplotlib, Seaborn, Quarto, and GitHub Pages.
The engineering decisions
Make intermediate data reusable. Columnar Parquet outputs in S3 let enriched data feed later analysis. PySpark DataFrames supported filtering, derived columns, UDFs, aggregations, and temporal comparisons across communities.
Resolve the runtime before trusting the pipeline. My troubleshooting covered Java gateway startup, missing S3A and credential-provider classes, and Hadoop/AWS dependency compatibility. These practical integration details determined whether the processing environment could access data and run reliably.
Match model inference to available compute. Full-corpus transformer inference exceeded practical compute constraints, so the team applied pretrained CARDS RoBERTa to sampled comments. This was pretrained inference, rather than training or fine-tuning RoBERTa.
Build a separate downstream prediction workflow. The team combined text length, sentiment, community, and other metadata in Spark ML pipelines, using categorical encoding, feature assembly, scaling, minority oversampling, logistic regression, and gradient-boosted trees. Evaluation included confusion matrices and cross-validation.
Outcomes and interpretation
The project produced code, enriched datasets, visualizations, and a published research report examining discussion patterns and model-identified denial claims. It demonstrates how cloud-based data preparation can support a broader analytical workflow.
The approximately 17 million records describe the overall research corpus. RoBERTa ran on smaller, uneven samples; the downstream classifier notebook reports 19,532 input records. These are distinct stages with different scales.
Findings are descriptive and model-dependent. The downstream labels came from RoBERTa rather than independent human annotation, and uneven sampling limits comparisons between communities. Evaluation also warrants care around preprocessing before the train/test split and oversampling during cross-validation. This was an academic research workflow, with lessons that inform thoughtful engineering and model evaluation.
Explore the research
Explore the sentiment pipeline ↗