publications

2025

1. 

   Making FETCH! Happen: Finding Emergent Dog Whistles Through Common Habitats

   Kuleen Sasse, Carlos Alejandro Aguirre, Isabel Cachola, and 2 more authors

   In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2025

   Abs DOI

   Dog whistles are coded expressions with dual meanings: one intended for the general public (outgroup) and another that conveys a specific message to an intended audience (ingroup). Often, these expressions are used to convey controversial political opinions while maintaining plausible deniability and slip by content moderation filters. Identification of dog whistles relies on curated lexicons, which have trouble keeping up to date. We introduce FETCH!, a task for finding novel dog whistles in massive social media corpora. We find that state-of-the-art systems fail to achieve meaningful results across three distinct social media case studies. We present EarShot, a strong baseline system that combines the strengths of vector databases and Large Language Models (LLMs) to efficiently and effectively identify new dog whistles.

2. Controllable Hybrid Captioner for Improved Long-form Video Understanding

   Kuleen Sasse, Efsun Sarioglu Kayi, and Arun Reddy

   Jul 2025

3. 

   debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias

   Kuleen Sasse, Shan Chen, Jackson Pond, and 2 more authors

   Jul 2025

4. 

   Sparse Autoencoder Features for Classifications and Transferability

   Jack Gallifant, Shan Chen, Kuleen Sasse, and 3 more authors

   Jul 2025

2024

1. Disease Entity Recognition and Normalization is Improved with Large Language Model Derived Synthetic Normalized Mentions

   Kuleen Sasse, Shinjitha Vadlakonda, Richard E. Kennedy, and 1 more author

   Jul 2024
2. Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models’ Ability to Resist Requests for Misinformation

   Shan Chen, Mingye Gao, Kuleen Sasse, and 6 more authors

   Jul 2024
3. Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models

   Carlos Alejandro Aguirre, Kuleen Sasse, Isabel Alyssa Cachola, and 1 more author

   In Proceedings of the Third Workshop on NLP for Positive Impact, Nov 2024

   Abs DOI

   Recently, work in NLP has shifted to few-shot (in-context) learning, with large language models (LLMs) performing well across a range of tasks. However, while fairness evaluations have become a standard for supervised methods, little is known about the fairness of LLMs as prediction systems. Further, common standard methods for fairness involve access to model weights or are applied during finetuning, which are not applicable in few-shot learning. Do LLMs exhibit prediction biases when used for standard NLP tasks?In this work, we analyze the effect of shots, which directly affect the performance of models, on the fairness of LLMs as NLP classification systems. We consider how different shot selection strategies, both existing and new demographically sensitive methods, affect model fairness across three standard fairness datasets. We find that overall the performance of LLMs is not indicative of their fairness, and there is not a single method that fits all scenarios. In light of these facts, we discuss how future work can include LLM fairness in evaluations.

4. Understanding the determinants of vaccine hesitancy in the United States: A comparison of social surveys and social media

   Kuleen Sasse, Ron Mahabir, Olga Gkountouna, and 2 more authors

   PLOS ONE, Jun 2024

   Abs DOI

   The COVID-19 pandemic prompted governments worldwide to implement a range of containment measures, including mass gathering restrictions, social distancing, and school closures. Despite these efforts, vaccines continue to be the safest and most effective means of combating such viruses. Yet, vaccine hesitancy persists, posing a significant public health concern, particularly with the emergence of new COVID-19 variants. To effectively address this issue, timely data is crucial for understanding the various factors contributing to vaccine hesitancy. While previous research has largely relied on traditional surveys for this information, recent sources of data, such as social media, have gained attention. However, the potential of social media data as a reliable proxy for information on population hesitancy, especially when compared with survey data, remains underexplored. This paper aims to bridge this gap. Our approach uses social, demographic, and economic data to predict vaccine hesitancy levels in the ten most populous US metropolitan areas. We employ machine learning algorithms to compare a set of baseline models that contain only these variables with models that incorporate survey data and social media data separately. Our results show that XGBoost algorithm consistently outperforms Random Forest and Linear Regression, with marginal differences between Random Forest and XGBoost. This was especially the case with models that incorporate survey or social media data, thus highlighting the promise of the latter data as a complementary information source. Results also reveal variations in influential variables across the five hesitancy classes, such as age, ethnicity, occupation, and political inclination. Further, the application of models to different MSAs yields mixed results, emphasizing the uniqueness of communities and the need for complementary data approaches. In summary, this study underscores social media data’s potential for understanding vaccine hesitancy, emphasizes the importance of tailoring interventions to specific communities, and suggests the value of combining different data sources.

2023

1. To Burst or Not to Burst: Generating and Quantifying Improbable Text

   Kuleen Sasse, Efsun Sarioglu Kayi, Samuel Barham, and 1 more author

   In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Dec 2023

   Abs

   While large language models (LLMs) are extremely capable at text generation, their outputs are still distinguishable from human-authored text. We explore this separation across many metrics over text, many sampling techniques, many types of text data, and across two popular LLMs, LLaMA and Vicuna. Along the way, we introduce a new metric, recoverability, to highlight differences between human and machine text; and we propose a new sampling technique, burst sampling, designed to close this gap. We find that LLaMA and Vicuna have distinct distributions under many of the metrics, and that this influences our results: Recoverability separates real from fake text better than any other metric when using LLaMA. When using Vicuna, burst sampling produces text which is distributionally closer to real text compared to other sampling techniques.

2021

1. Adopt a Pixel 3 km: A Multiscale Data Set Linking Remotely Sensed Land Cover Imagery With Field Based Citizen Science Observation

   Frontiers in Climate, Dec 2021

   DOI

2020

1. Identification of Cancer Entities in Clinical Text Combining Transformers with Dictionary Features.

   John D Osborne, Tobias O’Leary, James Del Monte, and 1 more author

   In IberLEF@ SEPLN, Dec 2020