To Burst or Not to Burst: Generating and Quantifying Improbable Text
Published in GEM @ EMNLP 2023, 2023
Recommended citation: To Burst or Not to Burst: Generating and Quantifying Improbable Text. K Sasse, E Kayi, S Barham, E Staley - GEM Workshop @ EMNLP, 2023. https://aclanthology.org/2023.gem-1.24/
While large language models (LLMs) are extremely capable at text generation, their outputs are still distinguishable from human-authored text. We explore this separation across many metrics over text, many sampling techniques, many types of text data, and across two popular LLMs, LLaMA and Vicuna. Along the way, we introduce a new metric, recoverability, to highlight differences between human and machine text; and we propose a new sampling technique, burst sampling, designed to close this gap.
Recommended citation: To Burst or Not to Burst: Generating and Quantifying Improbable Text. K Sasse, E Kayi, S Barham, E Staley - GEM Workshop @ EMNLP, 2023.