Ted Underwood in Elk: "@andrewpiper @grvsmth @dh Will I get ..."

Hypothesis:

are going to make the distinction between "distant" and "close" reading pretty much moot in 5 yrs, by a) reducing barriers to entry for large-scale analysis and b) making it possible to do fun and revealing computational things at a much smaller scale, even with slippery questions about character and plot. @dh

@TedUnderwood @dh This is not a testable hypothesis, it's a prediction about future events that are only going to happen once.

Still, they just aren't. Models are tools for communicating. LLMs are horrible at that, because they tell you basically nothing useful about the language sample you've applied them to.

I've been disagreeing with your takes on these things for years now and haven't changed your mind, Ted, so I don't expect to now, but I want to be on record as disagreeing.

@grvsmth @TedUnderwood @dh

curious how you see this playing out. Doing analytical things using natural language? Curious how you would flesh out LLMs for analytical purposes.

I'm, uh, working on it? Probably some mixture of natural language questions about texts and good old statistics on the results. I haven't figured it out yet.

Needless to say, we'll need to validate the models. But, you know, that's what we already do with human readers; we call it inter-rater reliability.

@TedUnderwood @andrewpiper What is the value of adding another non-deterministic evaluator? @dh

@grvsmth @andrewpiper @dh

Well, for instance, I can ask an LLM to extract the motive of the killer in 10,000 detective stories. It is rather hard to do that with corpus linguistics.

@andrewpiper @grvsmth @dh

Will I get consistent results? Will they tell us something interesting? I don’t know yet; that’s what it means to experiment.

January 14, 2023 at 1:33:38 AM

@TedUnderwood @andrewpiper @dh

LLMs are notoriously unreliable on a factual level, so I doubt you'll get consistent results.

"Interesting" is notoriously subjective, so I'm not going to pull a Chomsky on you!