There’s a new language model out. It’s trained on text from before 1931.
It doesn’t know about World War II. It doesn’t know about television. It has never heard the word “computer” in the modern sense. It knows Jazz Age America, the League of Nations, Model T Fords, and the silent horror of the Great Depression just beginning to bite. Meet talkie, a 13B parameter LM trained on 260 billion tokens of historical pre-1931 text.






