The August 30, 1927 New Yorker is trying to convince me that the term "buff" -- as in "enthusiast" -- came about in 1865, when New York's first paid fire department was organized. The older volunteers who USED to fight the fires still wanted to help out, so they'd sleep overnight in the fire houses, waiting for a chance to rush off with the official firemen and help put out fires.
Since the fire department wouldn't give them any bedding, the fire enthusiasts slept in heavy buffalo robes...hence "buff" as a term for somebody who is enthusiastic about something (eg. "film buff").
This sounded like a load of bunk to me, but it appears (in my short pre-work researches) to be pretty much true.