Kelm's father was reportedly abusive, and within a few years of his birth, his parents divorced. Kelm's father was Jewish, and his mother was a Catholic German immigrant from Hamburg.
Arthur Andrew Kelm was born in Manhattan, New York City, the son of Gertrude ( née Gelien) and Charles Kelm.