Remington was born in Canton, New York in 1861 to Seth Pierrepont Remington (1830–1880) and Clarissa “Clara” Bascom Sackrider (1836–1912). His paternal family owned hardware stores and emigrated from Alsace-Lorraine in the early 18th century. His maternal family of the Bascom line was of French Basque ancestry, coming to America in the early 1600s and founding Windsor, Connecticut. Remington’s father was a Union army officer, a colonel, in the American Civil War whose family had arrived in America from England in 1637. He was a newspaper editor and postmaster, and the family was active in local politics and staunchly Republican.