According to the highly specific genealogical volume "Scots and Scots' Descendants," the Bannerman family were a proud clan that claimed their surname from Bannockburn, "where an ancestor rescued the clan pennant, whereupon Bruce cut off the streamer from the Royal ensign and conferred upon him the honour of 'bannerman.'" The eldest son in each generation was named Frank, making things very simple (or impossible) for future researchers. Francis V brought him family to the United States in 1854, settling in Brooklyn. Francis VI left school at 10, when his father went to the Civil War in 1861. In addition to working in a law office and selling newspapers, young Frank dragged the river with a grapple for bits of chain and rope which he sold to junk dealers. His father returned from war disabled, and became a dealer in his son's collected materials, and this business grew into a ship chandlery.
In 1872, Francis VI started a new business, buying up at auction useful weapons and war materiel that were being auctioned for their scrap value. He started sending out an illustrated catalog for collectors, and also started converting rifles into fowling pieces for frontiersmen and Quaker guns for boys' brigades and military schools. His business grew with stores in New York City, where he fitted out many regiments for the Spanish-American War. At the conclusion of that bit of military theater, he bought up more than 90 percent of the captured war material and a little island in the Hudson Highlands called Pollopel's Island. (Spell it how you like.) The name was changed to Bannerman's Island, and here he erected arsenals "patterned after the Scottish baronial castles." The island became his summer home, and he directed the creation of an incredible castle, reportedly seeing to every detail.
In 1905, he bought 501 Broadway from the trustees of the Metropolitan Museum, who "greatly reduced the price in recognition o fhis maintenance of a Free Public War Museum." There were seven stories of museum and salesroom, containing weapons both ancient and modern. His terms were cash: "Even the Standard Oil Company in purchasing had to send check with order."
Bannerman became an acknowledged expert on weapons as well as perhaps the world's largest dealer in them; the author of his biographical sketch wanted it known that Bannerman "has consistently refused to sell to revolutionists, or to minors or irresponsible persons." He was also called "a great lover of boys," which had a distinctly different ring in a more innocent age, or at least one in which the boys were heavily armed.
Until recently, the name "Bannermans Island Arsenal" could be clearly seen along the eastern wall, but significant portions of the structure have collapsed in recent years. You can help support the stabilization and restoration of this incredible historic structure by voting in the Chase Community Giving process, http://www.bannermancastle.org/ or on Facebook.
If you're not familiar with it, there's a pool of pictures
of Bannerman's Castle on Flickr.