The 1651 purchase of Shelter Island included all of its 8,000 acres. Over time, as land was distributed to heirs, sold to newcomers, and needs shifted, the property gradually diminished in size and scope. What now remains is the 243-acre former plantation core, encompassing farm fields, wet and wood lands, formal gardens and grass lawns, numerous barns and buildings, and the Manor house.
In addition to the central structures at the heart of the property, the extended Manor grounds hold a wealth of diverse cultural inventories that directly link the land with social and ethnic interactions, changes in agricultural and commercial use of the property, religious persecution and tolerance, racial inequities, and how these factors collectively impacted the lives of people living and working at the Manor. Their stories live in the land.
On these grounds are the remains of ancient hunter-gatherer Manhansett Indian settlements, predating European arrival by at least one thousand years. A short distance to the south of the residence is a small rises of land between the driveway forks, where by tradition as many as 200 enslaved servants and laborers are buried. The burial ground lies amidst a huge stand of white pine trees that were planted at the turn of the last century, when it was believed that their clean scent filtered the air and protected against some diseases.
To the west near the creek lies another Manor cemetery, commemorated by Cornelia Horsford in 1884 to all residents of the property over the centuries, including several persecuted Quakers who found refuge with the Sylvesters in the 17th century. There are no headstones for any of the founding members of the Sylvester family here. Today, the site is the warm-weather meeting grounds for Shelter Island chapter of The Society of Friends.
The property has always revolved around food and land. The fields of the Manor have been in agricultural use for at least 360 years, cultivated for plantation, commercial and subsistence production. The “Watermelon Patch”, located to the north of the Manor House, is believed to be the first field put to culture in the 17th century. At one time, the Sylvester fields spread north toward Dering Harbor, east to Coecles Harbor, and south to the Menantic area.