Castle Hill (Virginia) is an historic, privately owned, 600-acre (243 ha) plantation located at the foot of the Southwest Mountains in Albemarle County, Virginia, near Monticello and the city of Charlottesville, and is recognized by the Virginia Landmarks Register and the National Register of Historic Places. Castle Hill was the home of Dr. Thomas Walker (1715–1794) and his wife, Mildred Thornton Meriwether (widow of Nicholas Meriwether III). Walker was a close friend and the physician of Peter Jefferson, and later the guardian of young Thomas Jefferson after his father's death.

Through his marriage to Mildred Meriwether in 1741, Thomas Walker acquired the land comprising approximately 15,000 acres (6,100 ha) which would become the site for Castle Hill. In its square hall, the youthful, music-loving Jefferson once played the violin, while the still younger Madison danced. Here in 1781, Walker's wife delayed the British Colonel Banastre Tarleton to give the patriot Jack Jouett time to warn Governor Thomas Jefferson and the Virginia legislators of Tarleton's plan to capture them.

In addition to frequent visits by Thomas Jefferson, Castle Hill has entertained other U. S. Presidents and historic figures including George Washington, James Madison, James Monroe, Patrick Henry, Andrew Jackson, Robert E. Lee, James Buchanan, Martin Van Buren, John Tyler, Henry Clay, and Daniel Webster.

The Walkers' youngest son, Francis Walker (1764–1806), married Jane Byrd Nelson, the daughter of Governor Thomas Nelson of Yorktown, and inherited Castle Hill. The estate was next inherited by Thomas & Mildred Walker's granddaughter, Judith Page Walker (1802–1882), who married U. S. Senator William Cabell Rives (1793–1868). William Cabell Rives studied law under Thomas Jefferson at Monticello, and was a friend of James Madison. At Castle Hill, Rives wrote a three-volume biography on Madison, entitled The Life and Times of James Madison (Little Brown & Co., Boston, 1859, 1866, 1868). A close friend of Dolley Madison, Judith Rives authored the novel Home and the World (D. Appleton and Company, New York, 1857), in which she wrote of life at Castle Hill as the fictitious “Avonmore.”

Colonel Alfred Landon Rives (1830–1903), son of William and Judith Rives, and chief of engineers to General Robert E. Lee during the American Civil War, inherited Castle Hill. After Rives died in 1903, and his widow, Sarah Catherine MacMurdo Rives, died in 1909, their daughter, Amélie Louise Rives Troubetzkoy (1863–1945), prominent romantic novelist, early feminist, and wife of artist, Russian Prince Pierre Troubetzkoy (1864–1936), inherited the property. At least three of Amélie's novels, Virginia of Virginia (Harper & Brothers, New York, 1888), The Quick or the Dead? (J. B. Lippincott Co., Philadelphia, 1888), and Barbara Dering (J. B. Lippincott Co., Philadelphia, 1893), are drawn from life at Castle Hill. Amélie and Castle Hill were both featured in a New York Times article published on April 15, 1906.

In the early 20th century, Amélie's sister, Mrs. Gertrude Rives Potts, who managed the estate after their father's death, was recognized as the first woman Master of Foxhounds. While at Castle Hill, Gertrude imported and trained a pack of English Foxhounds, bred and schooled her own horses, organized a hunting staff, and enlisted the consent of neighboring landowners to form a suitable country for the “Castle Hill Hounds.” Castle Hill later became part of the Keswick Hunt Club district.

Castle Hill's history as a slave plantation was chronicled in Lorenzo Dickerson's documentaries The Coachman and Byrdland, about the Dickersons and Byrds, families enslaved at Castle Hill by several generations of the Walkers.

The original clapboard, colonial residence was built by Walker in 1764, with a front porch facing west and six dormer windows. William and Judith Rives added the brick, federal style addition to the home in 1824, which was built by Captain John Perry, one of Thomas Jefferson's master brickmasons. In 1844 the home's columned conservatories were added to each end of Perry's addition by another Jefferson brickmason, William B. Phillips.

Castle Hill's formal gardens were featured in an article written by Gertrude Potts published in the Historic Gardens of Virginia (The James River Garden Club, Richmond, 1923). The great box-hedges which stand at the top of the gardens are considered to be some of the tallest and most impressive boxwood in Virginia. The current expanded, multi-terraced and symmetrical formal gardens were designed in 1997 by Landscape architect Rachel M. Lily.

Castle Hill is a private residence.

• Thomas Walker (Explorer)
• William Cabell Rives
• Amélie Rives Troubetzkoy
• Julia Magruder

• K. Edward Lay, The Architecture of Jefferson Country (University Press of Virginia, Charlottesville and London, 2000)
• Richard Channing Moore Page, M.D., Genealogy of the Page Family in Virginia (C. J. Carrier Company, Bridgewater, Virginia, 1965)
• James Rives Childs, Reliques of the Rives (Reprint, Heritage Books, 2006)
• Welford Dunaway Taylor, Amélie Rives (Princess Troubetzkoy) (Twayne Publishers, New York, 1973)

• "A Week's End Visit with Amélie Rives," New York Times Article, April 15, 1906
• Journey Through Hallowed Ground, "Castle Hill", National Register of Historic Places
• Nature Conservancy Virginia, "Historic Castle Hill Preserved"