Carlton is a city in Madison County, Georgia, United States. The population was 263 at the 2020 census.

An early variant name of Carlton was "Berkeley", to avoid confusion with Carrollton, Georgia. The Georgia General Assembly incorporated Carlton in 1892.

Carlton is located in southeastern Madison County. The city lies along former Georgia State Route 72, 5 miles (8 km) east of Comer and 12 miles (19 km) southwest of Elberton. The South Fork of the Broad River flows just south of Carlton, reaching the Broad River 3 miles (5 km) east of town. Watson Mill Bridge State Park, on the South Fork, is 2.5 miles (4.0 km) southwest of Carlton.

According to the United States Census Bureau, the city has a total area of 1.0 square mile (2.6 km), of which 0.003 square miles (1.9 acres), or 0.30%, are water.

Historical population

Census	Pop.	Note	%±
1900	277		—
1910	325		17.3%
1920	342		5.2%
1930	344		0.6%
1940	284		−17.4%
1950	249		−12.3%
1960	321		28.9%
1970	294		−8.4%
1980	291		−1.0%
1990	282		−3.1%
2000	233		−17.4%
2010	260		11.6%
2020	263		1.2%
U.S. Decennial Census

As of the census of 2000, there were 233 people, 100 households, and 65 families residing in the city. The population density was 232.3 inhabitants per square mile (89.7/km). There were 118 housing units at an average density of 117.6 per square mile (45.4/km). The racial makeup of the city was 74.25% White, 22.75% African American, 1.72% Asian, and 1.29% from two or more races. Hispanic or Latino of any race were 0.43% of the population.

There were 100 households, out of which 26.0% had children under the age of 18 living with them, 51.0% were married couples living together, 11.0% had a female householder with no husband present, and 35.0% were non-families. 31.0% of all households were made up of individuals, and 20.0% had someone living alone who was 65 years of age or older. The average household size was 2.33 and the average family size was 2.88.

In the city, the population was spread out, with 21.0% under the age of 18, 8.6% from 18 to 24, 25.3% from 25 to 44, 24.5% from 45 to 64, and 20.6% who were 65 years of age or older. The median age was 42 years. For every 100 females, there were 100.9 males. For every 100 females age 18 and over, there were 89.7 males.

The median income for a household in the city was $26,250, and the median income for a family was $29,583. Males had a median income of $23,750 versus $20,625 for females. The per capita income for the city was $16,303. About 14.3% of families and 19.6% of the population were below the poverty line, including 29.2% of those under the age of eighteen and 26.8% of those 65 or over.