Charter Oak Mills is a ghost town in the town of Albion, Jackson County, Wisconsin, United States. Charter Oak Mills was located along what is now Wisconsin Highway 54 2.1 miles (3.4 km) south-southwest of Black River Falls.