Small-scale gold mining is said to have started at Toi around 1370 during the period of the Ashikaga shogunate. The gold mine was operated on a large scale from the time of Tokugawa Ieyasu in the late 16th century. Several mines were open in 1577, but Tokugawa Ieyasu expanded production from 1601. He put the exploitation of the mine under the responsibility of a Kinzan Bugyō selected from the Ōkubo clan.

Toi was one of around 60 goldmines located in the Izu Peninsula, including Yugashima and Nawaji. The gold and silver produced by these mines permitted the production of Tokugawa coinage, and allowed for the prosperity of the Tokugawa. The village of Toi itself became highly prosperous, with numerous trades flooding in to service the workers and the administration at the gold mine, so that Toi became known as "Toi Sengen" (土肥千軒, "Toi of the 1,000 shops").

The mine became less productive as it became flooded. Workers were killed because of the exhausting conditions due to seeping hot springs and poor oxygen content of air, leading to the installation of water pumps and ventilators at numerous intervals.

In 1917, gold was again discovered at the mine, and exploitation continued under the company Toi Kinzan KK. In 1931, the mine entered Sumitomo Group, and passed under Toi Kōgyō KK in 1942. The mine was ultimately closed in 1965 and then reopened for tourism.

Toi was the second most productive gold mine in Japan, after the gold mine of Sado in Niigata Prefecture. During its period of exploitation, it produced in total 40 tons of gold and 400 tons of silver, whereas Sado produced as much as 80 tons of gold. One ton of rock would produce in average 5 to 10 grams of gold, although 30 grams ore was common, and some rock has yielded as much as 600 g of gold per ton.

The galleries of the mine total about 100 kilometers in length, over a surface of 37 hectares, and go as deep as 180 meters below sea level. The area visible for tourism is about 350m long, and goes about 150 meters deep into mountain rock.

The mine is now partially open for visits, and has become a tourist attraction. A "Shrine of the mine gods" (山神社) is visible inside the galleries.

The Toi Gold Museum (土肥黄金館) built nearby, describes the history of the mine and gold mining in Japan. The museum received some fame for housing the world's largest gold bar, weighing 250 kg, and representing a value of about $14.5 million in September 2022. The bar obtained an official Guinness record certificate for "The largest manufactured pure gold bar":

"The largest manufactured pure gold bar weighs 250 kg (551.150 lb) and was made by Mitsubishi Materials Corporation on 11 June 2005 at the Naoshima Smelter and Refinery, Kagawa Prefecture, Japan"

— Guinness World Records Certificate.