The earliest existing historical records of the church date back to the year 1328, but it was not new that year. The original church site was along the shore, about 250 metres (820 ft) inland from the coast, just west of the village of Varhaug, about 3 kilometres (1.9 mi) from the present site of the church. In 1627, the old stave church was torn down and replaced with a new building on the same site.
In 1828, the church was torn down again and rebuilt, reusing many of the same materials from the old building. In 1904, a new Varhaug Church was built about 3 kilometres (1.9 mi) to the southeast of the old church. The new building was consecrated on 7 December 1904 and then the old church was torn down the following spring in 1905.
