The first governor, Frederick North, 5th Earl of Guilford was the son of British Prime Minister, Frederick North (2nd Earl of Guilford). The house was planned by the governor himself, the building was later known as 'The Doric' due to the architectural design of the columns which was similar to the Ancient Greek Doric order style. It was built at the beginning of the nineteenth century (between 1801-1804) to revive and supervise the pearl fisheries. Other than being used as the residence for the governor, it was later used by "other governors, government agents, and other officials, including superintendents of pearl fishery."
The two story building was constructed using bricks and mortar though the exterior walls were decorated with chunam which was made from the lime of burnt oyster shells and was described as appearing like 'marble'.
Having been built on a low cliff near the beach, exposed to extreme weather and lack of maintenance, it is now mostly ruins. Restoration has been proposed several times but no work has been carried out despite it being declared a protected archaeological monument.
A number of unverified tales surround the site and it has been 'locally ascribed to a legendary Queen of the Mahabarata which refers to Allarani who was said to have a palace at the site. There are also unsubstantiated claims that the Portuguese built the Doric and it was used to protect Dona Catherina of Kandy circa 1580