Though Prospect Hill was not settled as a direct result of the Special Survey system, there were settlers in the area from as early as c1840. It was settled as a farming district by English, Scottish and Irish immigrants.

Prospect Hill was hit hard by the 1983 Ash Wednesday bushfires and lost many historical sites. Fortunately the former Post Office/General Store survived and is now an interesting and intriguing museum.