Any person aggrieved by the refusal of a building permit under this article may request a hearing on the matter by an appeals board. Except as hereinafter provided the governing body of the county is the appeals board for the county. If the governing body of any county determines that the number of hearings requested is so numerous that they cannot be handled expeditiously by such body, it shall appoint an appeals board to conduct such hearings.