Each member of an ad hoc advisory committee shall serve for such period as the Assistant Secretary may prescribe in his notice of appointment. Appointment of a member to the Committee for a fixed time period shall not affect the authority of the Secretary to remove, in his or her discretion, any member at any time. If a member resigns or is removed before his or her term expires, the Secretary of Labor may appoint a new member to serve for the remaining portion of the period prescribed in the notice appointing the original member of the committee.