Fair Voice Biometrics: Impact of Demographic Imbalance on Group Fairness in Speaker Recognition