Federated Learning (FL) is highly useful for the applications which suffer silo effect and privacy preserving, such as healthcare, finance, education, etc. Existing FL approaches generally do not account for disparities in the quality of local data labels. However, the participants tend to suffer from label noise due to annotators’ varying skill-levels, biases or malicious tampering. In this chapter, we propose an alternative approach to address this challenge. It maintains a small set of benchmark samples on the FL coordinator and quantifies the credibility of the participants’ local data without directly observing them by computing the mutual cross-entropy between performance of the FL model on the local datasets and that of the participant’s local model on the benchmark dataset. Then, a credit-weighted orchestration is performed to adjust the weight assigned to participants in the FL model based on their credibility values. By experimentally evaluating on both synthetic data and real-world data, the results show that the proposed approach effectively identifies participants with noisy labels and reduces their impact on the FL model performance, thereby significantly outperforming existing FL approaches. © 2020, Springer Nature Switzerland AG.