Bias in Federated Learning: Factors, Effects, Mitigations, and Open Issues

Machine Learning (ML) has become pervasive across domains, powering services from image recognition to personalized recommendations. The success of ML critically depends on access to massive datasets that fuel model development. However, aggregation of large centralized datasets poses significant privacy concerns and security risks [1]. Federated learning (FL) is a distributed collaborative learning paradigm introduced to address this limitation.

In FL, data remains decentralized on the client devices like mobile phones or hospital servers. A shared global model is trained collectively without direct access to raw private data [2]. The model training process involves multiple rounds of communication between the clients and a central aggregation server [3]. In each round, the server first broadcasts the current state of the global model to a subset of available clients. Each client then performs local computation on this model using their private local dataset to generate model parameter updates. Only these update vectors are shared with the server which are aggregated to improve the global model. After several rounds of this federated training loop, the model converges to an optimal solution trained on the collective data [4, 5].

FL provides multiple advantages compared to ML:

• Enhanced privacy and confidentiality as raw data remains decentralized.
• Reduced security risks from single point of failure.
• Compliance with regulations on control of user data.
• Mitigates data silos problem across different entities.
• Ability to leverage scattered, non-Independent and Identically Distributed (non-IID) data located on endpoints.

This has enabled deployment of FL across diverse applications like next word prediction on smartphones [6], disease detection in healthcare [7], fraud detection in finance [8] and more. Leading technology companies like Google, Apple, Meta, Microsoft, Uber and IBM have incorporated FL into products and services [2]. The decentralized nature of FL also makes it suitable for emerging paradigms like edge computing which push intelligence to the network edge [9].

However, FL introduces statistical and systems challenges compared to centralized ML [5]:

• Non-IID, unbalanced and sparse local client data.
• Systems heterogeneity in compute capabilities of clients.
• Threat of inference attacks and privacy leaks.
• Communication overhead and convergence issues.
• Misaligned incentives between competitive clients.

These unique constraints impact the collaborative learning and can introduce biases that lead to discrimination and unfairness issues [10]. Recent studies have empirically shown that ignoring the heterogeneous decentralized distributions in FL can lead to suboptimal accuracy and algorithmic harms against certain subgroups [11, 12].

There are growing societal concerns around potential biases encoded in AI systems [13]. Bias refers to any systematic error which can lead to unfairness, discrimination and suboptimal model performance on certain subgroups [14]. Biases manifest due to historical prejudices, representation imbalance a systemic inequity ingrained in the data get propagated into the algorithms, leading to discriminatory predictions and decisions [15]. Some sources of bias specific to FL include:

• Sample selection bias: The local datasets available to different FL clients may not represent the true population distribution. This can introduce sampling bias against minorities who are underrepresented. Location biases are common as the data may overrepresent certain geographic regions based on app usage.
• Population distribution shift: The demographics and data distributions often drift over time and vary across clients. Models trained on past biased data may not generalize fairly to new user populations [16].
• Biased local data: The local data itself can contain systemic biases and discrimination against protected groups which get encoded into the models [11]. For instance, differential judgement and labeling of minority groups.
• Algorithmic biases: The objective functions and model architectures may implicitly embed assumptions that disadvantage certain subgroups. Optimizing accuracy often comes at the cost of fairness [17].
• Systemic societal biases: Long-standing structural inequalities along gender, racial and socioeconomic divides manifest as background statistical biases [14].

These biases accumulate during training and get propagated to the global model. Ignoring fairness can exacerbate harms against already marginalized communities. This underscores the critical need for tackling biases in FL to prevent exclusions and ensure equitable access to the technology benefits. However, mitigating biases is significantly more complex in federated settings compared to centralized ML due to constraints around visibility into local client distributions, coordination between distrusting entities, systems heterogeneity, and misaligned incentives [2]. Care must be taken to balance accuracy, fairness and privacy tradeoffs given the decentralized nature of FL [14]. Therefore, tailored solutions are required that account for the unique challenges.

In this paper, we provide a comprehensive analysis of the various sources of bias that can manifest in FL systems and examine mitigation strategies tailored to the decentralized environment. We categorize techniques based on principles such as robust aggregation algorithms, cryptographic protocols, algorithmic and data debiasing, personalized modeling, and participative evaluation. We also discuss key open challenges around miscoordination, privacy constraints, security, systems heterogeneity, incentives, governance, and participation that remain to be addressed to realize trustworthy and fair FL.

The key contributions of this work are:

• A taxonomy of bias sources unique to FL including sample selection bias, population drift, locally biased data, societal biases, algorithmic biases, and representational biases.
• Categorization of bias mitigation techniques tailored for FL based on principles like robust aggregation algorithms, cryptographic protocols, data and algorithmic debiasing, personalized modeling, and participative evaluation.
• Discussion of open research challenges around miscoordination, privacy, systems heterogeneity, incentives, governance, and participation that remain to be tackled.

The rest of this paper is organized as follows: Section 2 provides background on FL, contrasting it with centralized ML and discussing architectures, algorithms, applications, and challenges. Section 3 analyzes sources of bias in FL and their effects. Section 4 examines bias mitigation techniques in FL, categorizing them into data-based, algorithmic, and hybrid approaches. Section 5 examines key open research issues and future directions around developing fair and accountable FL systems. Finally, Section 6 concludes with a summary and outlook on bias mitigation for trustworthy FL.

5.png

Sample selection bias can arise in FL due to differences in participation and selection of clients for training rounds [69]. The decentralized datasets in FL are determined by which users, devices or organizations participate as clients in the model training process. However, the client population may not accurately represent the true underlying population distribution [70]. Certain subgroups may end up being overrepresented or underrepresented among the clients based on factors like geographic location, demographics, device types, etc. [69]. For example, patients from larger hospitals may dominate in healthcare FL while smaller clinics are underrepresented [7]. This can lead to biased, non-representative data distributions among the decentralized client datasets. Minority demographic groups and less dominant data patterns may get excluded or underrepresented. Overrepresented groups will have an outsized influence on the model compared to underrepresented groups. As a consequence, the global model may not sufficiently capture diverse perspectives and vulnerabilities, potentially resulting in discrimination against certain minorities excluded from the training process. Furthermore, the model may also not generalize well to underrepresented segments of the population [71]. Without corrective techniques, this sample selection bias can get imprinted into the global model during federated training [72].

For instance, in Google's implementation of FL for Gboard, the mobile keyboard app, sample selection bias was observed due to uneven participation of users. Users with high-end devices and reliable internet connections were more likely to participate in training rounds, leading to overrepresentation of certain demographic groups. This resulted in the language model performing better for these groups while underperforming for underrepresented demographics. The bias manifested as less accurate next-word predictions for users from underrepresented groups, affecting user experience and potentially widening the digital divide.

3.1.1 Factors

There are several factors that can skew the sampling of clients in FL:

• Device and Connectivity Biases: Clients participate voluntarily based on device capabilities and connectivity. Lower-resourced devices may drop out due to power or bandwidth limitations [3]. Regions with poor connectivity provide limited client samples. This leads to biased geographic and demographic representations.
• User Demographic Biases: Due to variability in technology adoption, the decentralized samples tend to overrepresent certain population segments more proficient with technology, while underrepresenting minorities [21]. Age, income, education and cultural gaps can skew the client population.
• Adversarial Manipulation: Malicious entities can deliberately poison or manipulate client sampling through targeted device infections or Sybil attacks simulating fake clients [5]. This allows injecting biases by influencing which devices participate in FL.

3.1.2 Effects

These sampling biases lead to some groups being overrepresented while minorities are underrepresented among the clients. This has several detrimental effects [2]:

• Model fairness can degrade for underrepresented or excluded groups. Predictions tend to favor majority demographics.
• Biases against minorities and protected groups are amplified without diversity in sampling.
• Groups not represented in the client sample can be negatively affected by model decisions.
• There is a lack of visibility into client sample representativeness at the central server.

3.1.3 Potential mitigations

Addressing sample selection bias remains an active area of research in FL. A range of techniques have been proposed to detect, limit and correct for sampling biases in the decentralized client population:

• Careful client selection and recruitment strategies can help improve coverage of the underlying population distribution. The server can selectively recruit new clients to improve diversity on dimensions like geographic location, device types and demographics [73]. Incentive mechanisms through rewards or service benefits can also encourage broader user participation [74, 75]. However, this requires additional coordination overhead.
• Robust aggregation algorithms help limit the influence of any manipulated or biased local updates on the global model [16]. Techniques like trimmed mean ignore extreme outlier updates during aggregation [76]. The Federated Averaging algorithm assigns weights to clients based on characteristics like data size and reliability. This reduces the impact of falsified model contributions.
• Statistical bias detection techniques can diagnose sampling issues before launching each training round [77]. Exploratory analysis of client characteristics helps identify underrepresented groups. Proxy metrics can be derived to estimate representation biases. This enables corrective actions like targeted recruitment.
• Differential privacy mechanisms add noise to updates before aggregation to cloak client identities [78]. This limits reconstructing sensitive attributes that could enable manipulating client samples. Cryptographic secure aggregation also prevents leaking client properties [1].

While these approaches help mitigate sample selection bias, fully addressing the root causes requires expanding decentralized participation. Better incentives, user engagement and representativeness metrics can enhance diversity over time.

3.2 Population distribution shift

The independent datasets distributed across clients in FL are not static over time. There can be significant drift in the underlying data distributions as user populations and behaviors evolve [79, 80]. For example, demographics like age groups, language preferences, cultural affiliations, etc. can change across geographic regions that are represented by different clients [1].

New trends and emerging use cases lead to shifts in usage patterns and data characteristics. The interests and needs of users may also vary over time. In healthcare, new patient groups and disorders can arise while incidence of existing diseases may decline [7]. Such changes can lead to the problem of concept drift, where models trained on past client data distributions do not generalize well to new emerging distributions [81, 82]. Historical biases can get entrenched into the models without accounting for shifting populations and trends over time. As an illustration, consider banks training fraud detection models using FL across their branches. If the model was trained only on historical data, it may not catch new fraud patterns arising at a faster rate in certain newer regions [83]. Without explicit retraining or adaptation, model performance can degrade rapidly.

3.2.1 Factors

Several factors can cause shifts in the decentralized distributions that client devices see:

• User Demographic Shifts: Populations inherently change as new user groups emerge and old cohorts decline. For example, generational differences lead to varying app usage patterns [84]. Geographic migration also causes distribution drifts.
• User Behavior Shifts: Interests, habits and needs of users evolve over time. Social trends lead to changing how apps and devices are utilized. For instance, growth of short-form video apps like TikTok [85].
• App and Device Version Shifts: As apps get updated with new features, the resulting data patterns change. New device models alter data types, apps, and usage. Software and hardware evolution leads to drift [86].
• Adversarial Drift Injection: Attackers can manipulate the local data or model updates to intentionally cause distribution shifts that degrade model performance over time [87]. They may target particular subpopulations.

3.2.2 Effects

Unchecked, such drift can lead to models becoming stale and inaccurate on new data distributions, even if they were robust when first deployed [88]. Two key effects include:

• Overall Performance Degradation: The global model starts making increasingly erroneous predictions as its assumptions become outdated [89]. Metrics like accuracy drop without realizing data has changed.
• Subgroup Performance Issues: Model quality deteriorates rapidly on segments of users exhibiting the most distribution drift without being detected [90]. Minority groups often suffer the most.

3.2.3 Potential mitigations

As user behaviors, environments, and systems evolve in FL, adaptive solutions are necessary to detect and respond to concept drift across decentralized devices [91-93]:

• Local drift detection techniques can analyze data distributions before aggregation to identify shifts from historical baselines [94]. Devices flag significant deviations to the central server prompting retraining. However, drift invisible to individual clients can accumulate globally.
• Personalized FL customizes models to adapt to changing local distributions [95]. This allows specializing in emerging user cohorts and data types. However, personalization risks fragmenting the global model.
• Robust optimization methods like distributionally robust optimization assume worst-case distributions to maintain performance despite drift [96]. Model rigor comes at the cost of lower accuracy on stable distributions.
• Continuous aggregated analytics helps detect distribution shifts globally by monitoring metrics like decreasing accuracy, increasing loss, and higher variance in updates [97]. Retraining can then recalibrate models.
• Simulating anticipated drifts via synthetic data allows proactively adapting models [98]. But this requires resources to generate future data profiles. And unexpected shifts may still occur.
• Hybrid cloud-federated architectures utilize centralized cloud resources to rapidly retrain models when unmanageable drift is detected [99]. But this partially compromises privacy.
• Adversarial domain adaptation explicitly trains models to adapt to different distributions, enhancing generalization [94]. But assumptions of shiftable domains are required.
• Fully tackling decentralized drift requires coordination frameworks to align client objectives, server oversight to diagnose drift, and configurable robustness against uncertainty [16].

3.3 Biased local data

In FL, biases can originate from the local datasets held by the clients themselves even before training begins [100]. The data collection and annotation process may be skewed against certain protected groups leading to underrepresentation or measurement bias.

For instance, a study by Kaissis et al. [101] demonstrated that FL models for medical imaging inherited biases from local datasets. Hospitals with more advanced imaging equipment contributed higher-quality data, while those with older equipment supplied lower-quality images. This disparity led to a model that performed better on data similar to that from hospitals with advanced equipment, disadvantaging patients from under-resourced hospitals.

3.3.1 Factors

There are several factors that can introduce bias into the decentralized data:

• Biased Data Collection: The data collection process may systemically underrepresent certain population groups. For example, clinical trials often focus on majority demographics while excluding minorities [102].
• Measurement Biases: Systemic issues in how data is measured and annotated can lead to bias. Examples include label skew, stereotypes in human labeling, and distortion in self-reported data [103].
• Omitted Variables: Relevant factors needed to make fair decisions may be missing from the data. Sensitive attributes like race, age or gender might be excluded.
• Proxy Encoding: Even if sensitive attributes are excluded, other features may encode similar information leading to indirect bias [104]. Location or income data could correlate to race for instance.

The local datasets may also disproportionately represent and further amplify existing societal biases [104]. Discrimination faced by marginalized communities propagates into the data. Seemingly objective data can perpetuate systemic inequities. If such issues in the decentralized data are not addressed, the biases will naturally get propagated to the global model during federated training and aggregation [11].

3.3.2 Effects

If left unaddressed, these biases in local data can get propagated to the global model during federated training. Biased data also makes models vulnerable to deliberate manipulation:

• Data Poisoning Attacks: Bad actors can inject poisoned local data with backdoors or malevolent biases that taint the global model [105]. This is harder to detect in decentralized models.
• Model Evasion: Biased data can be used to generate adversarial examples to evade detection by models like fraud classifiers [106].
• Difficult to Audit: Due to privacy requirements, auditing local data quality directly is difficult in FL, allowing bad data to go undetected.

Biased local data can thus lead to loss of fairness, performance issues, and security vulnerabilities. But mitigating decentralized data bias raises challenges around coordination, privacy, and incentive alignment between competitive clients [2].

3.3.3 Potential mitigations

Addressing biases in the decentralized datasets of FL clients poses unique challenges around preserving privacy and securely coordinating among untrusting parties. A range of techniques have been proposed to detect, limit, and correct for biases in local client data:

• Client-Side Data Auditing: Adapting centralized dataset auditing tools for decentralized execution allows clients to analyze their local data for biases and coverage issues before training models [107]. Privacy-preserving metrics quantify label skew, profiling gaps, stereotypes in annotations, proxy discrimination through related variables, and other data quality issues without requiring to share actual data [108]. Aggregated reports allow assessment of biases across the federated network to prioritize improvements.
• Local Data Debiasing: Similar to centralized debiasing, privacy-preserving algorithms can pre-process local datasets to remove biases while preserving utility [109]. Removal or anonymization of sensitive variables, reweighting, resampling, and generative data augmentation changes balance representations before training. Debiasing also makes models more robust to data poisoning attacks.
• Secure Aggregation for Bias: Using secure multiparty computation and differential privacy, aggregation queries can be run across decentralized clients to detect outliers indicative of manipulated or intentionally poisoned local data [78, 110]. Noise addition during aggregation provides privacy for such diagnostics. Data trusts which hold local data encrypted can also run validation.
• Reputation Systems: Central servers can track metrics like variance between updates from a client, prediction errors on new data, anomalies in parameters, etc. to build reputation scores for reliable clients over time [105]. Biased or manipulated data contributions then get lower weightage during aggregation.
• Incentive Mechanisms: Proper incentives that reward good behavior can help improve data quality [111-113]. For example, well-behaved clients who provide useful updates are paid or provided higher model quality. Penalties can deter detectable manipulations and data poisoning.
• Robust Secure Aggregation: Carefully designed robust federated algorithms ignore or down-weight extreme parameter updates and give higher priority to trusted clients [76, 114]. This limits the impact of intentionally poisoned data contributions, but may overlook valid outliers.

While these help in mitigating biased local data, approaches to expand diversity and inclusion in data collection are also needed to address systemic root causes. Careful alignment of incentives, coordination and education can enhance data quality over time.

3.4 Systemic biases in data

The data used to train ML models often reflects and amplifies systemic societal biases that have persisted historically against certain groups [14]. Even datasets that appear neutral on the surface can propagate unfair social prejudices if not carefully examined [115]. For instance, healthcare data has been shown to contain inherent biases that disadvantage ethnic minorities. Clinical studies disproportionately focus on majority populations, excluding historically marginalized groups like racial minorities and poorer socioeconomic segments [116]. Predictive models trained on such data perpetuate the sampling and coverage biases, leading to inequitable healthcare access and outcomes for minorities. Further, face recognition systems have higher error rates for darker skin tones due to unbalanced training data [117]. Furthermore, finance data also reflects systemic biases in lending practices and income disparities across demographic factors like gender and race. Models built on such data can deny opportunities to minorities by repeating historical discrimination [118].

3.4.1 Factors

There are several factors that allow systemic biases to manifest in data:

• Historical Discrimination: Past discrimination faced by groups like racial minorities due to unethical laws, policies and practices over decades gets captured in data documenting those eras [119, 120]. Datasets thus bake in historical harms.
• Measurement Biases: The practices and tools used for data collection may systemically underrepresent certain population segments leading to sampling bias [121]. Surveys that fail to sample minorities, sensors unavailable in low-income regions, human annotation from majority demographics, etc. cause systemic data collection biases.
• Proxy Discrimination: Even when sensitive attributes like race, gender or age are explicitly excluded, other superficial attributes may correlate to them enabling indirect systemic biases [122]. Geography, income, language, profession and other factors can act as proxies.
• Anchor Biases: When labeling training data or defining categories, majority priorities and contexts are unconsciously anchored on leading to biases against minorities with different needs [123].
• Social Stereotypes: Human cognitive biases and prevailing societal prejudices unconsciously get imprinted into data [124, 125]. Cultural stereotypes around race, gender, age, ethnicity, etc. distort human annotation of training data as well as user-generated data.
• Unexamined Assumptions: Design choices that ignore minority interests perpetuate the status quo. Questionnaire design, feature selection, problem formalization and other assumptions disadvantage minorities [126-128].

3.4.2 Effects

When applied to data reflecting systemic biases, machine learning models inherit these issues which then get amplified due to feedback loops:

• Underrepresentation: Minority groups most impacted by systemic biases have insufficient data leading to outcomes optimized for the majority groups [129]. Their interests are excluded.
• Measurement and Feature Bias: Systemically distorted data provides inaccurate ground truth for modeling minority behavior leading to skewed models, and sensitive attributes may be proxied [14]. Moreover, Models latch onto features correlated with identity rather than meaningful drivers leading to proxy discrimination [130].
• Objective Bias: Model objectives like accuracy optimize for the majority groups at the cost of minorities by assuming balanced data [131].
• Miscalibration: Even when overall metrics seem robust, performance on minority groups suffers due to systemic underrepresentation and measurement biases [129].

While FL keeps the sensitive raw data decentralized, the models can still ingest systemic biases present in the local datasets. This can lead to scenarios where the AI systems exclude, misrepresent or disproportionately target groups suffering from structural marginalization [132]. Unless countermeasures are taken, the aggregated models will reflect the accumulated societal biases. Techniques to audit datasets and algorithms as well as incentivize equitable engagement are necessary to mitigate harm from historically ingrained biases [133]. The compounding effects of minor data imbalances also need to be considered.

3.4.3 Potential mitigations

Addressing systemic biases that have accumulated in data requires a multifaceted approach:

• Decentralized Bias Auditing: Tools adapted for decentralized execution can allow clients to audit local data and coordinate audits on aggregated data to quantify bias and identify impacted groups [108]. Privacy-preserving metrics assess representation imbalances, distortion, profiling harms.
• Dataset Nutrition Labels: Attaching documentation detailing known gaps, assumptions, reporting subgroups, etc. provides transparency into limitations to address in system design. Data statements enable informed usage.
• Dataset Augmentation: Synthetically generating additional data samples from minority groups can improve coverage [134]. However, measuring impact on bias needs care.
• Re-weighting Data: Assigning higher weights to minority samples during model training can rebalance contributions [135]. But directly optimizing on biased data has risks.
• Debiasing Algorithms: Algorithms that pre-process data to remove proxy variables, reweight groups, and normalize representations can be applied locally before federated training [136, 137].
• Inclusive Data Collection: Expanding diversity in data collection practices to cover beyond majority demographics, languages, contexts, etc. provides more representative data.
• Regulations and Standards: Requiring standardized reporting, impact assessments, risk management and other oversight measures allows governing use of systemically biased data [138].

The unique constraints of FL require adapting anti-bias approaches to the decentralized environment. Key technical interventions include improving local data quality, algorithmic debiasing, and rigorous subgroup validation. However, technical steps alone are insufficient to address systemic societal issues. Participative auditing, representation in governance, ensuring accountability, and reforming unjust structures are imperative [139].

3.5 Social biases

ML systems do not operate in isolation, but rather reflect prevailing societal attitudes and ingrained human prejudices [140]. Data generated by humans naturally captures embedded cultural stereotypes and unconscious biases around factors like race, gender, age, ethnicity, etc. [141]. When datasets exhibiting social biases are used to train AI systems, the models inherit and amplify these biases. Seemingly neutral factors can encode demographic attributes leading to proxy discrimination [142].

3.5.1 Factors

There are various complex sociotechnical factors that allow social biases to become ingrained in data and algorithms [143]:

• Historical Discrimination: Legacies of unethical laws, policies, practices, and social norms against minority groups in the past propagate biases. Discriminatory practices get captured in historical datasets documenting those eras [119, 120]. Models trained on such data perpetuate historical prejudices.
• Representation Imbalances: Due to gaps in access, awareness, and technical skills, certain demographic segments end up underrepresented in datasets used to train AI systems. Without diversity, systems optimize for majority groups [129].
• Social Stereotypes: Human cognitive biases and prevailing cultural stereotypes unconsciously get imprinted into data generated by people. Annotators apply societal prejudices when labeling training data, which gets propagated into models [124].
• Anchor Biases: When defining problems, choosing categories, and labeling data, majority priorities, contexts and needs are anchored onto. Minority interests are overlooked, leading to biases against them.
• Implicit Associations: Models pick up on correlations in data that reflect social stereotypes and societal associations, even if not present in the variables directly used for training [144]. It get imprinted into models.
• Feedback Loops: Due to self-reinforcing cycles, minor biases accumulate and get amplified over time. Biased models produce skewed outputs which further distort data used for retraining [145].

3.5.2 Effect

Unchecked social biases perpetuated through data and algorithms lead to the following discriminatory impacts [143]:

• Underrepresentation: Minority groups left out of training data receive lower quality outcomes as models never learn to optimize for them. Their interests are excluded.
• Feature Biases: Models rely on factors correlating with sensitive attributes like race or gender rather than meaningful drivers of decisions. This leads to proxy discrimination.
• Poor Generalization: Performance disparities arise for minority dialects, cultural contexts, age groups, etc. that differ from majority training data.
• Stereotyping: Recommendations align with and reinforce stereotypes around race, gender, age or ethnicity leading to pigeonholing.
• Exclusion: Requirements like video interviews, majority cultural settings, etc. disadvantage subgroups with less access to technology and dominant cultures.
• Denial of Opportunities: Biased models result in fewer opportunities in areas like credit, employment, housing, etc. for impacted groups.
• Abusive Targeting: Models explicitly target minorities for predatory practices like aggressive policing, high interest loans, or addiction promotion.
• Loss of Autonomy: Biased systems override minority self-determination by encoding majority priorities and perspectives.

Social biases in FL lead to disproportionate errors, exclusion or problematic recommendations against protected groups facing structural inequities [146-148]. Without concerted efforts, federated models will encode accumulated societal prejudices leading to discriminatory impacts.

3.5.3 Potential mitigations

Addressing social bias remains an active area of research in FL requiring a multifaceted approach:

• Diverse Clients: Promoting participation from a diverse and representative range of users, devices, and organizations helps create more varied decentralized data to train on.
• Bias Auditing: Adapting bias quantification tools to run locally allows clients to audit their data. Secure aggregation enables collective auditing to identify systemic issues [108, 149].
• Data Augmentation: Generative models can be used to synthesize new data samples from underrepresented groups, locally improving data diversity [150].
• Dataset Nutrition Labels: Attaching documentation about known data biases improves transparency for those aggregating models [151].
• Algorithmic Fairness: Regularization terms and constraints are incorporated into the global model optimization to encourage fairness across groups [152].
• Subgroup Validation: Maintaining segmented test sets for evaluating model performance on minority groups allows detecting disparities [153].

3.6 Algorithmic biases

In addition to data issues, biases can also arise from the model architectures, objective functions, and assumptions made during the ML pipeline [14]. Choices that seem neutral can unintentionally introduce algorithmic harms against certain groups. For instance, commonly used performance metrics like accuracy implicitly assume class balance and can optimize for the majority groups, disadvantaging minorities [124]. Maximizing accuracy leads models to disproportionately focus on improving predictions for well-represented groups.

For instance, an implementation of FL for music recommendation revealed algorithmic bias due to the optimization objective favoring majority user preferences [154]. The model prioritized genres favored by the dominant user groups in the training data, underrepresenting music genres preferred by minority users.

3.6.1 Factors

There are various ways that bias can inadvertently become encoded into the algorithms and models themselves:

• Model objectives like accuracy and Root Mean Square Error (RMSE) optimize for overall aggregate performance but ignore effects on subgroups. Maximizing accuracy leads systems to focus on improving predictions for well-represented majority groups at the cost of minorities [131]. Classifier performance metrics assume balanced data and proportional subgroups.
• Many regularization techniques to prevent overfitting like parameter norm penalties are optimized for centralized settings with abundant IID data. They do not sufficiently account for the decentralized unbalanced distributions often present in FL [54]. This degrades model robustness and fairness on underrepresented groups.
• The choice of model family, architectures and hyperparameters reflects inherent assumptions that may not universally hold across diverse subgroups. Different subgroups may require different model forms to capture nuanced data relationships.
• Deep learning models can latch onto spurious correlations during training that reflect historical biases and discrimination rather than meaningful drivers of decisions [155]. The black box nature of complex models makes auditing these latent biases difficult.
• During federated training, the process of aggregating algorithmic biases from various decentralized clients can compound harms. Differing optimization constraints and incentives between competitive clients further complicate the effects [5].

3.6.2 Effect

Unchecked algorithmic biases can result in the following discriminatory impacts on minority groups:

• Suboptimal performance due to poor generalization on data patterns not fitting simplified modeling assumptions based on majority trends [129]. Tailored model selection is needed to address subgroup needs.
• Perpetuating historical harms by replicating ingrained patterns of discrimination that correlate superficially in the data but have no true explanatory relationship [156].
• Denying opportunities by optimizing only for the overall metric improvements which enables sacrificing minority subgroups if it benefits the majority aggregate.
• Violating privacy and self-determination by enabling inference of sensitive attributes from model outputs even if not an explicit input [157]. Individuals lose control over use of their personal information.
• Avoiding accountability by providing ambiguous opaque systems that deflect responsibility for unfair outcomes by attributing them to algorithmic determinism [158].

Unless algorithmic biases are mitigated through thoughtful selection of performance metrics, model forms and training objectives, FL risks exacerbating discrimination through its algorithms.

3.6.3 Potential mitigations

While data biases entering models can be addressed through preprocessing and augmentation techniques, biases can also arise from the model development process itself. From unfair performance metrics to poor generalizability on minority data patterns, a range of technical choices can inadvertently introduce algorithmic harms:

• Subgroup Validation: Maintain segmented test sets for each demographic group to evaluate model performance across subgroups and detect disparities [153, 159].
• Fairness Regularization: Incorporate constraints or terms in the global model optimization that penalize discrimination and encourage fairness across groups [160, 161].
• Flexible Model Selection: Use adaptive model selection and hyperparameter tuning tailored to optimize performance for individual clients to improve personalization [95].
• Multimodal Modeling: Ensemble approaches combining diverse model families tuned on representations learned from subset data can improve robustness.
• Counterfactual Evaluation: Assess models by systematically simulating conditions with biases removed to quantify impact on subgroups [162].
• Modular Transparency: Adopt interpretable, modular and auditable model architectures to enable better oversight [163].

While technical interventions help, addressing algorithmic biases requires examining how problems are formulated, performance is measured, and who is centered in development. Wider community participation in designing and auditing algorithms can surface harmful assumptions [164].

3.7 Representational biases

Representational biases refer to issues that arise from how data is structured, problems formalized, and models designed in ways that marginalize certain populations [115, 158]. Choices that may seem neutral can inadvertently encode assumptions that disadvantage minority groups. For example, the way data is formatted often normalizes attributes common in majority demographics while minorities end up represented as edge cases or exceptions [159].

For instance, FL models trained on text data from globally distributed users may underrepresent low-resource languages [165, 166]. Users typing in less common languages contribute less to the model updates, leading to poorer language processing capabilities for those languages.

3.7.1 Factors

There are various ways that representational biases can manifest in the FL pipeline:

• Client Data Formatting: The way local decentralized data is preprocessed, formatted and encoded often normalizes attributes common in majority demographics while minorities are edge cases [14].
• Problem Framing: Objectives set by aggregators often ignore needs of underrepresented groups. Narrow assumptions result in poor personalization [151].
• Global Model Design: Choices in model family, architecture, hyperparameters may be suboptimal for tailoring to diverse clients’ data patterns [115].
• Evaluation Metrics: Global validation datasets and metrics like overall accuracy insufficient to assess subgroup impacts [167].

Together these representational choices by central aggregators can center majority groups while marginalizing minorities in the federated environment. This leads to embedding biases that disadvantage subgroups among the decentralized clients.

3.7.2 Effect

Representational biases manifest in FL models through the following effects:

• Poor Personalization: Global models fail to specialize well for minority clients with atypical data distributions, dialects, and use cases [168-170].
• Limited Accessibility: Systems lack localization, inclusivity, and accessibility limiting adoption by diverse decentralized groups [171].
• Skewed Optimization: Aggregate metrics like overall accuracy lead to models optimized for majority clients at the cost of minorities [16].
• Privacy Violations: Centralized coordinators could infer sensitive attributes about local client data from model updates [78].
• Entrenched Biases: Global models perpetuate systemic representational biases without reforms to inclusive coordination [172].
• Lack of Recourse: Opaque federated systems deflect accountability for unfair outcomes by attributing them to algorithmic determinism [165].

Decentralized participatory approaches are necessary to ensure representations do not exclude or disadvantage minority clients.

3.7.3 Potential mitigations

Addressing representational biases remains an active area of research in FL requiring technical interventions combined with inclusive participative design and decentralized governance approaches:

• Inclusive Design: Engage wider community participation in problem formulation, model prototyping and testing to surface excluded perspectives.
• Personalized Models: Adapt model selection and hyperparameters for each client to improve personalization and prevent one-fits-all effects [95].
• Subgroup Validation: Use segmented test sets and metrics to ensure model performance is evaluated across diverse clients and environments.
• Bias Metrics: Define quantifiable metrics to assess model fairness and inclusion issues during development [108].
• Nutrition Labels: Require documentation of design choices, assumptions, limitations to inform aggregation and oversight.

While technical interventions help, addressing root causes requires examining who is centered in FL design. Participative, decentralized, peer-based approaches can help reform exclusionary assumptions and structures. Community voices should guide problem formulation, not just passive data contributors. Standards preventing extractive, unethical data practices are also necessary [173]. Dual technical and social responses attuned to marginalized groups can accelerate progress.

While progress has been made in developing bias mitigating solutions tailored to FL, significant open questions and challenges remain to be addressed. The unique constraints arising from decentralization, systems heterogeneity, competing incentives, and privacy considerations pose difficulties in directly applying centralized techniques. Novel advancements are needed across areas ranging from coordinated evaluation to privacy-preserving auditing and incentive mechanisms for promoting voluntary adoption of debiasing techniques. The emerging field of fair and accountable FL continues to be an active research domain, with impacts on developing trustworthy and inclusive AI systems. In this section, we explore critical unresolved challenges and promising research directions to fully harness FL's potential for addressing societal needs and enhancing public welfare. These challenges span technical, ethical, and practical dimensions that must be examined to ensure FL can effectively serve humanitarian causes.

5.1 Lack of coordination

A core assumption in many bias mitigation techniques is the ability to coordinate across the full dataset or training process. However, FL involves decentralized clients that are often distrusting entities with misaligned incentives and competing objectives. This poses challenges for bias mitigation compared to traditional centralized training where full coordination can be enforced. For example, techniques like reweighting underrepresented groups or oversampling minorities require a global view of the overall data distribution to ensure proper balancing. But transparency into other clients' local data distributions to calculate appropriate weights may violate privacy expectations and business interests. Introducing fake simulated data also needs coordination to prevent overlapping samples.

Furthermore, distributed validation of models on segmented test sets representing diverse groups is important to assess biases consistently. But this requires collective coordination in defining evaluation methodology and sharing results. Adversarial attacks exploiting lack of coordination are also harder to detect without global visibility across clients. Moreover, mechanisms for limited coordination could help balance these tensions. For example, secure distributed analytics and differential privacy may provide aggregated insights into bias without exposing raw client data. Further, economic incentives and reputation systems could encourage coordination behaviors aligned with mitigating bias. Furthermore, transfer learning can propagate useful patterns across models without sharing actual data [178].

However, fully decentralized bias mitigation without any coordination remains an open research challenge. Options like trusted industry-specific authorities to govern coordination, norms-based self-organization, and voluntary coordination around social responsibilities may help in specific contexts. But ensuring mitigation at scale without centralized oversight remains an open issue.

5.2 Privacy constraints

While transparency and auditing can help identify biases, strict privacy protections in FL pose challenges. Directly analyzing raw decentralized data to quantify bias metrics would provide the most insight but violates client privacy expectations [179]. Furthermore, differential privacy and secure aggregation techniques allow some validation and analytics in a privacy-preserving manner. But these introduce noise that limits utility [180]. The level of noise required to fully mask membership or attribute inference may render validation metrics too distorted to draw fair conclusions.

Moreover, transparency reports on aggregate demographics and bias metrics can help coordinate mitigation efforts. But this reveals sensitive information about clients so is often avoided. Detailed model cards on performance across groups require profiling users. Therefore, novel privacy-preserving auditing techniques are needed that enable unbiased evaluation without compromising confidential data. Secure multi-party computation shows promise to run distributed diagnostics. Models can also be trained to predict biases without exposing data [181].

However, partially relaxing privacy, such as within industry consortiums, can enable transparency for bias mitigation. But this risks exclusion if marginalized groups lie outside such consortiums. Limited, accountable disclosure may help balance risks [94]. Thus, fully decentralized approaches without transparency remain challenging. Social coordination through norms and voluntary self-disclosure could raise accountability. But ensuring comprehensive, privacy-preserving auditing at scale remains an open research area.

5.3 Evaluating decentralized non-IID data

A key challenge in FL is evaluating models for bias on decentralized non-IID client data. Overall performance metrics like accuracy, calculated centrally after aggregating model updates, can miss disparate impacts on underrepresented groups that exhibit distribution skew [14]. Drilling down to assess model behavior across heterogeneous local datasets is important to quantify fairness. However, this requires sharing samples from sensitive client data, which violates privacy expectations.

While centralized evaluation on representative test sets is common in ML, this poses difficulties in federated settings. Clients are often unwilling to share local data for pooled validation due to confidentiality concerns. Coordinating clients to create standardized test sets that cover diverse subgroups is also arduous. Differences in evaluation methodology, metrics, and labeling schemas across organizations further complicate consistent assessment [182]. Therefore, new decentralized protocols are required that can validate model performance and fairness on heterogeneous local data in a privacy-preserving manner. Secure multiparty computation techniques enable aggregating subgroup metrics across clients without exposing raw data. Differentially private mechanisms allow discerning bias while limiting attribute disclosure. Emerging techniques like federated meta-learning assess models by exchanging trainable parameters instead of actual data samples.

However, practical adoption of such emerging secure evaluation techniques faces barriers. Compliance from competitive clients is difficult to ensure without oversight [183]. For instance, peer auditing raises additional tensions around proprietary model comparisons. Relative benchmarking against fluctuating baselines provides limited insight into absolute model biases. Thus, beyond technical solutions, progress requires establishing community standards, participative governance and incentives promoting accountability [184].

5.4 Adversarial attacks and model poisoning

The decentralized nature of FL makes models susceptible to adversarial attacks and intentional data poisoning aimed at compromising fairness [185]. Malicious clients could inject poisoned updates with embedded biases against certain subgroups that taint the global model. Since the origin of contributed updates is obscured, it becomes challenging to detect such manipulated contributions encoding malevolent biases. Discriminatory attacks that would be evident in centralized settings can remain invisible in FL without full visibility into heterogeneous client behaviors [186].

While techniques like robust aggregation can ignore or downweight outlier updates, advanced attacks craft poisoned updates that appear as normal local changes to bypass defenses. Carefully coordinated injection of poisoning data across multiple colluding devices can further mask bias detection, which is harder without global oversight. Simulating centralized retraining on all raw data can diagnose poisoning but leaks private data. Developing robust anomaly detection tailored to federated environments remains an open area needing novel privacy-preserving inspection of contributions across distrusting parties to identify manipulation of biases. Monitoring metrics indicating emerging skew against protected groups can help [187]. Cryptographic reward systems that incentivize fair updates and flag suspicious behaviors could enhance resilience. Addressing these emerging threats is critical for securing FL against adversarially encoded biases.

5.5 Communication and computation constraints

Many bias detection and mitigation techniques designed for centralized environments involve additional communication and computational overhead, which poses challenges in adopting them for FL. Complex multi-round algorithms or hosting large validation datasets may be infeasible given the limitations of decentralized devices and connectivity constraints. For example, bandwidth-heavy gradient exchange between clients for consensus-based federated averaging results in high latency [188]. Similarly, continuously monitoring detailed metrics and rerunning expensive model diagnostics like full factorization for bias can be prohibitive [1].

Therefore, developing efficient and optimized solutions is crucial to enable practical bias mitigation under federated constraints. Approaches tailored to minimize communication such as exchanging only model updates rather than full parameters can assist adoption. Further, strategies like secure hierarchical aggregation avoids all-to-all exchanges. Furthermore, predictive modeling using proxy data can analyze biases without heavy diagnostics. Moreover, sparse collective matrices reduce intersection computations. Parallelizing operations across clients hastens convergence. Thus, achieving efficacy and robustness against biases within the cost limitations of decentralized environments remains an active research goal needing novel frugal approaches.

5.6 Personalization vs fragmentation for bias mitigation

There exists a tradeoff between customizing FL models to mitigate local biases versus maintaining a synchronized global model. Personalization allows adapting to address distinct biases arising from localized data distributions and needs [95]. However, excessive flexibility can fragment the global model with heterogeneous variations that compound bias [189].

On one hand, personalized federated algorithms tailored to each client's constraints and subgroups enable improved fairness on specific populations [91]. Specialization captures nuanced correlations behind local biases invisible in one-size-fits-all models [190]. However, unchecked personalization can result in clients diverging into isolated localized models that overfit unique biases and fail to generalize fairly.

Carefully regulating model customization is therefore necessary to balance bias mitigation gains from specialization with robustness arising from global coordination. Selective relaxation of certain parameters for local adaptation while coordinating on a shared core model offers one path. Regularization terms that penalize divergence away from global fairness solutions incentivize alignment. Peer-based consensus algorithms propagating useful personalized variations globally provide another approach. Further research is needed to develop adaptive, secure frameworks enabling responsible flexibility balanced with coordination for bias mitigation.

5.7 Incentives for voluntary bias mitigation

In federated settings without centralized control, competitive clients lack inherent incentives to employ voluntary bias mitigation practices that may compromise their individual utilization goals. Enterprises may resist techniques like restricting sensitive attributes or balanced sampling that reduce biases but lower accuracy on their local objectives. Uncoordinated self-interests can perpetuate inequities even if harms are collectively suboptimal [112].

To encourage voluntary adoption of debiasing techniques, novel incentive mechanisms tuned to the decentralized nature of FL are needed. Crypto-economic approaches based on token rewards for good behavior show promise to incentivize fairness [191]. For example, clients can be compensated for adopting updates that enhance equity while temporarily reducing private metrics. Gamification through leaderboards celebrating top contributors tackling bias creates motivational incentives. However, malicious actors may still manipulate such mechanisms if robustness is insufficient [192]. Careful incentive alignment balanced with security remains an open research challenge. Beyond technical solutions, legislation and policies may also be required to mandate responsible bias mitigation practices.

5.8 Emerging best practices for bias mitigation

As FL expands, best practices around ethical data sharing, representation, model interpretability, and algorithmic accountability tailored to decentralized environments are still evolving. While technical interventions help, holistic responses spanning governance, transparency, and participative design are imperative for mitigating bias.

Establishing norms against excluding subgroups, requiring localized testing, instituting peer audits, attaching model fact sheets, and enacting regulations around algorithmic harms can steer progress. Community participation in shaping problem formulation, metrics, and standards prevents narrowly technocratic solutions. Translating technical insights about model behaviors and uncertainties using interactive visualizations improves transparency.

However, operationalizing responsible bias mitigation practices faces barriers around coordination, privacy, and misaligned incentives in federated ecosystems. Developing minimal disclosures, distributed audits, and incentives balancing rigor with confidentiality offers paths forward. Focusing bias mitigation on enhancing equity instead of just detecting deficiencies reorients efforts towards social impact. Ultimately, the path towards trustworthy and inclusive FL necessitates interdisciplinary perspectives attuned to marginalized communities.

5.9 Governance, transparency and participation

Beyond specific technical solutions, addressing biases in FL raises broader societal questions around governance, transparency, and participation that remain open issues. Mechanisms for oversight and accountability are unclear in decentralized ecosystems involving distrusting parties. Standards and policies specifically regulating bias and representation in federated contexts are still emerging across sectors [193].

Transparency around bias mitigation practices, performance differences, and accountability is limited but critical for detecting exclusion errors. However, this conflicts with privacy expectations in federated ecosystems. Developing minimal, participative transparency frameworks balancing accountability with confidentiality is an open challenge. Enabling impacted communities to shape problem formulation, audit biases, and steer solutions centrally recognizes their self-determination.

Realizing the benefits of FL for social good requires inclusive governance and participative design. Co-creating decentralized architectures integrating peer oversight, contestations, and transparency with marginalized groups' interests centered can enhance legitimacy and representation. Beyond technical bias mitigation, broader questions around reforming unjust social structures enabling comprehensive participation remain imperative [194].

5.10 Scalability concerns

As FL systems expand to include a large number of clients, implementing bias mitigation strategies introduces scalability challenges. Techniques such as robust aggregation, personalized modeling, and cryptographic protocols may incur significant computational and communication overhead when applied at scale. The diversity in client devices and network conditions can further complicate the efficient deployment of these strategies in larger networks. Addressing scalability is crucial to ensure that bias mitigation remains practical and effective as FL systems grow, enabling widespread adoption without compromising performance or fairness.