AI is becoming increasingly woven into the fabric of our everyday lives. It is, therefore, imperative that we address the potential harms associated with bias in AI systems – that means eliminating algorithms that perpetuate discrimination, reinforce inequality and yield unfair outcomes.
This article will guide you through how to build a bias measuring and mitigation dashboard app in just five easy steps, using Python alongside the Holistic AI, sklearn and Streamlit libraries.
How to build a bias measuring and mitigation dashboard with Holistic AI open-source library
The Holistic AI library is an open-source tool used to assess and improve the trustworthiness of AI systems. The current version of the library offers a set of techniques to easily measure and mitigate bias across a variety of tasks, facilitating the development of fair, transparent, and ethical AI systems.
We will explore the issue of bias mitigation using the example of admission rates for two distinct applicant groups – group a (white) and group b (non-white).
Step 1 — Setting up the python libraries and front page
The initial step in a data science or machine learning project typically involves importing the necessary Python libraries and setting up the front page or user interface of the project.
That is the case in our bias mitigation dashboard too, as expressed in the code snippet below.
# import libraries
import streamlit as st
import holisticai
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split from sklearn.metrics import RocCurveDisplay
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.ensemble import RandomForestClassifier
from holisticai.datasets import load_law_school
from holisticai.bias.plots import group_pie_plot, distribution_plot
from holisticai.bias.mitigation import Reweighing
# set up app title
st.markdown("# Classification with Machine Learning")
# set up sidebar text
st.sidebar.write(
"""This is a dashboard to measure and mitigate bias with
HolisticAI library.""" )
# load dataframe df = load_law_school()['frame']
# create one tab for each step
step1, step2, step3, step4 = st.tabs(["Step 1: Data Description",
"Step 2: Training Model",
"Step 3: Bias Metrics",
"Step 4: Bias Mitigation"])
Step 2 — Data visualization
For the next step, create a simple data visualization with a pie plot to represent the percentage of white and non-white people in the dataset — and distplot to represent the distribution of students’ undergraduate GPA.
with step1:
st.subheader("Descriptive Analysis")
# protected attribute (race)
p_attr = df['race1']
# binary label vector
y = df['bar']
fig1, ax = plt.subplots(1, 2, figsize=(10,3))
# create a pie plot with protected attribute
group_pie_plot(p_attr, ax=ax[0])
# create a distplot with target and gender variables
distribution_plot(df['ugpagt3'], df['gender'], ax=ax[1])
plt.tight_layout()
# show fig1 in app
st.pyplot(fig1)
Step 3 — Model selection and training
Next, we select a model and create an ROC curve. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. We also compute the area under the ROC curve (AUC), which we use as a metric to quantify the overall performance of the model.
# split features and target, then train test split
X = df_enc.drop(columns=['bar', 'ugpagt3'])
y = df_enc['bar']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
In this step, we compute bias metrics to evaluate whether there are any disparities in the model’s performance across different subgroups of the population – white and non-white applicants in this example. Bias metrics are used to measure the fairness of the model and to identify any potential sources of bias that may be present in the data or the model.
with step3:
# set up groups, prediction array and true (aka target/label) array.
group_a = X_test["race1"]=='non-white' # non-white vector
group_b = X_test["race1"]=='white' # white vector
y_true = y_test # true vector
# create a table with classification bias metrics
bias_metrics = classification_bias_metrics(group_a, group_b, y_pred, y_test, metric_type='both')
# show table with bias metrics in app
st.table(bias_metrics)
Step 5 — Mitigator selection
Finally, we use the reweighing strategy to compute the bias metrics. Reweighing is a commonly used technique in machine learning to mitigate bias in datasets. The strategy involves assigning different weights to different samples in the dataset, based on their group membership, to balance the representation of different groups.
And that's all there is to it. Using the Holistic AI library and Streamlit framework in Python, you can create a user-friendly interface, allowing you to showcase the results of your bias mitigation efforts in machine learning systems.
With the rapid proliferation of AI throughout society, practical solutions for mitigating algorithmic bias have never been more important.
A customisable dashboard that intuitively displays visualisations, tables and other data is the perfect way to visualise data and present it to your stakeholders.