Privacy-Preserving and Trustless Verifiable Fairness Audit of Machine Learning Models

—In the big data era, machine learning has developed prominently and is widely used in real-world systems. Yet, machine learning raises fairness concerns, which incurs discrimination against groups determined by sensitive attributes such as gender and race. Many researchers have focused on developing fairness audit technique of machine learning model that enable users to protect themselves from discrimination. Existing solutions, however, rely on additional external trust assumptions, either on third-party entities or external components, that significantly lower the security. In this study, we propose a trustless verifiable fairness audit framework that assesses the fairness of ML algorithms while addressing potential security issues such as data privacy, model secrecy, and trustworthiness. With succinctness and non-interactive of zero knowledge proof, our framework not only guarantees audit integrity, but also clearly enhance security, enabling fair ML models to be publicly auditable and any client to verify audit results without extra trust assumption. Our evaluation on various machine learning models and real-world datasets shows that our framework achieves practical performance.


I. INTRODUCTION
Machine Learning has seen great success in decisionmaking and decision-support tasks in recent years [1] [2] [3], being deployed in various applications and products in practice, such as loans and hiring decisions. However, concerns are rising that algorithms amplify bias and discrimination from the training data, and fairness is becoming an essential metric for evaluating machine learning models [4]- [8]. Consequently, fairness has become a roadblock to widespread machine learning applications. To address this formally, many works towards considering how algorithm fairness can be assessed by proposing various measures and how discrimination in machine learning systems can be mitigated by pre-processing [9], [10], inter-processing [11], [12], and post-processing methods [5] [13].
In practice, there is a need for guarantees that the result of fairness audit are correctly calculated with respect to specific fairness metrics, which is referred to as the audit integrity of fairness. One of the basic ideas to ensure that users are protected from discrimination is to ensure the integrity of the audit. In order to get a fair model, the server usually requires the user's sensitive data, such as gender and race, to train the machine learning model. However, this requirement is often contrary to the interests of the user. First, users are usually reluctant to share their data, even if it is a reasonable aim, 2 Corresponding Author.
because it would expand their exposure to privacy risks. In addition, the collection of sensitive user data is subject to legal restrictions. For example, the EU's General Data Protection Regulation (GDPR) highlights the minimal prerequisites for collecting sensitive data [14]. If the model itself is not a secret, anyone can potentially run tests on it to establish its purported fairness without exposing its data. However, this approach may be contrary to the benefits of the model owner due to intellectual property. Although there are fair learning approaches [15], [16], training fair models without the sensitive data have been proposed, it is still required to have the sensitive data for assessing the fairness of the trained model [17]. We call this problem as sensitive data availability.
To overcome sensitive data availability issues in providing audit integrity, Veale and Binns [14] introduce a trusted third party with sensitive data to certify the fairness of a machine learning model. Although this model works well, it requires a strong trust relationship between the third party and the model owner. Either the third party has access to the ML model, or the model owner has access to the sensitive data, which may be against their interests. To audit the model publicly while protecting sensitive data's privacy and keeping the model confidentiality, Kilbertus et al. [18] and Segal et al. [19] proposed to utilize multi-party computation (MPC) approach. Those approaches enable a public fairness audit under the assumption of a semi-honest security model and are extended by Pentyala et al. [20] to a malicious security model. Park et al. [17] propose a framework to enable secure fairness audit by leveraging confidential computing based on hardware enclave under the malicious security model.
The problem. While previous work have been work well, existing solutions still suffer from extra trust assumption. This is problematic for two reasons. First, the additional trust assumptions mean the third party determines the audit integrity. Second, relying on a third party can lead to single-point failures. Specifically, the MPC-based approach assumes that the server and the third party are running all required steps in a protocol. Moreover, the hardware-based approach introduces additional hardware security assumptions and also suffers from hardware vulnerabilities [21] [22]. Furthermore, when we are in a situation where we want to audit the model used for several different domains, we need to establish credible relationships with more third parties or hardware enclaves.
To get secure and robust service, we need a much more robust security guarantee: each party only trusts itself. This raises our question: Can we design a framework for auditing the fairness of machine learning models under no trusted party existing scenario? Or can we guarantee security for audit integrity without external trust assumptions? For example, we want to support fairness audits as a service in the model market to achieve fairness integrity.
We answer the question above positively in this paper by proposing a fair audit framework, which enables a publicly verifiable fairness audit of the ML model without disclosing model parameters and guarantees audit integrity of the fair audit. The main idea is to leverage the progress of zero knowledge succinct non-interactive arguments of knowledge (zk-SNAKRs) [23]- [29] recently. A zk-SNARK enables the third party to efficiently convince the verifier that the computation of fairness audit is correctly calculated. We solve the critical challenge of adapting zk-SNARK to this work under the malicious threat model. In summary, the contributions of this work are: • We provide a generic framework to audit the fairness of machine learning model under the trustless condition. We can support generic machine learning models with arbitrary fairness metrics.
• We formally define security requirements and instantiate the framework described above. We have solved performance challenging problems in the instantiation process.
• We implement our framework and evaluate its performance on several real-world datasets. The experimental results show that our framework achieves practical performance.

A. Fairness Audit
Despite training a machine learning model is a fundamental problem, bringing the model to reality is also important. A fundamental question is how to ensure that the model used is non-discriminatory. There is a line of work to discuss this problem. Veale and Binns [14] introduced highly trusted third parties selectively storing data and performing discrimination auditing to achieve fairness in machine learning. However, they assume the modeler must disclose their model to a third party or trust it in order to obtain the model prediction on test data, which may be incompatible with modeler's intellectual property. To resolve these problems, other privacy-preserving approaches such as multi-party computation or trusted execute environment can be applied. Kibertus et al. [18] and Segal et al. [19] proposed privacy-preserving fair certification and inference of ML model that protect sensitive attributes and model confidentiality by using MPC. However, they assume that two honest-but-curious server and require high communication.
The following work PrivFair [20] extend their security model to active security threat models in 2-or 3-server setups. Park et al. [17] provided a generic fairness audit framework that relies on hardware enclaves and explores more potential threats and attacks in the fairness certification process. Although their approach has a small computational overhead, their require additional hardware and trustworthiness to TEE, which is not our goal. And TEE also face many unknown vulnerability [21]. All of these work require additional trusts, and does not provide public verifiable. There computation integrity rely on their trust on third party. In this study, we explore a publicly verifiable security audit protocol based on zero knowledge proof with lower level of trust. Also [19] and [17] explore the auditing dataset are publicly known during model training, which makes the model certification harder by allowing the modeler can adaptive training their model on the audit data. We are also using this approach to improve the reliability and robustness of fairness audits, which is seen as a promising direction for fairness certification.

B. Zero Knowledge Proof
Zero knowledge proofs were introduced by Goldwasser [30] and generic constructions based on probabilistically checkable proofs were proposed in the seminal works of Kilian [31] and Micali [32]. In recent years there has been significant progress in efficient ZKP protocols and systems. A radically different approach in zero-knowledge proof, categorized by their underlying techniques and assumptions, there are pairingbased schemes [24], [25], [27], discrete-log-based schemes [33], interactive-proof [34], [35], interactive oracle proofs (IOP) [26], [36], and so on. They provide different trade-offs between prover runtime, proof size and verifier runtime and so on. Please refer to [37] for more details on the performance and comparisons of different ZKP schemes. Zero knowledge proof has been widely used in blockchains and cryptocurrencies to achieve privacy [38] and scalability. More recently, it also found new applications in zero-knowledge machine learning [39], [40], zero-knowledge middlebox [41], and so on.

III. PRELIMINARIES
In this section, we introduce the fairness notions in machine learning and the cryptographic primitives used in our framework.
Notions. We use λ to denote the security parameter. Let [n] denote the set {0, 1, . . . , n − 1}; a vector be denoted by a boldface letter, e.g., x. And x ← X denote that x is sampled from a distribution X .

A. Fairness Notions in Machine Learning
There is plenty of fairness definitions [42] such as group fairness, causal discrimination, and counterfactual fairness. In this study, we mainly focus on statistical fairness definitions that require protected data, such as demographic parity (or statistical parity) [43], equalized odds [5], equal of opportunity [5], and disparate impact [44]. Demographic parity means that both protected and unprotected groups have an equal probability of being assigned to the positive predicted class. Equalized odds enforces both equal bias and equal accuracy in all demographics. Equal of opportunity is a relaxation of equalized odds, which only focus the positive predication outcome. Disparate impact implies that the decision outcomes disproportionately benefit or hurt members of certain sensitive attribute value groups.
Let M be a trained machine learning model for a classification task. Suppose possible inputs X , sensitive or protected attribute G (relevant for fairness, e.g., ethnic or sex), the true class label y and the predicationŷ = M (x, g), where x ∈ X , g ∈ G. And we use tuple (M, X , Y, G) represent the audit sample D. Consider g = 0 designates the unprotected (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 14, No. 2, 2023 group and g = 1 designates the protected group. We recall these fairness definition below.

B. Cryptography Primitives
Bilinear Groups. A bilinear group is given by a description GK = (p, G 1 , G 2 , G T ) such that • G 1 , G 2 are cyclic groups of prime order p and generators are g ∈ G 1 and h ∈ G 2 • Bilinear map e : A commitment scheme allows a committee to commit a secret value and later open the commitment and reveal the value to the verifier. We recall the commitment scheme definition.
Definition 1: A commitment scheme is a tuple of algorithms Com = (Setup, Commit, VerifyCommit) that works as follows.
• Setup(1 λ ) → ck takes as input the security parameter λ and outputs a commitment key ck.
• Commit(ck, m) → (c, o) takes as input the commitment key ck and a secret value m, and output a commitment c m and an opening o.
• VerCom(ck, c m , m, o) → b takes as input a commitment c m , a value m and an opening o, and output accept (b = 1) or reject (b = 0).
The commitment scheme is required to be both binding and hiding. In the study, we will be using Perdersen-like commitment scheme [27] which is statistically hiding and computationally binding under suit assumptions.
zkSNARKs. Zero knowledge proof enables a prover to prove to a verifier the result y of a computation C satisfying y = C(x, w), where x is public input and w is secret witness of prover. The popular zero knowledge proof notions used in practice are zero-knowledge succinct non-interactive arguments of knowledge (zkSNARKs, for short) [23]- [25].
Here we recall the definition of zkSNARKs as follows: Definition 2: A zk-SNARK for a relation R is a tuple of algorithms Σ = (KeyGen, Prove, Verify, Sim) as follows: algorithm takes a relation R ∈ R λ and security parameter as input, and returns a common reference string CRS and a simulation trapdoor τ .
• Prove(ek, x, w) → π The prove algorithm takes a evaluation key ek from CRS, and (x, w) ∈ R as inputs, and generates a proof π.
• Sim(CRS, τ, R) The Sim algorithm takes a CRS, a simulation trapdoor td, and a relation R as input, and returns a proof π.
A zkSNARKs scheme should satisfy the following properties: • Completeness: For any pair (x, w) ∈ R, the verifier always accepts the corresponding proof.
• Knowledge Soundness: it holds if the prover must know a witness and such knowledge can be efficiently extracted from the prover by using a knowledge extractor.
• Zero knowledge: An argument is zero-knowledge if it does not leak any information other than the truth of the statement. There are exist a simulator without secrets can generate valid proofs.
• Succinctness: The size of a proof is |π| ≤ poly(k)polylog(|x| + |w|) Commit-and-prove SNARK. Commit-and-prove SNARK is a SNARK(cp-SNARK, for short) [27] that can prove knowledge of (x, w) such that R(x, w) = 1 holds w.r.t. a witness w = (u, w) and u opens a commitment c u as follow: We denote a cp-SNARK as a triple of algorithm CP = (KeyGen, Prove, VerProof).
, w) → π outputs the proof of correct commitment.
The above definition has perfect completeness, computational knowledge soundness and zero knowledge in the random oracle model. Please refer to [27] for more details on the formal definition.

A. System Model
As shown in Fig. 1, our framework involves four entities: server, regulators, client, and bulletin board. With the involvement of the entities, we consider such a scenario problem: the server in possession of a trained model seeks to convince any later-coming client that the model satisfies a set of fairness metrics typically defined by a group of specialist regulators, while not revealing the model parameters. We note that we consider multiple specialist regulators who hold different fairness metrics dependent on the policy, law regulation, and environment of their domains, so as to ensure an all-around fairness assessment of a trained model. To address the above scenario problem, our framework contains three phases, including the query phase, the auditing phase, and the verification phase, with the following basic workflow: • The sever commits to the model that will be evaluated, and meanwhile, the regulators commits to their test data that are used for evaluating model fairness.
• Any one regulator can send the test data to the server, and the model is evaluated on the data, thereby obtaining the corresponding evaluation result. Also, the commitment on the evaluation result and the proof regarding evaluation correctness are submitted to the bulletin board. Here refers to the query phase.
• A regulator can audit the model fairness with the evaluation result he obtains, according to the fairness metric he holds. As a result, the regulator submits the auditing result and a proof on correct auditing to the bulletin board. It refers to the auditing phase.
• Any later-coming client who questions the model fairness is able to browse the fairness auditing results and assert the truth with the proofs from the bulletin board. Here refers to the verification phase.
Remarks. We remark that the server provides a black-box query inference towards the regulators, without exposing the model parameters.

B. Threat Assumptions
We consider either the server or the regulators have the motivation to cheat the client with respect to the model fairness in our scenario problem. We now clarify our concrete threat assumptions on the involved entities. Server: The server is considered to be malicious, which may arbitrarily deviate from the evaluation of model fairness. Specifically, 1) it may give incorrect model predication on test data, such as random values or predication not on the committed model or test data; 2) it also may use an unfair model to interact with the regulator, trying to trick the regulator. We note the server has access to the regulator's test dataset in plaintext in our scenario. First, given that the source and amount of data is limited, there is a tendency to audit model fairness on publicly available datasets [19] to obtain richer test data. Second, in scenarios where privacy is required for the test dataset, since each of the regulator's test data only contains sensitive attributes and an identifier as stated in work [14], this represents a lesser privacy risk.
Regulator: We consider regulator as malicious. The regulator might give wrong fairness audit results, such as auditing an unfair model as fair or vice versa, due to conflict of interest or just machine failure. The regulator also might use test datasets that differs from the previously committed data.
Client: We assume client is honest. The client can know the algorithms for model prediction and fairness evaluation, but does not have access to the model parameters. The commitments, audit result, and proofs are always available to client.
Bulletin Board: We assume that integrity and availability hold for bulletin board. The bulletin board can be instantiated by using a blockchain system, such as Bitcoin or Ethereum. We make the assumption like existing work [45]. We also assume the existence of a secure communication channel between any two entities.
Remarks. Note that we do not consider how to train a fair machine learning model and we do not discuss some machine learning attack, such as model extraction, model inversion, and evasion attacks [46] [47]. These studies are outside the scope of this work.

C. Security Goals
We aim to propose a framework for model fairness auditing in trustless setting, which departs from previous works. The framework establishes an evidence on the server-side model fairness, such that an off-line or later-coming client can be faithfully convinced of the truth respective to model fairness.
To be specific, we should achieve the following security goals.
Trustless Verifiable Model Fairness: We require that the truth of model fairness can be efficiently verified by any one non-designated client. Concretely, a client of interest as a verifier can verify the public proofs posted by the server and the regulators, so as to determine if the server's model indeed reaches the fairness degree of the regulators over specific test data.
Model Privacy: We require that neither evaluation results, auditing results or proofs reveals the private information of the model against any one verifier.
Audit Integrity: We can provide the publicly verifiable audit integrity to convince that any later-coming client.
Accountability: We require that a verifier can account the misbehavior of the server or the regulator, if evaluation proof or audit proof cannot be verified. This property cannot be provided by MPC-based works.

V. TECHNICAL CHALLENGE
Above scenario allows the server to know the regulator's test data for evaluating his model, and the regulators to obtain the corresponding evaluation results for auditing model fairness. When any a later-coming client gets the auditing results, we do not desire the client to learn any private information of the server's model, the regulator's test data and the evaluation results. We therefore need to achieve that any client (as non-designated verifier) can check the correctness of model evaluation and fairness auditing without private information of the model, test data and evaluation results.
Zero-knowledge proof technique can be used to address the above scenario problem without leaking private information. As Fig. 1 demonstrated, a regulator can firstly commit to test data (the server also commits to the model). The server then conducts the model evaluation with the test data, yielding the corresponding evaluation results. It also commits to the evaluation results, and generates a proof π 1 that the model is indeed evaluated over the test data, with the committed evaluation results as output. After that, the regulator executes the computations of fairness auditing, and generates a proof π 2 regarding the execution correctness. Lastly, a verifier checks the validity of both π 1 and π 2 without knowing the previously mentioned model, test data and evaluation results. Despite the easy-following technical roadmap, we encounter the following two challenges for efficiency:

A. Supporting Lightweight Verification
Our work adopts the state-of-the-art zkSNARK scheme constructed by Groth [25] (refer to Groth16) due to its short proof size (three group elements) and efficient verification. Despite the optimal performance of the scheme, the direct adoption without any modification cannot satisfy our scenario requirements. Specifically, to test model fairness and ensure the result reliability require sufficiently large test data, e.g., 6800 test inputs per regulator, as mentioned in [19]. Based on the scenario, the overall proof size will increase linearly with the amount of data, although the single proof of the Groth16 scheme is succinct. But a verifier is considered a thin device with restricted resources, and thus the result proof as described above easily becomes a computational and storage burden on the lightweight verifier. Furthermore, this situation becomes even worse when we want to verify the results of the fairness audit of multiple regulators with different test data and different fairness metrics. Therefore, the challenge is implementing more efficient verification when the model fairness is audited by large test data from multiple regulators.

B. Improving Proving Performance
Recall that the proof generation process based on an zk-SNARK scheme involves compiling a computation (e.g., model fairness auditing) into a circuit, such as arithmetic circuit and Boolean circuit, and then expressing the circuit with rank-1 constraint system (R1CS) used for generating proof. The involved arithmetic and Boolean operations generally determine the efficiency of the zkSNARK scheme used in practical. For example, the overhead of the Groth16 scheme is friendly for arithmetic operations in the finite field but unfriendly for Boolean operations (due to the radical blow-up in the circuit size required to compile Boolean operations into the arithmetic circuit). Such technical feature, however, is in conflict with the concrete computation of our scenario, since model fairness auditing involves many Boolean-efficient operations such as comparison. Our challenge thus is efficiently handling Boolean operations in proof generation.

VI. CONCRETE DESIGN
We now present our framework that enables the fairness audit for an ML model while keep model confidentiality in an efficient publicly verifiable manner, enabling each one to assess model fairness individually and thus minimize trust dependency between server, regulator and client. It builds on Groth16 zk-SNARK scheme, Pedersen commitment and signature scheme. As mentioned above, our framework consists of three phase: query phase, audit phase, and verification phase. (1) In the query phase, the regulator query the server's model using the committed test dataset. Then the server evaluate model on test data and generate a proof for correct evaluation, and sends the evaluation results and the evaluation proof to the regulator. (2) The regulator verifies the evaluation proof and evaluate fairness metrics of model. And then the regulator generate auditing proof. And release auditing result and auditing proof to the bulletin board. (3) The client verifies the evaluation proof, audit proof and determines the fairness of the server's model.
A. Query Phase 1) Model evaluation.: In this phase, the main goal of regulator is to obtain the evaluation of the server's model on test data that can be used to audit the model's fairness. Firstly, the regulator collects sufficient test dataset X = {x 0 , . . . , x n−1 } (in a legally compliant way or directly extracted from public audit datasets [19]) and commits it to cm x using the Pedersen commitment scheme. The server also commits its ML model parameter to cm m using the same commitment scheme. Then both parties posting their commitments cm x and cm m to the Bulletin board respectively. Noted that the ML model structure is known to the verifier, we only protect the model parameter privacy, e.g. weight information. Secondly, the regulator transmits test data X = {x 0 , . . . , x n−1 } to the server, then the server computes the evaluations of model on the test datâ Y = {y 0 , . . . , y n−1 } and generates the evaluation proof π e . Moreover, the server commit the evaluation of model to cmŷ and publish it and the evaluation proof π e to the Bulletin board, and the server sends the model evaluationsŶ to the regulator.
2) Proof generation.: Our framework conducts the commitand-prove paradigms [27] so that we can support zeroknowledge evaluation for both secret input and secret models in a straightforward way. Specially, in our scenario, we allow the server get the test data in plaintext, and the regulator obtains the evaluation results for audit the model. However, from the client's perspective, the privacy of the model, test data, and the evaluation result all is preserved.
The claim from [19] stated that an ML model M is ϵ-fair with confidence 1 − δ if: where m g denotes the number of occurrences of g in T . Takes EF G = 0.05, ϵ = 0.1, δ = 0.2 and |G| = 100 as example, we need sample number m g ≈ 6800. In this scenario, although the individual evaluation proof are small, thousands of test data make verifying multiple evaluation proofs expensive. As stated above, however, our goal is to keep the succinctness of proof due to we want to support a lightweight client who has limited memory and computation resources. There are two common techniques to keep succinctness of multiple proofs in literature, one is SNARK recursion [48] [49], and other is proof aggregation [50] [51]. The SNARK recursion can prove the proof is correct, and we can compress a sequence of proofs into one proof. Specially, we can aggregate proofs via recursive composition that create another SNARK for the circuit that contains n copies of the Groth16 verifier circuit [48]. But the SNARK recursion incur significant practical overhead due to we need to compiler the verify algorithm into a circuit, and this is the bottleneck of recursion SNARK efficiency. For example, computing a pairing on the BLS12-377 curve require ∼ 15000 constraints [48].
3) Proof aggregation.: In our work, we adopt proof aggregation technique. Inspired by SnarkPack [50] [51], we resort to utilize special structure of proof to aggregate multiple proof. SnarkPack propose an approach to reduce the overhead in communication and verification time for verify multiple proofs without the need of further larger trusted setup ceremonies. The SnarkPack allows to aggregate n Groth16 zk-SNARKs proofs with O(log n) proof size and verifier time and can be constructed from two different existing ceremonies (e.g., the "power of tau" for Zcash [38] and Filecoin [52]).
We explain the aggregation protocol used in SnarkPack and how it can adapt to our CP-SNARK scenario below. First, we recall the verification process used in Groth16. A detailed description of Groth16 SNARK protocol can be found in [25].
The proof π in Groth16 consists of three group element π = (A, B, C) ∈ G 1 × G 2 × G 1 . For the verification algorithm, we need the verification key vk: The verifier need check that pairing equations is satisfy: The high level idea of Groth16 aggregation is straightforward: instead of checking that n different pairing equations are simultaneously satisfied, it is sufficient to prove that only one inner pairing product of a random linear combination of these equations defined by a verifier's random challenge r ∈ Z p holds. The same idea is heavily exploited exploited in polynomial commitment, SNARK batch, SNARK recursive [48].
The aggregation will instead check a single randomized equation: And we rewritten above equation as: And then we will check that Z AB , Z C are consistent with the initial proof triples. Here we use two notions: the target inner pairing product (TIPP) and the multiexponentiation inner product (MIPP) (detail can see [50]).
• TIPP: takes some committed vector A ∈ G n 1 , B ∈ G n 2 and shows that Z AB = n−1 i=0 e(A i , B i ); • MIPP: takes a committed vector C ∈ G n 1 and a vector r ∈ Z n p and shows that Z C = After that we can use TIPP and MIPP to generate the aggregated proof π = (π t , π m , T AB , U AB , T C , U C , Z AB , Z C ), where the last two elements are required to verify the Groth16 equation, the first two elements used to verify the TIPP and MIPP arguments, and other elements are required for the verifier derive randomness r in Fiat-Shamir transformation [53]. After verify TIPP and MIPP proof, the regulator use Z AB , Z C as the linear combination of the proofs. Then the regulator verify the Groth16 equation using the aggregated proof Z AB , Z C and decides whether to move to the next stage.
In the case of CP-SNARK, we need additional element D of the proof that contains a commitment to the data and to create a CPlink to link D to the external commitment. Also we need some additional element in CRS to create D and the CPlink. Nevertheless, the special structure of the proof does In order to verify the proof, we only need verify equation of the structure: e(A, B) = Y · e(C, h δ ). Thus, we can obtain the aggregated proof in the same way.

B. Auditing Phase
In this phase, multiple regulators aim to evaluate particular fairness metrics depending on the environment, policy, and industry to determine the fairness of the ML model.

1) Fairness audit:
Following the common approach in [42], we can use a confusion matrix (see Table I) to compute statistical metrics of ML model, which is what most statistical measures of fairness rely on. The basic notion in confusion matrix as follow: • True positive (TN): a case when the predicted and actual evaluation are both in the positive class.
• False positive (FP): a case predicted to be in the positive class when the actual outcome belongs to the negative class.
• False negative (FN): a case predicted to be in the negative class when the actual outcome belongs to the positive class.
• True negative (TN): a case when the predicted and actual evaluation are both in the negative class.
Based on these basic concepts, we calculate the fairness metrics. Specifically, the fairness notion we took in this work is as follows: Demographic parity (DP), Equalize odds (EO), Equal opportunity, and Disparate impact (DI). Use these formulation, following the basic paradigm in zk-SNARKs, we can express the whole fairness computation as arithmetic circuit and then generate proof.
2) Technical observation: As mentioned earlier, the challenge in generating proof is that we must handle many nonlinear operations because we adopt group-based fairness notions, e.g., divides, comparison, sorting, and so on. To resolve the challenge, our framework uses the observation that for a prover to convince a verifier that it knows the output of some non-linear operation, the prover does not actually need to execute the non-linear operation in the circuit. Instead, the prover just needs to prove that the output of the non-linear operations is correct. For example, suppose the prover wants to prove z = x/y to the verifier. In that case, the prover does not need to straight to compiler divide to a ciruit but simply provides a divided result z as an "advice" and then prove multiplication operation x = z×y. Note that the multiplication operation only contain one multiplication gate, so it much more efficient than naive encoding the divide operation to a circuit (e.g., compute inverse use Fermat's little theorem, a · a p−2 = 1). The observation is broadly used in literature [24], [39], [54], [55]. Next, we will show how to bring this idea into our framework to help model audit.
3) Handle non-linear operations: At a high level, we can split the computation in the audit phase into two-component: one is to evaluate fairness metrics, and the other is to compute the empirical fairness gap. First, we summarize the typical non-linear operation in the auditing process below and then design a protocol to prompt efficiency. The typical operations in statistical fairness evaluate as follow: 1) divides; 2) absolute value; 3) comparison; and 4) maximum or minimum value. To clarify the research methodology, we consider the following examples. First, the regulator needs to compute the basic metrics: TP, FP, TN, and FN. And then calculate fairness metrics like TPR and FRP. Finally, the regulator computes the EFG corresponding fairness notion, such as DP.
For division operations, we have solved as above. For comparison operation, e.g. x ≥ 0, we ask the prover to provider the bit decomposition (a 0 , . . . , a k ) of x as a witness (a k denote sign bit, e.g. positive(1) or negative(0)). Then we can check that: 1) each a i is binary: a i (a i − 1) = 0, ∀i ∈ [k]; 2) The correctness of bit-decomposition of x: In calculating the EFG, we need to find the maximum metrics gap between different group. Naive computation is not feasible because it requires us to make a two-by-two difference between all the elements in the group and then select the maximum gap value. A more thoughtful way is to sort the array and then use the maximum value minus the minimum value to get EFG. Nevertheless, directly representing a sorting algorithm such as QuickSort as a circuit requires a comparison that contains O(n log n), which is expensive. Following the above observation, we asked the prover to provide some "advice" as a witness to improve efficiency, and we can combine absolute and maximum value into one relation. Concretely, assume that we have (x 0 , . . . , x n−1 ) denote the fairness metrics on n test data and want to compute max |x i − x j | between different group. The prover is required to provide the maximum x max and the minimum value x min in list as an auxiliary witness. Then we can check that x max is actually the maximum number as follows: The first condition can be checked by bit-decomposing of x max − x j and then checks are exactly the same as the comparison operation. Similarly, the check of x min is exactly the same as x max above.
Overall, comparing to straight compute maximize value above, the prover only additionally provides bit-decomposing of maximize and minimize value, and the protocol checks two additional bit decomposition. After all of computation done, the regulator publish all of the commitments, proofs, and audit result with associated signature on bullet board.

C. Verification Phase
Depending on the application area, regulator is selected to verify the correctness of the audit results. After get the audit result from y, the client request committed values for the ML models cm m , test data cm x , and evaluation results cm y . The client then request evaluation proof π e and auditing proof π a from the bulletin board.
Upon reception of all of message, the client verifies the signature using the public key of the selected regulators and server for the authenticity of commitment, proof and results. If satisfied, client then verifies the evaluation proof π e and audit proof π a using the verification key and commitment. If all these verification pass, then the client will be convinced by the regulator's auditing results and thus determinate the fairness of the model.
If the client verify failure then one can identify the misbehaving party and take penalize it, in the form of reputation evaluation, incentives, etc. Then the party responsible for the misleading behavior is deterred and restart a new auditing process. After all checks are successful, the client can determine the correctness of the ML model auditing results.

VII. EVALUATION
We implemented our fairness audit framework and we present the experiment result in this section.

A. Setup
We have implemented the fairness audit framework in C++ using the libsnark [56] library. We run all of the experiments on 4-core Intel i7-5600k (2.6 GHZ, 8 physical cores) and 48 GB of RAM with Ubuntu18.04. Note that we run our experiments on Docker container. Our current implementation is only use a single CPU core. We report the time in seconds and take the average of 10 runs per experiment as the result.
We used three real-world datasets from various domains: German credit dataset (German), Bank marketing dataset (Bank), and Adult income dataset (Adult) [57]. The datasets vary in size and disparity of minority groups and as such some can be used to create fair or unfair models. In this experiment, we train fair logistic regression (FLR) 1 . Note that due to zk-SNARK systems only support group elements, we use a generic 8-bit unsigned quantization technique to transform float into integers.

B. Performance of Model Query
In this section, we report the performance of the query phase of the model. First, we use FLR model to measure the time to generate evaluation proofs for all three datasets. Since matrix multiplication is involved in all three models, a representative FLR is selected to measure the performance. Table II shows the results of evaluation proof in model query phase. The FLR model needs matrix-vector multiplication between the weight vector and input matrix of test data, where the sizes of input matrix are 1000 × 20 and 45550 × 20, and 48880 × 15. As a result, the input test data and the number of samples affect the circuit constraints and thus the performance.
After that we measure the overhead incurred by aggregating the proofs. Due to the different sizes of the three datasets to ensure the fairness of the model, we set different fairness gaps and confidence levels just like in [19]. We performed our test on the mentioned datasets with δ = 0.05 and ϵ = 0.1. Our evaluation result can see in Fig. 2. The results show that aggregated proofs of thousands of proofs are relatively small in size and can be verified in a few seconds. Although the proofs take more time, this is not impractical. In practice, aggregation operations can be performed offline without affecting the online use of the model after deployment.

C. Performance of Fairness Audit
After query phase, the regulator calculates fairness metrics based on the evaluation results from the server. Thus, we measure computation times of fairness metrics in three datasets. Naturally, we split the data into a training and test subset. The sizes of the three datasets are 200, 9100 and 9000, respectively. In Bank dataset, we use the marital status feature as binary sensitive attribute and income for labels. We considered gender as a binary sensitive variable in Adult and German. Table III shows the evaluation results of fairness audit. The experimental results prove the usefulness of our framework, which takes only a few milliseconds to verify the correctness of the fairness audit results, making it easy for anyone to ensure that they are protected from discrimination. In addition, both preprocessing and prove times are within acceptable bounds.

VIII. DISCUSSION AND FUTURE WORK
One problem with the Groth16 system used in the framework instantiation, despite its state-of-the-art proof scale, is that it requires a trusted third party to generate the CRS used to construct the proofs. While it is possible to use MPC to generate reliable CRS [58], a transparent zkp system [28] could be used instead to avoid trusted party, which is left for future exploration. We expect that better ZK systems will emerge to replace the ZKP schemes we use and thus improve the efficiency of proposed framework. In order to achieve optimal ZK systems in all aspects, such as proof size, proofing time, etc., one promising direction is proof composition [37], [59]. Also, we can naturally extend our work to support confidential model prediction and model accuracy assessment [39], [40].
For future work, we want to explore further protecting the privacy of test data on the server. Especially since there are no publicly available test data and the user data is susceptible in some scenarios. A promising direction is the use of verifiable   [60] to ensure integrity and confidentiality. Finally, it is also interesting to explore other fairness definitions beyond group-based as a research direction.

IX. CONCLUSION
This work proposes a framework to prompt publicly fair audits for a machine learning model. Unlike previous work, our construction only assumes the third-party collection test data and does not rely on the third party. We minimize the trust between the server, the regulator, and the client. Also, our framework can support multiple regulators to provide more strength and border fairness without additional trust assumptions. Our experimental evaluation confirms that our framework is practical for fairness auditing ML models with real datasets.