remote attestation for openfl participants

[anshubit5516]

Summary

As part of approved changes for design discussion, added the changes for providing attestation reports for openFl participants.

Type of Change (Mandatory)

Specify the type of change being made.

• Feature enhancement

Description (Mandatory)

This PR adds basic sgx attestation feature to openFl participants. Feature is controlled via plan flag : enable_remote_attestation. If enable_remote_attestation is True, then before any participant starts, it generates it enclave identity which includes mrenclave/ecdsa key pair and certs/ and attested token of it sgx quote via ITA.

if enable_remote_attestation is True, then there are few additional parameters are required which must be passed an env vars.

• ITA_API_KEY : Its a api key which user must pass to interact with ITA.
• AVS_URL : Its ITA base url
• ATTESTATION_REPORT_PATH : Its the path which attestation reports will be added like public key/ ITA token report.

Example on how to pass these params :

CERT_HOST_PATH="<host_path>r"
CERT_CONTAINER_PATH="/workspace/attestation/"

docker run --rm
--name "aggregator_container"
--network host
--security-opt seccomp=unconfined
--device=/dev/sgx_enclave
-v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket
--mount type=bind,source=./cert_agg.tar,target=/certs.tar
--mount type=bind,source=$CERT_HOST_PATH,target=$CERT_CONTAINER_PATH
--env ITA_API_KEY=$ITA_API_KEY
--env AVS_URL=$AVS_URL
--env ATTESTATION_REPORT_PATH=$CERT_CONTAINER_PATH
${workspace_name} bash -c "mkdir -p $CERT_CONTAINER_PATH && tar -xf /certs.tar && gramine-sgx fx aggregator start" > /dev/null 2>&1 &

As summarized in design discussion, currently user is expected to verify the authenticity of other participants manually.

Testing

[Describe the testing done for this PR. If applicable include screenshots.]

[teoparvanov]

@anshubit5516, a couple of general comments regarding the PR quality:

• please provide a more thorough description, including a detailed Testing section
• could you sign your commit(s) via git commit -s -m "...", so that the DCO check passes and the CI can start?
• make sure the code formatting check passes too (I currently see some non-compliant spacing/indentation in certain places across the PR).

[teoparvanov]

LGTM overall, left a few comments from my first review pass.

[theakshaypant]

1. Would suggest running the formatting script to fix the code formatting pipeline failure. sh scripts/format.sh
2. Tests failing

FAILED tests/openfl/interface/test_aggregator_api.py::test_aggregator_start - KeyError: 'network'
FAILED tests/openfl/interface/test_collaborator.py::test_collaborator_start - TypeError: 'Mock' object is not subscriptable

3. Would suggest adding the copyright header in the newly added files.

[theakshaypant]

I would also suggest adding a page in the documentation to educate the user on how to use this feature.

[teoparvanov]

It's looking good @anshubit5516, thanks a ton ! I've added a couple of comments from the perspective of how the AttestationManager would be used downstream (f.e. in OpenFL-Security).

[teoparvanov]

LGTM, with a final set of comments:

[teoparvanov]

same here

[anshubit5516]

This check is added as unit tests are sending Mock Object which doesn't have config or is an instance of PLAN. Hence added the check.

[teoparvanov]

Why not mention the attestation report path too?

[teoparvanov]

same

[teoparvanov]

I don't think signer.py has anything to do with secagg, so please consider moving it to openfl.cryptography or openfl.utilities, as discussed previously.

[psfoley]

+1 to this. It belongs in openfl.cryptography

[psfoley]

Thanks for the PR, @anshubit5516. I see a couple gaps here:

• You have logic to create TLS credentials specific to the enclave, but it doesn't look like it gets used after the ITA report is generated (i.e. in the gRPC server). Am I missing something?
• I have some questions about why a private key path should be passed into the AttestationManager. I don't think there's a security hole because I think what's intended is the user runs the docker run command with ...gramine-sgx fx aggregator start as the entrypoint, but generally I think it would be safer to hardcode the path to ensure it's specific to the enclave's local FS
• Enabling remote attestation for aggregator / collaborators independently. Secure aggregator running inside SGX will be a very common ask and use case.
• Documentation: What remote attestation is, what problems it solves, and all of the steps to perform it with this PR.
• Tests: I understand it will not be easy to implement for E2E CI because of SGX dependency, but some tests would be useful here given the scope of the changes

[psfoley]

Why not this?

Suggested change

	from openfl.utilities.attestation import attestation_utils as attestation_utils
	from openfl.utilities.attestation import attestation_utils

[psfoley]

+1 to this. It belongs in openfl.cryptography

[psfoley]

I would suggest changing this and moving the enable_remote_attestation parameter to the aggregator and collaborator sections so that it can be configured independently; I expect it will be common that only the aggregator can run inside SGX

[teoparvanov]

Hey @psfoley , thank you for the detailed review!
Let me try to address your comments:

• You have logic to create TLS credentials specific to the enclave, but it doesn't look like it gets used after the ITA report is generated (i.e. in the gRPC server). Am I missing something?

We envisioned this approach as an initial iteration for 1.9, where the attestation "bundle" (cert+mrenclave+ITA token) is generated, but not wired in OpenFL, and only used E2E in OpenFL-Security (ref. design discussion summary). I understand this is incomplete from an OpenFL perspective, but it was intended as an intermediate step. Now that it becomes clear that the OpenFL-Security changes won't be able to make it into GA, we could consider the following options (among others potentially):

1. Deliver what we currently have for OpenFL 1.9, knowing that the ITA token and cert from the bundle can be verified together as a sort of "offline passport", but without using this identity in the actual TLS communication. This is probably of limited value, but it's a start (esp. for any upcoming live federations built on OpenFL 1.9)...
2. Attempt to course-correct by additionally wiring the cert generated within the enclave to the gRPC server, knowing that this might not make it into 1.9. The difficulty here is related to distributing the certificates, which is a manual process in OpenFL versus automated in OpenFL-Security. Short of implementing a Governor-like entity, one option would be if only the aggregator exposed an "attested" certificate. So, instead of generating a CSR and signing the Aggregator certificate via fx collaborator certify, in this approach the Aggregator's cert would be generated on enclave start up, and distributed to collaborators manually (as part of the attestation bundle).

• I have some questions about why a private key path should be passed into the AttestationManager. I don't think there's a security hole because I think what's intended is the user runs the docker run command with ...gramine-sgx fx aggregator start as the entrypoint, but generally I think it would be safer to hardcode the path to ensure it's specific to the enclave's local FS

I see, it makes sense. We'll discuss with @anshubit5516 how this can be achieved to align with the usage in OpenFL-Security.

• Enabling remote attestation for aggregator / collaborators independently. Secure aggregator running inside SGX will be a very common ask and use case.

Good point, we'll provide separate properties for those. Maybe we could even start with just the aggregator for this first iteration (enable_aggregator_attestation), because the workflow around semi-manual attestation of the collaborators isn't very clear at this point.

• Documentation: What remote attestation is, what problems it solves, and all of the steps to perform it with this PR.

This will be done either in this PR, or in a follow-up one, but it is indeed in scope for the task.

• Tests: I understand it will not be easy to implement for E2E CI because of SGX dependency, but some tests would be useful here given the scope of the changes

We will discuss with @anshubit5516 what can be done in a non-SGX context (maybe by using mock SGX device files and a mock ITA service?). It may be challenging though, so we will likely primarily rely on the E2E test by QE (manual, to begin with).

[teoparvanov]

Now that I re-look at this, it is likely not safe to save the private key into the attestation report path. Let's discuss additionally how to deal with this (hard-coded path vs defaults vs other options?).

[noopurintel]

Please ensure to add documentation changes as well to include this functionality or modify the README.md file.

[teoparvanov]

As pointed out by @psfoley , we will likely need to introduce a fx collaborator certify --import-self-signed command for the aggregator's attested certificate to be added to each collaborators' trust store.

[theakshaypant]

Removing these imports might break the secure aggregation flow. I would suggest keeping them unless it is causing any unexpected issues.

[anshubit5516]

In the last commit I have updated the code so that if enable_remote_attestation is True for Aggregator, then it will generate its own certificate and with the help of env var (ROOT_CERT_PATH) can export its certificate to host. Now if this cert is passed to collaborators use this as root cert it can allow connection to its grpc connection. But catch is we have to disable require_client_auth. It will only work if collaborators certs are available to aggregator before gRPC server starts.