The Mystery of the KBS Identity

kbs-identity.md

The Mystery of the KBS Identity

One simple question has confounded countless developers working on Confidential Containers; how do we know we are connecting to the correct KBS? For context, KBS is short for Key Broker Service, which is the trusted entity that conditionally grants access to client secrets. The term relying party could be used to describe the KBS. Inside the guest, there is a Key Broker Client (KBC) built into the Attestation Agent (AA). The KBC talks to the KBS to get container decryption keys among other things.

The connection between the KBC and the KBS is secured with public key cryptography. The KBC generates a random keypair and sends the public key to the KBS when requesting confidential resources. Since the KBC has the lifespan of one VM, it makes sense for it to have an ephemeral keypair. The hash of the public key is included in the hardware evidence, which is also sent to the KBS. With this evidence, the KBS (with the help of an Attestation Service) can verify that the public key it receives from the KBC was generated inside a real TEE with a certain initial TCB. This is precisely what the KBS needs to validate before releasing client secrets to the KBC.

While it's no problem to send the public key of the KBC to the KBS along with the hardware evidence, it's not as clear how we should get the public key of the KBS to the KBC. Some KBCs sidestep this question by setting up the secure channel via RSA, but even if RSA does not require the public key of the KBS to be known to the KBC, it might still seem like we should verify that we are talking to the right party.

Why don't we measure the key?

The KBS is a long running process with a fixed public key known to the guest owner. Since verifying the KBS public key would be part of the setup of the secure channel between, you can't use this secure channel to provision the key. The public key is not confidential so instead of secret injection, we could provide the key as part of the measured boot state of the guest. For SEV-SNP this could be done by putting the public key or a hash of the public key in the host data field. There is a corresponding field for TDX and we could even do this with SEV by injecting the key into the firmware binary. Unfortunately, there are two issues with these approaches.

First, Confidential Containers is designed to use a two-stage measurement system. The first stage is the measurement of the initial TCB by the confidential hardware. The second stage is the measurement/decryption of the workload container images. These stages are decoupled so that any container can run in the same confidential environment without adjustments to the initial TCB. The host data is a first-class entity in the SNP Attestation Report, so changes to it do not affect other parts of the measurement. Even so, continuously updating the host report is in tension with the goal of having a generic guest measurement and a relatively simple verification process. For platforms like SEV that do not have a host data field, measuring the KBS public key would complicate verifying the attestation evidence even more.

Even if the public key of the KBS were in the host data field, however, it's not clear that this would provide an additional security guarantee. The host data is validated by the KBS. Let's imagine that a malicious CSP tampers with the network to connect the KBC to a malicious KBS. The CSP could also change the host data field to point to the public key of that KBS. Since the KBS is malicious it can validate the hardware evidence any way it chooses and establish a secure channel with the KBC. The presence of the KBS public key in the host data field does absolutely nothing to guarantee that we are talking to the correct KBS. The KBS is essentially being asked to validate itself, something that a malicious KBS can do.

Fortunately, there is a much simpler way to know that we've connected to the correct KBS. Only the correct KBS will have our secrets. If we can run an encrypted container image, then we must have connected to the KBS with the keys to decrypt this image. More specifically, the execution of a confidential workload should be gated on the receipt of a secret. This could mean that the container image itself is encrypted and contains some identifying information, such as the credentials for a database. Since workloads can make requests to the Attestation Agent, we could also use a signed container image that requests secrets from the AA.

Unfortunately, not all resources that the KBS provides are confidential. The KBS can also be used to provision the policy and key information for validating image signatures. Image signature validation requires public keys. Since these aren't secret, these don't confirm the identity of the KBS. A malicious KBS could provide policies and keys to validate just about any image. As a result, workloads cannot rely on signature validation alone. This would break the edict mentioned above; workloads must be gated on the receipt of a secret. We can use signatures in combination with secrets, as many pods likely will.

Another Perspective

The above is the response that I have typically given to people asking about how the KBS public key is provisioned. Recently, however, I have begun to look at the issue from a different perspective that I think is much more intuitive. It turns out that even the simplest questions posed here rely on some assumptions that lead to confusion. Let's start from the basics.

Imagine that you have just inherited the Klopman Diamond. You want to keep it in a safe place so you visit your local bank. You decide to rent out a safe deposit box. Before depositing your diamond, you inspect the box to make sure that it is sturdy and that it is empty. This is akin to an attestation. You are validating the isolation mechanism and initial TCB of the box. Once you are satisfied, you use the key to lock up the box with the diamond inside. There is only one key. This is your secure connection to the box.

While the bank probably asked you for some money (much like a CSP) at no point did the safety deposit box try to inspect you. The safety deposit box is an inanimate container. It does not care who it is being rented by. The more important thing is that the renter is satisfied with the guarantees of the box. This maps directly onto Confidential Containers and perhaps confidential computing more generally. In the previous section we were conceptualizing the flow from the perspective of the KBC. Instead, we should think about it from the perspective of the KBS, because the KBS operates on behalf of the client. In short, we don't need to validate the identity of the KBS because we are the KBS.

In Confidential Computing we usually talk about the hardware as the root of trust. The hardware is the root of trust of an enclave, but in Confidential Containers, it's really the KBS that is the root of trust of a workload. Through attestation this trust is extended to the enclave. It's tempting to think about this process the other way around, but this causes confusion. If we start from the KBS, most of the questions evaporate. Including the question that started everything off.

Let’s say that the CSP spins up a VM for us, but then manipulates the network such that the KBC connects to a KBC that does not belong to us. This is no different from a bank renting out a safety deposit box to someone that is not us. There might be some orchestration snafus if the resources are misallocated, but this is not a security issue. The boxes are generic. I will use any one that meets my standards. There is a lurking concern, however. Surely it would be an issue if a user becomes convinced that they are communicating with a workload that does not actually belong to them. It’s important that only one guest can have privileged communication with an enclave. In other words, it’s important that an enclave connects only with one KBS. Otherwise, a pod might end up with a mix of containers representing different entities, some of which could be hostile to each other. Fortunately, this situation is relatively easy to avoid. The Attestation Agent must connect with only one KBS, just like a safety deposit box should have only one key.

There is one final scenario to consider. Let’s say that after you leave the bank one of the workers rearranges the labels on all the safety deposit boxes. This would be annoying, but it wouldn’t compromise confidentiality. You are still the only one who can access your box. It might take a while to find it again, but you’ll know you’ve got the right one when you use your key to open it and you find all your stuff inside. If you didn’t have anything in your safety deposit box or if you only deposited some non-identifying pocket lint, then you might not be as confident that you found your box again. This highlights the importance of having a non-generic workload. In Confidential Containers enclaves are generic. The secrets provided by the KBS are the identity of a guest.

These aren’t new conclusions. In fact, these are the same conclusions we reached in the first half of this article. To me this viewpoint from the perspective of the KBS rather than the KBC is a lot easier to think about.

@fitzthum @mingweishih @anakrish @jepio @dubek
I was thinking that if we had the following mechanisms, we might be able to solve our difficulties more gracefully and avoid the attacks discussed above:

The Attestation Agent (AA) requests a "Token" resource from KBS/AS in the startup phase. After the Attestation Evidence is verified, AA will get the Token. Token contains the following contents:

1. KBS certificate.
2. KBS User ID.
3. TEE pubkey.
4. AA Identity ID assigned by KBS. (This ID is unique).

After that, subsequent secret resource requests need to carry this token.

We can extend more content in this token, which may solve many problems in the future.

@fitzthum @mingweishih @anakrish @jepio @dubek I was thinking that if we had the following mechanisms, we might be able to solve our difficulties more gracefully and avoid the attacks discussed above:

The Attestation Agent (AA) requests a "Token" resource from KBS/AS in the startup phase. After the Attestation Evidence is verified, AA will get the Token. Token contains the following contents:

1. KBS certificate.
2. KBS User ID.
3. TEE pubkey.
4. AA Identity ID assigned by KBS. (This ID is unique).

After that, subsequent secret resource requests need to carry this token.

We can extend more content in this token, which may solve many problems in the future.

The design looks interesting , but I have some questions

• Does the design solve the KBS identity problem as putting KBS public key in the host data does?
• Given that KBS/AS validate AA based on the attestation evidence, would there be a concern that any pod created on same host is able to get the token?

Given that KBS/AS validate AA based on the attestation evidence, would there be a concern that any pod created on same host is able to get the token?

@mingweishih On your second question, I need to add one fact:
The fourth item of the Token is the ID assigned by KBS to AA, which means that the AA has registered its identity with KBS, and KBS can easily configure the maximum of the number of registered AA-IDs. For example, if the user only rents two pod instances, the maximum number should be 2.

Given that KBS/AS validate AA based on the attestation evidence, would there be a concern that any pod created on same host is able to get the token?

@mingweishih On your second question, I need to add one fact: The fourth item of the Token is the ID assigned by KBS to AA, which means that the AA has registered its identity with KBS, and KBS can easily configure the maximum of the number of registered AA-IDs. For example, if the user only rents two pod instances, the maximum number should be 2.

Got it. So, the approach is hard-coding the KBS URI in the AA, and AA is captured as part of the rootfs (thus is reflected in the attestation report). This approach effectively achieves the same as including KBS's public key in the host data.

Then I think this approach is similar to what Azure Confidential ACI does: Using a sidecar container (analogous to AA) that community to a KBS-like Azure service using hard-coded URI/root cert. The only difference is that AA in CoCo is measured as part of rootfs while the sidecar container in Confidential ACI is captured by a security policy via host data.

@mingweishih

Let me know if the above summary is correct. There could be more issues. Feel free to add if folks come up with anything. I'm now thinking more on the container metadata validation part and will add details in the RFC thread or the presentation next week.

There is one little detail. The container images don't actually need to be signed. We just have to make sure that the KBS connects to the KBC so that it can check that the public key matches the host data. We can do this by enabling signatures, which will require policy.json to be retrieved from the KBS. The policy.json file could allow any (unsigned) containers to run. We can provision the KBS with this permissive policy.json by default, which should make for a transparent experience for the client.

Container image metadata could actually serve a very similar purpose. If we get a metadata policy file from the KBS (rather than putting it in the host data), that will also lead to verification of the KBS public key. This could work very similarly to the signature policy, with the KBS being provisioned with a default permissive metadata policy that a client can update.

@jialez0

I think this could help mitigate the issue, but I'm not sure if fundamentally resolves it. A KBS would be able to notice that too many guests have tried to register with it, but the KBS can already do that today. I'm not sure that i would actually be much different from a KBS connecting to a KMS and forwarding the secrets back to the KBC.

@fitzthum

There is one little detail. The container images don't actually need to be signed. We just have to make sure that the KBS connects to the KBC so that it can check that the public key matches the host data. We can do this by enabling signatures, which will require policy.json to be retrieved from the KBS. The policy.json file could allow any (unsigned) containers to run. We can provision the KBS with this permissive policy.json by default, which should make for a transparent experience for the client.

Does this mean that the signature validation here is independent from the image signing or it's just one of the approaches to support Container-KBC-KBS binding (as the workflow I mentioned)?

Container image metadata could actually serve a very similar purpose. If we get a metadata policy file from the KBS (rather than putting it in the host data), that will also lead to verification of the KBS public key. This could work very similarly to the signature policy, with the KBS being provisioned with a default permissive metadata policy that a client can update.

Does this design mean that "successfully fetching a metadata policy from the KBS" implies that "KBS public key in the host data is validated", and the "metadata policy" controls what container can run? If so, without image signing, the metadata policy also needs to capture the image identity (e.g., image digest). Another issue is how does KBS determine what metadata policy to provision, which I recall you mentioned in the thread of the RFC. Also, I'm not sure if it's relevant: would KBS need to know if the policy enforcement succeeds before releasing secret? Let me think about it more while preparing the presentation.

@mingweishih

Does this mean that the signature validation here is independent from the image signing or it's just one of the approaches to support Container-KBC-KBS binding (as the workflow I mentioned)?

I don't quite understand this question. Maybe I can clarify. I think we can resolve the problem of the reusable evidence with two things.

1. Public key of KBS in host data
2. Make sure that containers cannot start without the KBC connecting to the KBS (and the AA checking that the public key of the KBS is the same as the public key in the host data).

It really doesn't matter what we get from the KBS as long as we have established a connection and checked that the public keys match. Today signatures give us an opportunity to do this because if signatures are enabled via the kernel command line (which is measured) then we will request policy.json from the KBS. Again, it doesn't matter what we actually request so the policy file could contain anything. A client should set some kind of policy that makes sense but that is a totally separate question.

We could also use have image metadata policy as the thing that is requested from the KBS. Again, it could be totally empty and it would be fine.

Does this design mean that "successfully fetching a metadata policy from the KBS" implies that "KBS public key in the host data is validated", and the "metadata policy" controls what container can run? If so, without image signing, the metadata policy also needs to capture the image identity (e.g., image digest). Another issue is how does KBS determine what metadata policy to provision, which I recall you mentioned in the thread of the RFC. Also, I'm not sure if it's relevant: would KBS need to know if the policy enforcement succeeds before releasing secret?

I think these questions are asking something slightly different. Like I said, as long as we request something from the KBS before running any containers, we should be fine. This could be a signature policy file or something for the container metadata or both. These questions seem to be more about how we should use signatures and container metadata together and how the KBS should provision them, which is probably more of a topic for the RFC.

Briefly though, I think that we should try not to duplicate functionality between signatures and container metadata. So I think that for maximum security people will use both. Signatures will verify which image is being run and the metadata policy will be a set of rules that the container metadata is checked against before the containers can start. We will probably need to introduce some kind of (unmeasured) workload ID so that the KBS can figure out which policies to deploy.

@fitzthum

It really doesn't matter what we get from the KBS as long as we have established a connection and checked that the public keys match. Today signatures give us an opportunity to do this because if signatures are enabled via the kernel command line (which is measured) then we will request policy.json from the KBS. Again, it doesn't matter what we actually request so the policy file could contain anything. A client should set some kind of policy that makes sense but that is a totally separate question.

Just make sure I understand this correctly:

• Signatures help because enabling the signatures in the kernel command line (a boolean value) ensures that KBC will talk to KBS in the current CoCo image pulling flow. This this is effectively a piggyback and does not really require image to be signed.

• If we want to further lock down what container can get the secret from the KBS, we could use either image signing or metadata policy (otherwise, any pod with KBS's public key in the host data on the same host can get the secret).

@mingweishih

Signatures help because enabling the signatures in the kernel command line (a boolean value) ensures that KBC will talk to KBS in the current CoCo image pulling flow. This this is effectively a piggyback and does not really require image to be signed.

Yeah.

If we want to further lock down what container can get the secret from the KBS, we could use either image signing or metadata policy (otherwise, any pod with KBS's public key in the host data on the same host can get the secret).

I think if we have the KBS public key in the host data, we don't need to bind the individual containers to the evidence. I think the KBS can just send over signature and metadata policies directly without them being part of the measurement (assuming the other mitigations are in place). Do you see any issues with this?

@fitzthum

I think if we have the KBS public key in the host data, we don't need to bind the individual containers to the evidence. I think the KBS can just send over signature and metadata policies directly without them being part of the measurement (assuming the other mitigations are in place). Do you see any issues with this?

No. I was referring to that we need to enable at least one of image signing and metadata policy (in addition to enabling signatures kernel command) so that KBS can release secret after image-rs/kata-agent enforces signature or the policy check successfully. Having the KBS send the policy (policy.json or similar file for the metadata policy) over should be sufficient. If this aligns with your thought, I'll put more details with the part of "KBS passing metadata policy" to the RFC discussion.

@mingweishih

No. I was referring to that we need to enable at least one of image signing and metadata policy (in addition to enabling signatures kernel command) so that KBS can release secret after image-rs/kata-agent enforces signature or the policy check successfully. Having the KBS send the policy (policy.json or similar file for the metadata policy) over should be sufficient. If this aligns with your thought, I'll put more details with the part of "KBS passing metadata policy" to the RFC discussion.

Sounds good.

Having the KBS send the policy (policy.json or similar file for the metadata policy) over should be sufficient.

This would require some notion of pod-identity so that the KBS can send the correct "container metadata policy" to a pod.

@fitzthum

Great article! I really like the bank safety deposit box analogy and would like to expand it to facilitate the discussion about the identity of the containers running inside the protected VM.

We can consider the protected VM with its kernel/initrd measured by the TEE HW as the generic safety box manufactured by a reputable safety box manufacture. As the customer, I don’t need to trust the bank about the quality of the safety box. Instead, I can trust box manufacture, as long as I can confirm the box is from the trusted manufacture, with sufficient anti-tampering build, such as materials, thickness, lock, etc. (TEE attestation, with kernel/initrd measurement). The generic safety box will protect the content I deposit in it from access by the bank or anybody else. But I need a safety box with certain specific features to protect my valuables in it. For example, I want a box with well-padded compartment so I can use it to store my precious antique tea set, and I want another box with fire-proof liner to protect my stock certificates from possible fire incidence at the bank. Those features are accomplished through add-ons to the generic safety box, fitted by the trusted manufacture. The manufacture will fit the box with the add-ons, which might be from a third party, but only add-ons according to the specific order they received. In Confidential Container, we can consider the specific containers running in the protected VM as the safety box add-ons. Let’s envision the bank is offering a service to order the add-ons for me. That’s great for my convenience. I provide a list of the add-ons to the bank. It’s in the bank’s interest to order the right add-ons for my box, to keep me as a happy customer, but it’s possible that due to a processing mistake, an incompetent employee, or malicious intent, they sent a wrong order, and my box is fitted with wrong add-ons. With my requirement to protect specific valuables with specific add-ons in the safety box, I need to not only confirm the box is from a trusted manufacture with sufficient anti-tampering build, but also with the add-ons I specified. When I exam the fitted box, I notice the add-ons do not match my list, so I refuse to lock my valuables in the box. This process of examining the safety box and the add-ons maps to the Confidential Container attestation process, where both the identity of protected VM and the identities of the containers running inside the protected VM should be checked, before the Relying Party (KBS) releases the secret to the protected VM. At this point of the analogy, let’s leave the details of how the container identities are described and examined for later discussion.

The add-on order maps to the container allow list or container configuration policy in the Confidential Container scenario. It’s not a perfect analogy, as the TEE attestation only reflects what containers are allowed to load into the protected VM, not what containers are actually running inside the protected VM. The difference between attesting the allow list and attesting what’s actually running can be left for a future discussion. The container configuration policy need to cover container runtime configuration rules such as entry command, env. variables, and Kata Agent run time interface hardening rules, if those rules are not provided to the protected VM through another mean. In this discussion, I will ignore them for simplicity and focus on container images only.

When I specify the fire-proof liner add-on, I can specify the exact material, thickness, dimension (container image manifest digest, or hashes of container image layers). To avoid the hassle of remembering all the build details of the liner, I can specify instead:

• A brand/product name (signing key the container developer used to sign the container manifest digest or the container image layer hashes),
• A model number (a unique name to differentiate one container from other containers signed using the same key)
• A minimum revision number (security version number of the signed container digest or hashes). I want to specify a minimum revision number because an older revision has a known defect (security vulnerability).

It’s possible that for certain valuables, I don’t care too much about what the add-ons are. If the bank orders fireproof liner, paddings, and decorative accessories to the box, on their own (container configuration policy that allows any container to load, or allows any container signed by a key to load), I’m OK with it. But for many specific valuables, I need to specify the exact add-ons (container image measurement or signing-key/unique-name/minimum-SVN).

In the analogy of ordering add-ons for the safety box, I use a third-party ordering service to handle the ordering process (sending the container configuration policy to the protected VM to enforce). I can use a reputable third-party ordering service (in Confidential Container scenario, a trusted service, for example, the same KBS managing the pod-specific secret, or a different KBS), or just any ordering service, including the service the bank provided, as I will check the add-ons fitted in the safety box anyway.

Now, on to how to check the add-ons of the safety box. Imagining that the safety box manufacture fixes a tampering-proof (say, etched on the cover of the box) receipt of the add-ons fitted to the box, according to the order (container configuration policy) it received, and I trust the manufacturer to not make any mistake in this process, I can simply check the add-ons receipt instead of examining each add-on for all the details. The receipt is a summary of the add-ons ordered and fitted. In the case of using a reputable ordering service (trusted service to provide the container configuration policy), the receipt can simply list the ordering service who issued the order (public key of the service) and the order-id (identity of the pod the container configuration policy is for) to differentiate my order for tea set box and my order for stock certificate box.

For Confidential Container, the receipt maps to the hash of the container configuration policy or the hash of {Service public key, Pod-ID}, bound with TEE attestation through SEV-SNP HostData or TDX/SGX ConfigID, or bound to attestation by integrating the hash in the VM image for the protected VM. Is there a clear advantage of using a trusted service to provision the container configuration policy to the protected VM, compared with the scheme where the cloud service provider sends the container configuration policy to the protected VM, say through an annotation field in the Pod YAML file? It’s not clear. In the safety box analogy, I can simply use the bank’s ordering service, instead of auditing a third-party ordering service to make sure it’s trustworthy or build my own ordering service so I trust it inherently. Through attestation, the Relying Party (the KBS holding the secret) checks the container configuration policy enforced in both schemes.

I'm still thinking over the analogy for encrypted container case. I will post a follow-up if I can come up with a good description.

@bodzhang

Well first of all, if you've got all these stock certificates and an antique tea set, I don't see why you are still coming to work.

This is a great addition. Sorry it took me so long to get to it. One place where the metaphor kind of breaks down is that there is no Kata Agent or Attestation Agent inside the average safe deposit box. Maybe we could say that there is a benevolent gnome living inside the box, but this is a bit weird (and how would you measure the gnome??). I guess we could say that the client takes on the role of the attestation agent when they visit the bank for the first time and inspect the box. Ultimately it's going to be the Kata Agent and other components inside the enclave that enforce whatever policies we implement, so it's kind of a shame that they don't exist in this metaphor.

To me the unsettled question is how that policy is going to get to the Kata Agent and how it's going to be trusted. The two main options seem to be 1) putting the hash of the policy in the host data and passing the full policy through some unmeasured channel. 2) Sending the full policy directly from the KBS. In some ways we could say that the first one is like having the bank or some other service setup the add-ons in advance and give you some signed receipt. The second option might sound more like the client just installing the add-ons themselves when they show up. This makes it sound like the second option is going to be a lot more work for the client, but this is where things start to break down. Since the Kata Agent is going to be enforcing the policy either way both options share the same low cost for enforcing the policy. So the question is really more about if the client wants to examine a receipt that shows them how the safe was installed or if they want to show up at the bank with a list of the feature they want and watch some reliable builder install them. If anything, I think the second option is simpler.

To be clear I think that the approaches end up being roughly the same. If a KBS knows how to validate the policy, then it probably also knows how to send the policy directly to the guest. To me the second option is simpler, more generic, and closer to what we've already implemented.

That said, I think your additions highlight a fundamental question, which is how we want to interact with third parties. Under the surface of all the technical stuff, there are bigger questions about how to deploy this thing and how to create a good user experience. I think that both options are still quite feasible, but I need to think more about what makes sense.

I must ignore something import, but the problems I think significant are

1. How AA can confirm the identity of the KBS.
2. How the KBS can confirm the requester is the AA controlled by the start-up logic of guest, not a tenant-started one. (To avoid containers able to do remote attestation and request for KBS)

How about decoupling the remote attestation attestation and KBS resource retrievement? I mean

• Remote attestation only proves the TEE, software environment (guest firmware, kernel, rootfs containing kata-agent and AA) is prepared. This stage actually does not bring anything corresponding to the tenant defined pod information.
• KBS resource retrievement, especially policy.json. The privilege to request for a KBS should take a token (pod token in the following) as @jialez0 mentioned, which should include the identity of pod, tenant and deploy

In this way, we can divide the start-up process of a pod into following phases.

• Pre-provisioning: The tenant provides different pre-provisioning token to every pod that will be launched via host-data mechinism. The pre-provisioning token is useful to identify different pod instances when they share a common host or measurement.
• Remote Attestation and Provisioning (RA & P) : After the guest boots, it will connect to a Provisioning Service on the tenant side. The PS firstly do the remote attestation via AS check to ensure the hardware and software environment is legit. Then based on the pre-provisioning token, PS can provision:
  • A pod-token including sufficient information about the identity of the pod, tenant. The identity of the pod is related to the policy.json to be requested, and identity of tenant can somehow help a multi-user KBS mechinism.
  • KBS public keys.
• Resource Retrievement. When the kata-agent needs to request for confidential resources, it will use the pod-token to request from KBS.

How can they handle the problems mentioned before? Let me make a table to summurize some of the scenarios I've considered, surely ignore some. Suppose the CSP is malicious

Stage	Attack type	Details	Result	Safe
Pre-provisioning	naive	CSP launches a guest using a wrong pre-provisioning token	can not pass the Remote Attestation and Provisioning for the tenant can check whether the pre-provisioning token exists	Y
Pre-provisioning	Replay	CSP launches a guest using a right pre-provisioning token more than once	can not pass the Remote Attestation and Provisioning for the second time and later on	Y
RA & P	malicious PS & AS / MITM	CSP launches a guest using the right pre-provisioning token to connect a malicious PS & AS to pass the RA	It can not generate a legic pod token to request from a right KBS	Y
RA & P	Replay	CSP launches a guest using the right pre-provisioning token, gets a right pod token and reuse this pod token (by clone the guest VM. It can not extract for the software stack RA ensures it can not do this)	Not harmful. The CSP still can not access other resources in the KBS	Y
Resource Retrievement	naive	CSP uses a fake pod token to request from KBS	Only the PS provided pod token is valid, which could include a signature signed by the tenant. The KBS will validate the signature using the tenant's pubkey.	Y

So far we have seen a few different ideas about how we should handle the identity of the guest. Let me try to summarize the four potential options so far.

1. No identity

This is what we do today. It is the safe deposit box model described in the original post. This works pretty well, but there are some risks regarding the evidence factory attacks. One reason we were drawn to this approach originally is that it is a good fit for SEV(-ES).

2. Client identity

Here we would put something that corresponds to the client into the evidence. This is the kbs-uri proposal that I made above and in the FOSDEM talk.

3. Workload identity

Here we would put something that corresponds to the workload in the evidence. This would allow the evidence to be used independently, which is sometimes referred to as the passport model.

4. Guest identity

Here every guest would have an identity. I think this is what @Xynnn007 is suggesting above.

So which one should we do? I don't know. There are some interesting tradeoffs. It seems like the less generic the evidence is, the more tooling and state is required to manage it. One of the reasons that we originally went with the first approach is that it means that the verifier can be very simple.

3. Workload identity

Here we would put something that corresponds to the workload in the evidence. This would allow the evidence to be used independently, which is sometimes referred to as the passport model.

I think part of the Workload Identity should be a measurement of the TEE Security Policy document we discussed in https://github.com/confidential-containers/documentation/issues/81 . That Security Policy would be:

1. Strict enough to mitigate many of the container escape attacks.

2. Bound to the TEE using HOST_DATA/MRCONFIGID/CONFIGID.

Of course this is not quite as useful on a TEE that doesn't have HOST_DATA/MRCONFIGID/CONFIGID or similar feature. But such a TEE might have other Confidentiality problems too, so perhaps it shouldn't be the highest priority for Confidential Containers.

@danmihai1 Not to spoil next Thursday's meeting, but I think we should go with methods 1 and 3. We can resolve the evidence factory issues for method 1 by revoking access to the PSP. We will implement this in the cc_kbc. This will not support getting attestation reports at runtime, although you will still be able to get resources from a KBS that you connected to prior to starting the container. Offline KBCs (SEV, Z) will still be vulnerable.

We can also support option 3, which will allow for getting a report at runtime and will not require using a KBS during the initial launch flow. Ideally we would provide the workload identity via an extendable measurement like RTMR/vTPM, but in the short term we can implement something with the host data. See my comment on the other issue about how to best implement this. I do not think we should add policy enforcement to the Kata Agent directly. Rather we should introduce a more general logging functionality that can be used with vTPM/RTMR and can also be parsed by an internal verifier using the host data. Specifically, we can create a new passport KBS that validates the log before returning the evidence to the workload. Like I said this is what I described in the other issue. I have been thinking about it more and I think this is the best approach. I think this will be compatible with most of your existing policy code. This will not be vulnerable to evidence factory issues.

@Xynnn007 , in your proposal, is the "pre-provisioning token...that will be launched via host-data mechanism" basically an ID that refers to a specific pod (for example, Pod YAML file name) and pod-token the "allow list" of containers (for example, acceptable signing keys of signed containers if each container is signed with a unique key, etc) to be enforced by the software environment reflected in the TEE measurement?

@fitzthum , I guess it’s possible that I am missing something important about the RTMR/vTPM approach you mentioned. To help our discussion on Thursday, here are some quick thoughts that we should address:

a. RTMR/vTPM work reasonably well for measuring a short and predictable set of actions that have been performed already before attestation - e.g., for measuring the OS Loader, Kernel and initrd before attestation. However, RTMR/vTPM can be too brittle for:

• Measuring a complex set of actions
• Measuring actions that might take place in a non-deterministic order
• Measuring actions that didn’t take place yet

b. Example1: a customer wants to allow any of the following environment variables for a container:

"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/bin:/sbin"
"PATH=/usr/sbin:/usr/bin:/sbin:/bin"
"PATH=/usr/sbin:/usr/bin:/sbin:/bin:/mybin"

How would the customer come up with a single vTPM measurement that allows any of these PATHs?

c. Example2: Container1 starts and performs attestation. Container2 starts and performs attestation. Will this customer have to attest two different vTPM measurement values, one for each instance of attestation? The same can be achieved with a single policy that covers the functionality of both Container1 and Container2. A single measurement of the policy document is enough for both attestations.

d. Example3: Container1 starts and performs attestation. Container2 starts and doesn’t perform attestation. How can Container1’s Relying Party be confident that Container2 is not rogue?

e. Example4: Container1 starts and initiates attestation. Other actions that update the vTPM measurement take place in parallel with Container1’s attestation. Which value of the measurement will Container1’s attestation use? The old value or the updated value?

f. A customer might want to delegate the responsibility of authoring a fragment of their security policy. For example, they might trust the K8s operator to author the policy for the pause container. They don’t want to change the measurement being attested every time when the K8s operator changes the environment variables of the pause container. A policy engine like OPA allows customer’s policy measurement to remain constant, while dynamically loading another policy file specific to the pause container. The trust in the pause container's policy file could be established by verifying that it's signed by the K8s operator. Is there a good way to reduce friction by delegating trust in a vTPM world?

g. My understanding is that vTPM is not very popular across Cloud infra providers.

h. My understanding is that vTPM would typically be implemented under Cloud provider’s control. If that’s the case, the vTPM measurement would be generated outside the confidentiality boundary of the workload owner.

@danmihai1

a-e:

I don't think validating a log is any different from validating a policy in real time. If you can evaluate all of the kata agent's actions against a policy, why wouldn't you be able to validate a log of all the agent's actions against the same policy? To validate the TPM PCR values, you simply replay the log, validating each entry against the policy. There are no problems with non-determinism. You get exactly the same information as you would validating a pre-provisioned policy inside the guest. This validation will happen when secrets are requested. At this point every container in the pod will have been setup and started. The log will reflect this. If for some reason a container has yet to start, that will be visible in the log. Depending on implementation it's possible that different containers will have slightly different PCR values, but these can all still be validated by replaying the log. I guess this might be a bit inefficient, but it's not really an issue otherwise.

f:

It's no problem to get policy information from different places. The attester can simply look up policy from the CSP before validating the log or they can just blindly accept any portion of the log (which isn't really that different from getting a policy from the CSP). You can use OPA to do this validation. You can get a signed version of the CSP policy. It's really all the same.

g,h:

I believe that Azure currently has a preview offering of a (semi-)secure vTPM that is emulated inside the guest via the SVSM. I expect this approach to become more common. Traditional vTPMs are not suitable to confidential computing, but these new approaches are fine. I believe that Jiewen is giving a talk at OC3 about how to do something similar for TDX.

Anyway, I really think that most of the code will be the same for the log based approach. The log just adds a layer of indirection. There are a few true differences, though. For one thing, the log-based approach does not require anything to be provisioned ahead of time on the worker node. I see this as an advantage. A possible downside that you allude to is that the log-based approach does not give you any guarantees about what might happen after attestation has taken place. I need to think about this a little bit more, but in general I think it becomes the responsibility of our agent api hardening. Once the container has started, there shouldn't be much that can change.

Now even if you hate TPMs and never want to use them (which is somewhat fair), I think the internal log validator approach is reasonable and will end up looking very similar to what you originally proposed.

@fitzthum Thanks for explaining again! I missed initially the idea of replaying the detailed log during attestation.

I will think more about this, but currently I am down to this other set of concerns about the RTMR/vTPM + CoCo Log proposal:

1. Containers in a pod can be re-started, and livenessProbes can be executed after attestation too. Therefore, these actions wouldn’t be included in the attestation log. (This is the item you already mentioned that you’ll think more about)

2. Requires a custom CoCo protocol for providing the log to the Verifier and/or Relying Party.

3. Requires a custom CoCo implementation for replaying the log in the Verifier and/or Relying Party service.

• Checking the value of HOST_DATA/MRCONFIGID/CONFIGID in a Verifier and/or Relying Party service is more common, since those are fields intended for configuring the TEE.

4. After a successful hack of the guest VM software, it will be harder for the K8s operator to help with forensic analysis:

• It helps to establish which actions were allowed in the guest VM, and which ones were not allowed, by matching the value of HOST_DATA/MRCONFIGID/CONFIGID with a known policy file.
• For example, if the policy allows privileged containers, the next step might be to examine those containers.
• If the policy is obviously wrong, that could point to improvements needed in the policy generation and testing tools, user education, etc.
• The compromised guest VM software might be able to cover its tracks by sanitizing the vTPM and CoCo Log, but wouldn’t be able to change HOST_DATA/MRCONFIGID/CONFIGID.

5. Requires vTPM or RTMR. I believe vTPMs are not popular across Clouds – although I agree they are popular in Azure.

6. A customer already trusts the TEE. Trusting a vTPM implementation too weakens the confidentiality promise.

@Xynnn007 , in your proposal, is the "pre-provisioning token...that will be launched via host-data mechanism" basically an ID that refers to a specific pod (for example, Pod YAML file name) and pod-token the "allow list" of containers (for example, acceptable signing keys of signed containers if each container is signed with a unique key, etc) to be enforced by the software environment reflected in the TEE measurement?

@bodzhang
Yes, pre-provisioning token only helps the tenant to distinguish which pod is doing remote attestation, like the following

// pre-provisioning token inside REPORT_DATA/HOST_DATA/MRCONFIGID/CONFIGID
{
    "pod id": "xxxxx",
}

A pod shows a combination of a set of containers, thus I think it has the same meaning as workload. But one pre-provisioning token can only be registered once. If duplicated remote attestation report with pre-provisioning token comes, the PS can reject to prevent evidence factory attack.

pod-token is provisioned by tenant after remote attestation. Knowing which pod is being attested, the tenant can give enough but limit privileges inside the pod-token for the pod to access limited resources after the remote attestaion. like a token granted access to repo kbs://example.org/alice/* or something else. The pod-token cannot be forged because it could carry the signature of the tenant which will be verified by the KBS. For example, after the privisioning, the pod might get a token conceptually like the following

{
    "sig": "<signature from the tenant for this pod-token>",
    "signer/pubkey": "<pubkey of alice>/user id registried inside KBS",
    "permissions": {
        "example.org": [
            "alice/cosign-key/*",
            "alice/machine-learning-data-set-1/*",
            ...
        ]
    }
}

This example supposes the KBS support different users, have an out-of-band registration before.

@danmihai1

@fitzthum Thanks for explaining again! I missed initially the idea of replaying the detailed log during attestation.

I will think more about this, but currently I am down to this other set of concerns about the RTMR/vTPM + CoCo Log proposal:

1. Containers in a pod can be re-started, and livenessProbes can be executed after attestation too. Therefore, these actions wouldn’t be included in the attestation log. (This is the item you already mentioned that you’ll think more about)

afaik restarting containers isn't very common, but I agree that there are some cases we'll need to think about with timing.

2. Requires a custom CoCo protocol for providing the log to the Verifier and/or Relying Party.

I think we will want to add an endpoint to the KBC that a workload can use to get the evidence. This will return the attestation report and the log and some metadata. Exposing attestation reports to the guest directly means that a workload will be platform specific. We should be able to generalize that with a KBC endpoint. It's true that this format might not be immediately compatible with existing non-standard or proprietary verifiers, but that should mainly be a question of conversion.

3. Requires a custom CoCo implementation for replaying the log in the Verifier and/or Relying Party service.

Yeah, we would need some code to replay the log.

• Checking the value of HOST_DATA/MRCONFIGID/CONFIGID in a Verifier and/or Relying Party service is more common, since those are fields intended for configuring the TEE.

I don't know if any approach is particularly common yet.

4. After a successful hack of the guest VM software, it will be harder for the K8s operator to help with forensic analysis:

If any secrets were provisioned, the KBS would have the log, which should be handy for forensics. In a debug scenario you can use the debug console to read the log, which would be very useful.

...

6. A customer already trusts the TEE. Trusting a vTPM implementation too weakens the confidentiality promise.

If the vTPM is properly implemented (see AMD's linux-svsm), it will not change the trust model. The vTPM is part of the TEE. In TDX RTMRs are part of the standard measurement flow and don't change the trust model at all.

Here every guest would have an identity. I think this is what @Xynnn007 is suggesting above.

Not really. I want to say there two or three identities

• Workload identity. (Which I used pod to refer, because pod is a set of containers in Kubernetes context)
• TEE identity. This will be faked by evidence factory attack
• Tenant identity. This can be included in the workload identity

TEE identity only ensures the confidentiality of the execution environment, however the workload identity + tenant identity finally determine the permission to access the KBS.

I am 2 years late to this (very interesting) party, and I haven't had time to look through all the comments, but... I have recently been thinking about key release and trust bootstrap in a different context and wanted to contribute my two cents.

The interposition problem where a rogue key-release service (a malicious KBS in this case) interposes and can therefore steal the sealing key, can be avoided as long as the identity of the KBS (i.e., URI endpoint and CA cert) is measured during boot. This way, the successful release of the sealing key, which is predicated on the verification of the attestation evidence (which, in turn, contains the KBS identity measurement), becomes the implicit proof of the identity of the KBS for the KBC.