As is the case for many people working in the security industry, the last
few months of my life have been focused on dealing with people wanting to
use LLMs everywhere. From an enterprise security perspective that’s not an
inherent problem - what’s more of a problem is that people want those agents
to have access to resources like their calendar and email and so on, and now
we have somewhat non-deterministic agents that seem very enthusiastic to
achieve what you asked whether that’s a good idea or not, and we’re
combining this with credentials that give them access to sensitive data, and
leaving those credentials on disk where they can be committed into git repos
or exfiltrated to some other service to make use of them on the agent’s
behalf or well just any other number of things, at which point your CEO’s
email is suddenly readable by everyone and you’re having a bad day.
As I mentioned in my last
post, pretty
much every strong mechanism for keeping credentials in place is just not
supported in the wider world. We can imagine a universe where agents use
hardware (or at least hypervisor) backed certificates to obtain credentials
and any that end up leaking are worthless as a result. But, sadly, that’s
not an option for most people using existing identity providers. The state
of the art is that you use the device code
flow and a human authenticates and
the token ends up back inside the agent environment and then it proceeds to
do whatever it wants with it and you just hope that you wake up the next
morning without an awful infoleak occurring.
(An aside: I do not like the device code flow as used in enterprise
environments, and I never will. The identity provider doesn’t have a real
opportuity to inspect the security posture of the system asking for the
token, and as a result some identity providers will restrict tokens that are
issued in this way. The common alternative of doing stuff using a more
standard flow and having a redirect URI pointing at localhost works fine for
local systems and is a pain for remote ones, even if you can commit crimes
with SSH forwarding. I’m going to suggest something that I think is better,
and you are free to disagree)
I’m not in a position to get every identity provider and service provider to
change their security posture, so I’m somewhat stuck in terms of the tokens
they’re willing to issue me - largely either JWTs or opaque access tokens,
with no support for any mechanism of binding that token to an instance. The
token that’s going to have to be provided to the remote service is something
I have little influence over. But that doesn’t mean I can’t influence the
token that lands inside the agent’s environment. I can issue a placeholder
token to the agent, and force it to communicate via a proxy that swaps out
the placeholder for the real thing. The worst the agent can do is exfiltrate
the placeholder token, and as long as malicious actors don’t have access to
that proxy, it doesn’t matter - nobody else can do anything with the
placeholder.
This isn’t a terribly novel insight, and it seems like almost everybody has
reinvented this on their own. But a lot of these implementations involve you
somehow obtaining the real token in advance and then pasting that into
something that generates a placeholder that you provide to your agent
environment somehow, and it’s all a bit clunky and awkward, and it also
means that you need to deal with something that keeps track of the mapping
between placeholders and real tokens and oh no we’ve just invented a secret
store, and if you want this to work at scale and reliably you’re just
invented a high availability distributed secret store, and a lot of people
who’ve read that are now shaking their heads and reaching for gin. Can we
simplify this, and improve security at the same time? I think we can!
Remember when I said “as long as malicious actors don’t have access to that
proxy, it doesn’t matter”? What if they do? What if they compromise one
machine inside your environment and are then able to email a bunch of
employees and convince their agents to send more tokens back to them and
then delete the email before a human reads it? Now you have someone inside
the wall with access to those tokens, and presumably with access to the
proxy, and now they can be anyone whose agent was gullible enough to think
sending them a token was a good idea. This isn’t good!
So, I thought for a while, and I came up with a new idea. We can have a
broker service that obtains credentials for us. We can run that centrally,
away from the agents. A client in an agentic environment can request a
token, and that can result in a URL being generated and the user being
directed to open a URL in a browser and authenticate. When the user
authenticates, the authentication flow redirects the confirmation back via
the broker, and the broker obtains the real auth token. The obvious thing to
do now would be to return the auth token to the client in the agentic
environment, but we don’t do that. Instead, we mint a new JWT, and add a new
claim - one that contains an encrypted copy of the token. In the process we
can copy over all the original claims, because those aren’t secret - and now
even if the client inspects the token to figure out what access it has,
it’ll get a correct answer. We sign the new token with our own signing key,
and pass that back to the client. The client now has a legitimate JWT that
is utterly useless, because the signature isn’t trusted by anyone other than
us.
How does it use it? It makes an API request via a proxy, including the new
token in the Authorization: header. The proxy verifies the signature on the
token, and then decrypts the original token and swaps out the fake token for
the real one. The remote API sees what it expects, and everyone is
happy. There’s never a real token in the agentic environment, but also we
don’t need to store anyting anywhere. The only state is the encryption keys,
and those can be injected into the environment at startup. You need to
scale? Just start more of these processes. You need to support multiple
availability zones? Just start more of these processes in different
places. No persistent data is ever held in the broker or the proxy. You
don’t need to care about distributed databases or secret stores.
This felt wonderfully elegant and I felt smug about coming up with a better
idea, and then I went to a bar earlier this week and sat down to read RFC
8705 and the guy next to me
saw that over my shoulder and asked what I was reading and I explained why I
was interested and we talked about agentic identity and then he mentioned
that fly.io had something that sounded very
similar and I read that and gosh yes
it is very similar, so damn you fly.io for stealing my ideas 3 years before
I even had them. Anyway. Now I need to do better.
Remember that there’s still a risk around anyone who has access to the proxy
having access to the encrypted keys? We can remove that risk as well. It’s
not uncommon for agentic environments to have an identity issued via
something like SPIFFE, at which point they have a
client certificate. You can probably guess where I’m going with this. If we
require that an agent present a client cert to the broker when requesting a
token, we can embed a representation of that client cert into the token we
mint. The proxy can then require mTLS for the client connection, and can
verify that the presented certificate matches the one represented in the
token. If it does then whoever’s using the token has access to the private
key associated with the environment it was issued to. If we then ensure that
the private keys backing these certificates are either hardware or
hypervisor backed, and as such tied to a specific instance, we now have a
high degree of confidence that the token can only be used in its intended
environment. Even if our identity provider doesn’t support RFC 8705, we can.
This is fairly straightforward where you’re using a platform where your
identity provider is also the environment that’s consuming your tokens, and
more annoying for third parties. The broker potentially needs some amount of
third party vendor knowledge to make that work for everyone. This is even
more the case where login isn’t via your identity provider (thanks, github),
but none of this is insurmountable - just annoying. And where vendors issue
opaque tokens rather than JWTs, this still isn’t a problem; we can just mint
a new JWT that includes the opaque token as an encrypted claim, and include
the same certificate binding. The opaque token ends up being the thing
that’s presented to the third party, but only after we’ve verified the mTLS
binding.
In an ideal world none of this would be necessary - someone would spin up a
new agentic environment, a user would prove their identity, and a
certificate embodying that identity would be issued to the environment with
a private key that can’t be exfiltrated. That certificate would be
sufficient to obtain new certificates associated with the same private key,
and we could still bind that into mTLS identity. This would be much simpler,
but browsers don’t support it, so it’s not likely to happen any time soon.
Anyway. Even if we can’t have the best thing, we can do better than we are
at the moment, and also it would be lovely if we could standardise on this
rather than have everyone build their own thing. The end.
Log in
