Recommended
Recommended
More Related Content
Similar to Identity in OpenStack: the Challenge of Multitenancy
Similar to Identity in OpenStack: the Challenge of Multitenancy (20)
Recently uploaded
Recently uploaded (20)
Identity in OpenStack: the Challenge of Multitenancy
- 1. Identity in OpenStack Colleen Murphy cmurphy/@cmurpheus The Challenges of Multitenancy
- 2. who am I? Cloud Engineer at SUSE OpenStack contributor Project Team Lead for OpenStack Keystone cmurphy/@cmurpheu s
- 3. what is this talk about? introduction to keystone consequences of multitenancy cmurphy/@cmurpheu s
- 4. The Principles of Cloud multitenancy self-service cmurphy/@cmurpheu s
- 5. multitenancy collaboration in isolation secure sharing of physical resources cmurphy/@cmurpheu s
- 7. what is OpenStack? open source cloud platform Infrastructure as a Service virtualization cmurphy/@cmurpheu s
- 10. discovery entrypoint into the cloud bridge between components cmurphy/@cmurpheu s
- 11. $ openstack catalog list +-------------+----------------+---------------------------------------------------------------+ | Name | Type | Endpoints | +-------------+----------------+---------------------------------------------------------------+ | nova | compute | eu-central-1 | | | | internal: https://198.51.100.42:8774/v2.1 | | | | eu-central-1 | | | | public: https://public.cloud.example.de:8774/v2.1 | | | | | | glance | image | eu-central-1 | | | | public: https://public.cloud.example.de:9292 | | | | eu-central-1 | | | | internal: https://198.51.100.42:9292 | | | | | | swift | object-store | eu-central-1 | | | | internal: https://198.51.100.42:8081/swift/v1 | | | | eu-central-1 | | | | public: https://engcloud-swift.cloud.suse.de:8081/swift/v1 | | | | | | keystone | identity | eu-central-1 | | | | internal: https://198.51.100.42:5000/v3/ | | | | eu-central-1 | | | | public: https://public.cloud.example.de:5000/v3 | | | | | | cinder | volume | eu-central-1 | | | | internal: https://198.51.100.42:8776/v3 | | | | eu-central-1 | | | | public: https://public.cloud.example.de:8776/v3 | | | | | | neutron | network | eu-central-1 | | | | public: https://public.cloud.example.de:9696/ | | | | eu-central-1 | | | | internal: https://198.51.100.42:9696/ | | | | | +-------------+----------------+---------------------------------------------------------------+ cmurphy/@cmurpheu s
- 13. authentication token service • fernet • JWT • SAML2.0 cmurphy/@cmurpheu s
- 15. keystone-native authentication password application credential totp (MFA) cmurphy/@cmurpheu s
- 18. authorization "scoped" role-based access control role definitions owned by each component cmurphy/@cmurpheu s
- 19. roles and policies role names created in keystone role meaning defined by policy rules cmurphy/@cmurpheu s
- 20. # List all servers # GET /servers "os_compute_api:servers:index": "role:admin or project_id:%(project_id)s" # Show a server # GET /servers/{server_id} "os_compute_api:servers:show": "role:admin or project_id:%(project_id)s" # Create a server # POST /servers "os_compute_api:servers:create": "role:admin or project_id:%(project_id)s" # Delete a server # DELETE /servers/{server_id} "os_compute_api:servers:delete": "role:admin or project_id:%(project_id)s" # Update a server # PUT /servers/{server_id} "os_compute_api:servers:update": "role:admin or project_id:%(project_id)s" # Reboot a server # POST /servers/{server_id}/action (reboot) "os_compute_api:servers:reboot": "role:admin or project_id:%(project_id)s" # Start a server # POST /servers/{server_id}/action (os-start) #"os_compute_api:servers:start": "role:admin or project_id:%(project_id)s" # Stop a server # POST /servers/{server_id}/action (os-stop) "os_compute_api:servers:stop": "role:admin or project_id:%(project_id)s" cmurphy/@cmurpheu s
- 22. assignments user A has role B on project C cmurphy/@cmurpheu s
- 24. "Not only does the entire system need to be aware of identities, roles, and assignment, but it also needs to properly account for scope of specific operations." http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/policy-security-roadmap.html cmurphy/@cmurpheu s
- 25. problem #1: bug 968696 security hole that violates the Principle of Least Privilege https://bugs.launchpad.net/keystone/+bug/968696 cmurphy/@cmurpheu s
- 26. problem #2: no discovery no way to answer the question "what role do I need for this operation?" cmurphy/@cmurpheu s
- 27. problem #3: no self-service role definitions controlled by cloud operator cmurphy/@cmurpheu s
- 28. keystone beyond OpenStack identity and authentication • ceph • kubernetes authorization? • ... discovery? • ... cmurphy/@cmurpheu s
- 29. what's next? proxy identity provider system scope everywhere quotas cmurphy/@cmurpheu s
- 30. The Principles of Cloud multitenancy self-service cmurphy/@cmurpheu s
- 31. The Principles of Cloud multitenancy collaboration self-service automation cmurphy/@cmurpheu s
Editor's Notes
- This talk is about the identity component service in OpenStack, keystone, and its role in managing a multitenant computing environment like OpenStack. <number>
- I'm a developer at SUSE working on the Cloud team, I've been working at SUSE for about two years now. We build our Cloud product around OpenStack, which is one of the largest open source projects in the world. I'm also the PTL or Project Team Lead for the keystone project in OpenStack, which is a community-elected position and not part of my job title at SUSE, and that's why this talk will center on the keystone project. <number>
- Since I've been working on keystone for a few years now I wanted to give a talk that gives a broad introduction to the keystone project, and will try to make this an honest look at OpenStack and keystone and not just talk about why it's my favorite project. Beyond that, I'd like to take a step back from this project and talk about multitenancy and why it's a hard problem. <number>
- To set the stage, these are what I think are the founding principles of a Cloud and what makes cloud different from just virtualization. First, cloud infrastructure is multitenant, and second, cloud infrastructure is self-service. <number>
- To define that more - multitenancy is the abstraction that lets us collaborate on virtual resources within a team or organization while also allowing teams to securely share the underlying physical resources between teams in isolation from one another and without disrupting one another. Sharing resources in isolation from other users isn't a new idea, any kind of multi-user Unix system is supporting a form of multitenancy. What's more interesting about the OpenStack approach is that it is not just about giving individual users access to a system, it is about allowing groups of users to collaborate on a logical section of the system. And so in the cloud take on multitenancy, a tenant is not just a user but a team, and the whole thing is organized around teams of users. And moreover, a user could be a member of multiple teams and so would need access to more than one tenant. <number>
- Self-service then means users having full control over their applications, and having your infrastructure be fully automatable and discoverable. It should never require filing a ticket with your operator. Your applications should have enough information about the system to make their own decisions without a human's involvement, for example when it comes to autoscaling. <number>
- Before I can talk about the identity component of OpenStack, I need to explain what OpenStack is for anyone who isn't familiar with it. OpenStack is an open source cloud platform. It's usually used as a private cloud for cases where it's more cost effective to run your own infrastructure in your own datacenter, or you're concerned about data privacy, but you still want the agility that cloud gives you, or there are also many different public cloud offerings running OpenStack, which is convenient because you can use the same APIs to work with multiple different providers and you aren't locked into one public cloud provider. Another way to look at it is as Infrastructure as a Service, where you can request servers or block storage or networking on demand and not need to ask your IT team to buy and install a new server or boot up a VMware node. What OpenStack is not is virtualization, virtualization is a lower layer below cloud. <number>
- An important thing to know about OpenStack is that it is huge, and it is developed and operated as distinct building block component services, it's not a product that comes already put together and working out of the box, it's up to cloud operators to pick out the pieces they want and put them together. <number>
- With that in mind, let's talk about the keystone project. The keystone project implements the Identity API for OpenStack, which means it is responsible for user management as well as for authentication and authorization. Since it is central to all the other components in OpenStack, it also acts as a discovery service. <number>
- I'll talk about the discovery functionality first. It's kind of the odd one out, as it doesn't exactly fit into keystone's role as the identity service. Discovery is critical to the way OpenStack works because OpenStack is developed as separate building block components, developed and operated independently from each other. The reason it's part of keystone is that keystone is central to all of these components and so it is in the unique position to provide an entrypoint into the cloud and to act as a bridge between these disjoint components. <number>
- The way it operates is very simple: the component services register themselves in the keystone catalog. This allows clients to only have to know the keystone URL in order to be able to discover what other services are available in the cloud and at what endpoints and in which regions, and from there can discover what API versions are available and start negotiating how to talk to the service they are interested in. So this discovery mechanism allows client applications to autonomously interact with a cloud without previously knowing anything about it, and even to allow the client to interact with different clouds which may be configured differently and have different components installed. <number>
- The most obvious use of keystone is as an identity management service. It can do identity storage, where the user information and password hashes are stored in a database and managed directly by keystone. It can also act as an identity broker, where user information is stored in an LDAP directory and proxied through keystone so that your organization's LDAP can still act as the authoritative identity source but users interacting with keystone have the same experience as though they were local keystone users. Keystone is the identity service for OpenStack but in fact using it as an identity management system is completely optional, and what it really excels at is as an authentication service. <number>
- Authentication here means proving your identity in some way. Keystone's real power is as a token service: it supports a multitude of authentication methods, all of which result in a token. The client can then present the token to another services as proof that they are authenticated, which means that the other service needs to understand and validate the token. We support two different token formats: the fernet format is a specification invented by Heroku and basically not used outside of keystone, and the new format is JSON Web Tokens which is an accepted open standard. Keystone can also be used in a limited form to generate SAML assertions. <number>
- This is a visualization of how a request would go in OpenStack using keystone for authentication: the user's client needs to make two requests: the first is to request a token from keystone and authenticate with the user's credentials, and then in the second request the client passes the token to the service they are trying to access. We have a middleware layer that intercepts the request and validates that the token is genuine, and at that point the service can respond with the resource the user was requesting, which is typically something like a virtual machine. <number>
- There are a lot of different ways you can authenticate with keystone. The standard way is with normal password authentication, where either the user's password is validated against the hash in the keystone database or proxied to the LDAP service for validation. Another way is with application credentials, which are kind of like API keys and allow you to delegate a narrower level of authorization to your applications and not need to give your password to your applications. We can also support time-based one-time passwords which of course is most useful when doing multi-factor authentication. <number>
- Keystone can also lean on web server auth modules for authentication, in which the user stored in SQL or LDAP and keystone uses the REMOTE_USER variable coming from the web server to grant tokens to users authenticated by an external source. Some examples are using X.509 client certificates with the mod_ssl Apache module, or using kerberos, or even something like basic auth should work. <number>
- The exciting part of keystone is federated authentication. It's an extension of external authentication, but in this case the user need not be stored in a SQL or LDAP backend local to keystone, although they still could be. Rather than keystone having direct access to the user's attributes, the user authenticates with an external identity provider supporting a federated authentication protocol like SAML or OpenID Connect, and then an assertion is generated that provides keystone with the user's attributes and proves they are authenticated. Keystone then uses a mapping to translate the user attributes from the external identity provider to local attributes within keystone, and in this way we can achieve a federation of identity across multiple clouds or multiple systems. <number>
- So keystone does a pretty good job of identity management and user authentication. Authorization is the next step, which is about granting the right level of access to authenticated users, and this is where things get complicated and where keystone and OpenStack don't do very well. This is where multitenancy comes back into play. Keystone doesn't need to worry only about users logging into OpenStack but about users logging into a tenant and having the permissions they need to do operations within the tenant and nowhere else. This is what we've dubbed "scoped" RBAC, since regular role-based access control doesn't fully describe the tenancy component of authorization. This is all made harder by the fact that with the way we've designed OpenStack, the meaning of a role in keystone is set by and owned by the individual service components, not by keystone. <number>
- To explain more: roles are just names in the keystone database. What a role means is defined by policy rules which are owned by component services. This is a side-effect of how OpenStack is developed and operated as disjoint component services: the other services are inventing new APIs every day, keystone has no way of knowing what they are and what their access level should be, and so the defaults are set by the service itself and the overrides are configured by the cloud operator and not discoverable to the user. <number>
- This is an example of the policy rule configuration, it has its own syntax to specify the rule name and what roles and scopes are allowed for a given operation. <number>
- A user can't just have a role in OpenStack, they need to have a role and a scope. The most basic scope we have is called a project, which is essentially a tenant. A project is the abstraction that allows us to group virtual resources, like VMs or disk volumes, together and designate access to each grouping on an individual basis. A domain is like a top-level project, but it can also contain users and projects so that users within the domain can be self-organized. The system scope is a concept we've only introduced recently, and it's the scope you would use if you don't want to do multitenant operations but instead need to manage something that is global across the entire cloud or should be hidden from tenant users, like managing a compute hypervisor. <number>
- Role assignments are what put the two things together. Most non-multi-tenant systems would just say "user A has role B" and that defines what they are allowed to do globally within the system. OpenStack is different because we have this scope which defines what group of resources a user has permissions to operate on. So in this case you have to say "user A has role B on project C". <number>
- With the authorization component, this is a more complete visualization of how a request works in OpenStack: the client needs to request a scope when they are doing their authentication with keystone, and in return they get a scoped token. They pass that to the service, which has the token string translated into a token context, which gives the service the information about the user as well as the scope they are in and the role they have for that scope. Then based on that information, the service itself is responsible for deciding whether the user is authorized to do whatever they are trying to do, which it does using a policy library with decorators on the API routes. If the policy checker passes, then the service can grant the resource. <number>
- And here we have the crux of the issue, where OpenStack and keystone were designed poorly, because components were not and are not accounting for scope in their policy rules, nor did we have a scope that could describe an actual global system-level operation. <number>
- The first problem is this longstanding issue, which we've documented in this bug report but is really an OpenStack-wide design flaw, where most component services only account for a role and don't account for scope, which means that a user having an admin role on one project meant they had admin access to all projects. This was because there were some operations that were not tenant-specific, but until now we didn't have a scope that could accurately describe those kinds of system level operations. Keystone supports this type of scope now, but we need to change all the other components to understand it, plus re-train users and re-write their applications to understand it. <number>
- Another problem is that the policy rules are managed in server configuration by the cloud operator, so you could have a role but have no idea what it means until you try an operation and either succeed or fail, so we're violating that principle of discoverability here. A failed authorization will at least tell you the policy rule that was used and that can be used to talk to your operator about getting the right role. <number>
- And so not only can tenant users not know ahead of time how a role is defined, they also can't define roles themselves, which means an administrator of a team or organization that is using the cloud still needs to rely on the cloud operator to create new roles for them, even when they otherwise have control over the users and projects in their domain. The operator has to create the role by updating the policy configuration across every OpenStack component, which is difficult to do. <number>
- Keystone is already used as an identity and authentication service for ceph and kubernetes. With the introduction of the JWT and future advancements in SAML support we could see even further adoption of standalone keystone for identity and authentication. But for authorization, the future is fuzzy because not all applications outside of OpenStack really have or need this kind of multi-tenant authorization model, we wouldn't know how to apply this "scoped RBAC". Or they may already have some existing model of multitenancy, which is the case for both ceph and kubernetes - ceph has buckets, kubernetes has namespaces - which doesn't map 1-to-1 with keystone's modeling. Plus, because of the way we've delegated policy management to OpenStack components, it's unclear how roles would be defined in a non-OpenStack system. It's just very hard to map different authorization models to one another and would need to be done on a case-by-case basis. Similarly for discovery, it's unclear how and whether this would ever be useful outside of OpenStack, and there are other discovery mechanisms already available. <number>
- We want to build our successes for keystone as an identity management and authentication service by building more functionality into it and using it as a proxy identity provider which could be used to translate authentication information from many other external identity providers into a common format, and we want to explore using JSON Web Tokens in this area which should help make it a more flexible service to use with other applications outside the OpenStack ecosystem. To address the authorization issues, we will be working with the OpenStack community to bring the system scope to all the service components and have them use it in their default policies. We're also working on bring the management of resource quotas under the keystone umbrella instead of having each service enforce it themselves, which would help bring a more consistent multitenant user experience to OpenStack as a whole. Beyond that, we haven't started to talk about the right way to expose role definitions to end users. We had thought about trying to move the authorization enforcement point under keystone rather than under the individual services but it was too big of a change and the design we had couldn't really cover every use case that was out in the wild. And we haven't discussed yet how we can map our authorization model to other applications outside of OpenStack in a consistent way that won't end up being a one-off for every application. <number>
- When discussing the downturn of the OpenStack hype bubble, someone asked me once whether OpenStack would survive if big companies pulled out of supporting it, if it became a hobbyist's open source project. <number>
- When discussing the downturn of the OpenStack hype bubble, someone asked me once whether OpenStack would survive if big companies pulled out of supporting it, if it became a hobbyist's open source project. The answer is yes, but it is not the kind of project anyone would work on by themselves. OpenStack is all about collaboration. It's how we do development and it's the reason we make the software. As long as there is a need for collaborative computing, we will need to be building and improving ways of automating our infrastructure so that we can continue the pace of innovating more exciting applications on top of it. At the center of it is keystone, and I personally find a lot of joy in working on making keystone better at supporting team collaboration and making it easier to write automation for your infrastructure. <number>
- If you want to get involved and help us solve these problems, please join us on freenode or the mailing list. Thanks for listening and enjoy the rest of your Sunday! <number>
- <number>