While people expect a sort of battle between bare metal cloud and virtualization, our users are actually quite happy to use both at the same time and, more recently, use them together.
The most common use case our customers ask for is VMWare vSphere deployment automation. The vision is that you get the HCI's promise of entirely software-defined compute, network and storage but taken a step further by having the VMWare vSphere cluster actually be a TENANT of a larger multi-tenant private bare metal cloud.
The idea is that:
1. You can shift resources from one tenant to another, which reduces waste.
2. You can deliver a fully working cluster in less than 30 minutes, so you don't need to pre-provision anything.
3. You can deploy the cluster anywhere in the world (where you have a data center).
4. You can remove servers from one cluster and move them to another tenant or to a Kubernetes cluster.
5. Done right, container-based applications are much better at scaling to match the needed capacity precisely thus, over-provisioning is less of an issue, so you can use the excess capacity for something else.
Why not put everything INSIDE a giant VMWare cluster (or clusters) and get the same benefits (and more, like micro-segmentation vMotion)?
Performance considerations aside, in an ideal world, I guess you could. However, most enterprises and service providers have and will probably continue to have a zoo of technologies to deal with: Kubernetes, OpenStack, and OpenFaas or other serverless platform services but also SQL and NoSQL databases, big data (think Cloudera) clusters and more.
The ability to shift resources between these projects (which I refer to as infrastructure agility) is essential in supporting businesses to be agile themselves thus, the vision of some of our customers of looking at the least common denominator of all these platforms and trying to automate that is certainly spot on.