Server Manager (as of Win2012 at least) indeed uses PowerShell under the hood for many, if not all, operations. If you run Get-PSSessionConfiguration on a Win2012 server, you’ll even see the Server Manager workflow endpoint that Server Manager uses.
But you misunderstand what that means. PowerShell isn’t a CLI, actually. “Commands” are well-formed .NET classes, and you set properties (parameters) and call methods to make them do things. What you see as a CLI is actually a layer between you and the real PowerShell engine - a way for you to execute those class’ methods without writing an entire program to do so.
When a program like Server Manager, Exchange Management Console, ADAC, and so on “host” PowerShell, they’re instantiating the PowerShell engine - which is a large .NET class. They feed it “commands” and it runs them - but it’s just running .NET class methods. So it really IS using an API; it’s just an API that can also be exposed via a CLI that a human can use.
What makes PowerShell unique is that very construction. Tools like Server Manager aren’t just running external command line utilities like “ping.” You’re right - that would be inefficient, because those commands output text, and text is a poor API. But that isn’t what PowerShell is, and that isn’t what it does. PowerShell isn’t a fancy new Cmd.exe, nor is it a “Windows version of Bash” or some other shell. It’s a unique system that exposes modules of code through a consistent, normalized API, AND exposes them through a human-friendly CLI.
When you run PowerShell.exe, that’s not actually PowerShell. That’s the console host - an application that accepts your commands, parses them, and translates them into that underlying API. It deals with displaying the results in text, rather than making a tree view, icons, or some other display - which is what some other hosting applications, like the Exchange Management Console, do.
Be dubious no more.