DSC, Chef, and Puppet, starting the conversation anew

DSC
1

Finally, I am back here again. After investing a ton of effort at my last company to start them down the road of using DSC, a new job and over a year and a half later, I am back to driving the bus getting configuration management in place for another fortune 500 company.

After my last experience with DSC, and my conversations with many of you at the last PowerShell Summit, I was convinced that the best solution for configuration management is either Chef or Puppet.

I know that Don wrote a book on DSC, and he started the open source project for a DSC Pull server, TUG. I have not been looking at much for configuration management in the last 6 months or more, and a lot can change in that amount of time.

My current boss came from Microsoft and he tends to not to want to look at anything else if Microsoft has a solution. Yesterday, I setup a test configuration using Azure DSC, and added an Azure node and an on-premise server as another node and everything worked fine. But I don’t really see how All Nodes, and other configuration layers will be configured. I also went through the Pluralsight course on Azure DSC.

The two main issues that I see, is there are a significant number of servers across our environment that don’t have PowerShell 5 or higher. This doesn’t ever consider the number of x86 systems we still support for a possible year or more.

I guess where I am at is I would like some confirmation that I am on track with my thinking. Is Chef or Puppet still miles ahead for a true Enterprise solution for configuration management and monitoring?

If so, how can I best quantify this to my boss?

Then, what do I need to consider when comparing Chef and Puppet to each other? I am leaning toward Chef, because I really like Steve Murawski, and I know Chef has had a lot of skin in the game with Microsoft and DSC for a while. But I can’t ignore, the significant number of customer that are already on Puppet, and I continue to hear amazing things about them too.

Let’s hear it…

2

Well, bear in mind that Steve isn’t at Chef anymore :wink:

3

We managed close to a thousands hosts with DSC and an in-house developed config management API (a bit like TUG). We solved, or close to it, most of the pain points of using DSC in an Enterprise environment and then we stopped using run-time configuration management altogether. But that’s another story. Here’s my short answer.

Don’t use DSC unless you want to replace most of it yourself. The LCM has many significant shortcomings in its current version. MS is aware of them and has plans to fix, but it will be a major effort that will likely come after PS v6 is released.

Chef, Puppet, and Ansible are all valid solutions. And there’s Habitat and Otter.

Try the options. You’ll probably find one “feels” easier to work with or solves one of your pain points with less friction. If it reduces the effort you spend on managing systems you’re doing the right thing, but your problems are different than my problems so I can’t say which will work better for you. It might even be that DSC is the right answer for you, but probably not.

4

WHAT? Steve is no longer with Chef…? When did that happen? Where is he now? He is a great guy, I hope his is killing it somewhere. I saw him at MS Build, it must not have been that long ago.

5

Id advice against puppet or chef, no matter how much I like Steven for his awesome work with DSC before he moved to puppet.
From my point of view, dealing with MS technologies last 25 years, I dont see a reason I need to learn Ruby to be able to write my own code when dealing with Puppet and Chef.
On top of the fact both require you to install an agent.
Though both have substantial amount of users and contributors and have had a good amount of releases thus a mature product, I went on to learn Anisble with my Linux studies.
Ansible is python based so slightly easier for a PowerShell user then learning Ruby,
Ansbile doesn’t require any client installed
Ansbile talks directly via WinRM to windows machine thus any Jea solutions youre using will still work.
(that said both puppet and chef can for sometime now use their implementation of WinRM connection)
The “only” downside is you need a Linux machine.
So if youre a 100% MS oriented business that might be a too big of a burden
or it could give your company IT a little nudge to go and learn more and not stay “in the box”.

To that extent I recommend you read some of Adams articles on Ansible and PowerShell
(Simply search for Adam Bertram ansible ) and the amazing
work Trond has done via his contribution to ansible windows modules
http://hindenes.com/trondsworking/ and trondhindenes (Trond Hindenes) · GitHub

do note that all three mentioned config managers support DSC( you do have to turn LCM “off” because of the agents)
and all have extensions now in the VSTS marketplace.

DSC still lacks some orchestration tools to be considered a full configuration management but last EU PSSummit
there was a hint from Angel about a potential of some tooling…and ill stop here :wink:

edit: here’s the section if youre interested
(PowerShell Present and Future - Angel Calvo & Kenneth Hansen - YouTube)- dont hold your breath, it will take time

That said if you have the time and resources, id consider not using any of them,but rather combine what the community
has done so far…
seek Tug, Release Pipeline model, Lability, AutomatedLab, OVF and of course Pester

6

I am shocked this is still a question.
Chef is imperative like ansible. If you are interested in provisioning and short lived hosts that don’t have a lifecycle whereby ongoing system management is important those you can “get by with”. Chef is a disaster when it comes to data handling levels, snowflakes, runlist management and versioning.

My opinion chef is busted because I the source of truth is chef not git.

Puppet and Salt take a declarative approach that I think adds a lot more value over traditional scripting.

Overall I think puppet wins for Windows on maturity, ecosystem support and approach. Hieradata is leagues ahead on data modeling, which helps on snowflakes or globals but requires implementation savvy.

7

He joined Donovan’s DevOps team !!

8

Another thing I will add, don’t implement anything based on Murawski, Jones or myself or anyone else. Each solution can have valid use cases that need to be evaluated against the environment. Agents are both good and bad and only make one vs the other more advantageous in specific scenarios.

I practically invented idempotent config systems management on Windows. I have used in production DSC, chef and puppet on 1000s of servers and Salt in test, so it’s not a bias - I speak authoritatively and the community knows the work I’ve done.

The state of tooling is still meh at best and requires expertise beyond the layman. The “exec” will get you to posh for everything “else” I think the reason the tooling is meh is containers, kubernetes and a whole new wave of cloud tooling coming along that should bury all the conventional stuff. On Linux people have been moving on, but Windows still catching up. Windows as an API driven CLI (powershell) tends to require provider development on any platform. (DSC/powershell, Ruby for chef/puppet, and python for salt/ansible)

Config MGMT will ultimately manage nodes like CoreOS and the future Windows core.
(Previously nano outside containers). Inside the container CM has no place.

Ultimately if you fear the commitment to writing code, none of the tools will bring you 100% coverage except in the most basic environments.

I am especially harsh against chef server. Its just an out of the box rule breaker. Ansible should replace chef in every case, as it’s just a better design if you want imperative MGMT. I have always said, if imperative was the answer, then bash scripts/powershell would have been the end all be all.

9

Throwing my vote in for Puppet with Foreman.

From my (highly biased) perspective a pure DSC environment simply isn’t an environment at all. It’s a wonderful declarative language that can be leveraged by tools like Puppet/Chef but really doesn’t stand on it’s own, nor do the “DSC native” solutions out there really have enough open community support to catch up anytime soon.

I’m not a Chef fan because it tries to be it’s own ecosystem. It works with git, it works with DSC, but it really tries to provide the entire pipeline end to end … and while that sounds cool at first, if you do something outside Chef’s world view … it gets hard fast.

Ansible is just PowerShell for Linux. OK not entirely … but basically. With all the PSCore work being done I actually think PS will eat it’s lunch sooner rather than later. Right now it’s a real hit with the immutable crowd, but it has real problems with systems that are meant to persist which is where declarative rules. Unless you are going all-in immutable (in which case CM’s role shifts to be about provisioning and not maintaining), Ansible vs puppet is like PS scripts vs DSC. Sometimes perfect, sometimes falling short.

The pros on puppet to me:

  1. While it does require an agent, it’s dirt simple to set things up to listen for “puppet” so that a default agent install will “find” a puppet master. DSC as of v5.1 is pretty much built to be managed by an external agent anyway, so things integrate rather smoothly. Just make sure you include chocolatey and 5.1 into your golden image (or packer sequence) and I think you’ll find a very complete solution. Choco install puppet-agent. Done.

  2. Puppet pretty much natively supports DSC with it’s own declarative model … just prefix your declaration with dsc_ and it will hand it off to DSC instead of it’s own declarative engine. That alone means you really never have to touch ruby for windows despite what others have hinted at. I’ve yet to, at least.

  3. Foreman will give you wonderful visual reports and audits of who’s making what changes, and even provide a gui where you can bundle various declarations into groups and assign them to targets (with LDAP support). It’s not perfect … but it’s so far beyond raw DSC I’d call it a must have at this point. Yeah it means learning some Linux (not aware of any foreman containers to let you skip the install), but it’s not impossible and worth it.

  4. Couple the above with R10k which will allow you to to use webhooks to auto-update your repo on git commits and you end up with somethings pretty slick with modest effort.

  5. Need to be even fancier? Hiera will let you encrypt passcodes in a way that devs can safely check them into git.
    BTW Hiera is built into puppet now … though R10K will still take a download.

The hardest part of the IaC model is making sure people don’t shortcut it and actually spend time putting changes into code. Seriously, the amount of bemoaning at first is staggering. But once you get rolling there’s nothing quite like all changes going through PR before committing to the master branch and having things just start kicking in in 30min (or instantly if you configure the “run now” button).

I run at a startup so we only run a modest 150 servers, but we have them grouped, standardized, and a process in place that ensures every new server is assigned a computer group and thus properly managed by code.

Now our containers on the other hand … still wrestling with those …

10

you made a comment about servers not having PS5: is this unavoidable? Basically that breaks Puppet, but honestly it breaks DSC as well. If IOC is the goal, that’s a fundamental problem that needs to be addressed IMO. If these are older 2008/2003 boxes … well the next step should be obvious no matter how painful said move will likely be.

11

There’s a bit of mis-information here.

I have left Chef the company, but not Chef the community nor Chef the open source project. As noted, I’m at Microsoft on Donovan Brown’s team of Developer Advocates. I focus on the Chef community and WinOps community (and other DevOps scenarios).

Chef is a declarative (not imperative) config management tool. The source of truth for any of these tools depends on how you build your pipeline and I wouldn’t recommend anything that isn’t source control backed. Not sure where Rich is getting the idea that Chef is imperative. You can pretty much make any config management tool operate in an imperative mode, but it’s way more work.

In any case, config management is really a commodity. You can get the job done with Puppet, Chef, and DSC. All require some learning and investment in the tooling and workflow. In my experience with configuration management (I’ve used all three of those), the real value comes as part of your workflow. Building a solid foundation of testing, backing your CM with source control, using a CI server to continually validate the contributions to your CM policy, reporting the current state of your environment - what’s changed, what’s compliant, what’s not, etc… If you’ve got that and you find your current configuration management platform isn’t up to what you need, it’s easier to make a migration to a new tool if you have good tests for your environment. You can do it in an iterative manner at that point.

Learning Ruby vs Python - ¯_(ツ)_/¯ - one is not “easier” than the other objectively. They are just new languages to learn. The majority of learning is around how the projects use those languages and how quickly you get to a point where you need to extend things.

Back to the original question, which CM product/project should you use? You need to try them and see what the best is for your scenarios and needs.

12

We are almost exclusively a Windows shop, except we have Oracle. Not sure if OS really matters that much considering long term, configuration management will traverse the entire environment. At least that is my understanding at this point. As far as learning Python or Ruby, just like Don said 5 or so years ago, we all end up having to learn at least a couple of languages, they are all just variations on the same basic things. As far as having to install a client, or using Winrm, I just want something that works. Winrm, might be an issue on older OS’s. I just did SCCM and SCOM roll outs, and getting clients on thousands of machines in a diverse environment with a lot of technical Dept, is just a big pain, but once it is done, it is a non-issue.

We seem to be headed head long toward Azure. The use of SaaS and IaaS seems to be the big attraction. Continuous integration is consistently discussed goal. I am migrating all of our code into VSTS (Visual Studio Team Services). I see that MSN uses Chef.

I am not sure why Richard thinks that Chef is imperative either, I had never heard that before, and I see Steven disputes that.

The relationship that Puppet has with Chocolaty seems to be a plus for me, I have been a big fan of Chocolaty for a long time, and I am excited to see some of the new stuff they are doing.

Justin, thanks for the lengthy answer, it is helpful.
What is Forman and R10k and Hiera?

You also mentioned PowerShell 5 breaking puppet. So, there is no solution for managing the 2003 and 2008 boxes while we work to get off them?

Steven, I am excited to hear more about your new positions, congrats!!!

You said this:
Building a solid foundation of testing, backing your CM with source control, using a CI server to continually validate the contributions to your CM policy, reporting the current state of your environment.

Where can I get the best information for designing and pulling these pieces together?

13

So to be clear, It’s not PS5 breaking Puppet, it’s that there is a base DSC requirement. The oldest version of DSC will require PowerShell 4 with a few hot-fixes and that requires a minimum of 2008 R2. Puppet itself ups the requirement to WMF 5.0, but that doesn’t change the base OS requirements (still runs on 2008R2). If you want Infrastructure as code, you got to get off that 15 year old software. To my knowledge there’s simply no way around that … at least none I’d ever dare try. Bad news there, I’m afraid.

So Puppet has it’s own ecosystem, like everything else, so some of the parts that I mentioned:

R10K
This is a toolset that is focused on getting git onto puppet masters. Think of those as “pull servers” in DSC speak. Basically you can set it up to watch certain branches/tags/whatever of a repo (usually using webhooks) and it will automatically sync your server with said files. So if I make a change in my local copy of VSCode and sync the request to github, r10k will make sure the puppetmaster (the pull server) is always in sync. It also has some other functions like syncing modules and the like, but really it’s all about git.

Foreman
Foreman is a nice web front end and “ENC” for the free version of puppet. Without going into too much detail it gives you a decent interface with management and reporting options for free. Cards on the table it was a bit of effort to get it working the first time… but when it’s actually a very solid tool. Size of your company you may decide to just plunge into Puppet Enterprise or something like Chef where a consultant can get things going more quickly, but if you want to run the free route it’s a great companion.

Hiera
Probably the most complex topic to people who never touched puppet but also extremely useful to the point it’s officially built into Puppet now. It basically creates a hierarchical model (hiera, get it?) for parameters/variables to make it easier to write your manifests. One of it’s cooler features is it can also encrypt passwords at rest so you can call them by a public key in your code and not violate any security standards (don’t keep passwords in plain text in your github!!!)

14

Thanks again Justin for your responses. So are you using the free version? Is there a link that you can recommend to help me better understand the differences between the two options, and how I can try the free version? Is there any other recommended reading you can offer?

15

Steven, is there someone you might recommend for me to reach out to at Chef that can help me work through a proof of concept?

16

I had a call yesterday with Puppet and today I had a call with Chef. I have to say that the fact that Microsoft is using Chef for configuration management for MSN, Office365, Azure, Skype and their internal development team is a pretty powerful argument for why to use Chef. Not to mention they have put a production Chef Automate template in Azure that took me all of 2 min to stand up. Emphasis on production ready. Puppet has a template too, but I cannot choose to deploy it on SSD, and it won’t let me configure an internal NIC… Weird. Chef says that these Microsoft teams contribute to their recipes, and have agreed to maintain some of them. Considering our leaning toward Azure and Microsoft in general, I think I am starting to lean that way too.

Chef says they are more focused on continuous deployment, in a practical sense, how does that bare out? Can someone give more specifics as it relates to the way Chef does things?

Anyone using InSpec and have any practical information on the use of that?

We have a lot of legacy stuff still, does anyone have some experience with Habitat and how that has been useful?

17

Back again, and just a little confused by some newly released information. If you go to Chefs website, they have a link to the new report just published buy Forrester, which rates Puppet and Chef as the two top configuration management systems. But this report, and other explanations of configuration management are making me uncertain of where the use of these tools begins and where it ends.

First, in the reports section labeled “Leaders”, under Chef Open source it says:

› Chef. The Chef open source engine primarily uses an imperative approach with broad support
for various operating systems, containers, and cloud services. A code repository, called the Chef
Supermarket, hosts thousands of Cookbooks (Chef code) that allow users to download common
configurations. Chef partners support some Cookbooks, and users can “follow” Cookbooks in the
Chef Supermarket to receive automatic notification of any updates. The strengths of open source
Chef include the breadth of system support as well as community engagement. Users found Chef
powerful but said it was difficult to master.

Why is another source saying Chef server is imperative? Is this because you have to put your configurations into the Chef server where it holds and versions them. Or, am I missing something completely, like, if you don’t use Chef Automate, you don’t get to use clients and have your nodes check into Chef server? I am confused buy this being another resource saying Chef is imperative I don’t want this to be something unexpected once I get to far down one path.

The next issue I have is the whole inside out side configuration issue. If I am standing up a new server in Azure, I have 10 things I do every time I do this. Some of these things like adding a drive, changing the IP for dynamic to static (On the Azure side) and adding Azure back up (to my preconfigured 30 day retention configuration). My understanding is all of these things that are not controlled or configured buy Chef server because they are “Outside” things. But the “Inside” things like setting up the partitions on the disks, assigning drive letters, opening ports in the Windows Firewall, and turning on more firewall logging are all things I do with Chef, once the server is bootstrapped, installing the Chef client, getting the right certs, and it checks in with Chef server, becoming a node.

I keep hearing these things like Chef does Infrastructure as Code, but I don’t see that. I am assuming I will need an Orchestrator or Azure Automation if I want to configure and monitor those outside pieces.

Does Puppet do this different, or do they have a way to do the outside configuration stuff and keep it configured?

Lastly, is the reason that people say that Chef doesn’t use Git as it ultimate source of truth because you have to check your cookbooks in and version them on the Chef servers, therefore you may keep code in a Git repository, it is not the ultimate source of truth, because it is really the Chef server and it’s versioning itself that is?

18

So you are stumbling into what I consider a larger topic on the scope of what CM handles. I can assure you, that there are going to be as many opinions on this as people, but I’ll do my best to outline things from my perspective, which are my own my own yadda yadda yadda…

While I’ve not heard the term “inside vs outside configuration”, I’m going to assume it’s explicitly referring to the scope of CM, and what you intend to (or even can) manage with it.

Here’s the TLDR version: The scope of what I’ll call “traditional CM” (ala Chef/Puppet) is going to be directly related to the maturity and scale of your DevOps model, or more specifically: where you are in your “cattle vs pets” journey (which is not exactly the same but really tightly coupled).

Ready to get fun?

Right now you have pets. Lots and lots of pet servers that need configurations, patching, tweaking, etc. I’ll assume almost exclusively as you’re looking at CI with great interest.

As you start to declare configurations and standardize servers, you are going to start down the path of shifting from pets to cattle: things that are never taken care of, but rather replaced. It will start simple with web servers, app servers, but legacy systems and most databases servers will sit firmly in the realm of pets … even if you do manage large aspects of them with CM.

At this phase in the journey your CM usage will be massive … touching nearly everything. Yes: Puppet and Chef can deploy the entire server soup to nuts, but it will take hooks to other tools (remember I mentioned Foreman? That has an AWS/Azure provision-er so it can create servers for you then kick off the Chef/Puppet run). THings will be good and you’re going to do very well with it.

To be honest … within the MS ecosystem this is about as far as most get … but our Linux brethren have already evolved passed this. Lets talk scenario:

see what’s going to start happening is things like Azure ARM and AWS Cloud Formation start a shift in thinking: remember images? You may have heard Snover say the “golden image is dead” … but “long live the image”. See, spinning up a server then pushing a recipee/puppet manifest/dsc mof still means your servers will take minutes to be ready. 15+. This sounds decent now … but eventually you’re going to start falling in love the idea of pre-baked images again. Machines that boot in 10 seconds ready to go. This will be especially good for all that “auto scale out” you hear about. As you continue down this “go faster” path you’ll be gin to realise: you dont need CM after the box is deployed … they’re cattle after all. You just need the images perfect then you provide a huge list for people to use.

Suddenly chef and puppet retract. All they are doing is helping you build images and commit them to AWS/VMware where ever, oh and still handling that really complex stuff that can’t really live as an image very well (yet, you’ll get better and the toolsd will get better … and it will happen).

Then you hit your next realization: if all im doing is building images … why do I need DSC or any kind of declarative language at all? That thing that sounds awesome now becomes … overhead. I dont want to train people in a new language … just let them make a stupid PowerShell/bash script and build the image. It only has to build once … then the image is stored. Suddenly that invaluable CM tool that started you on this journey is a hindrance… except for those big massive oracle/mysql databases … still insanely valuable there.

As you continue to mature that container thing will happen. See at some point (if it hasn’t started already) your devs will want to build their app and test it locally then ship that finished thing. It’s way easier that way … but CM is going to fall flat and hard to handle this. You’re going to start looking into Azure Service Fabric, or Kubernetes clusters, or HashiCorp tools to start helping you out. And as you start to understand the role of schedulers and service discovery you’re going to realize something huge: that stuff helps with the legacy apps, too.

By the end of your journey you will likely still have some Puppet/Chef but in a way you’ll be right back where you started: very little of it covering very little (basically it manages only the lowest layer of your now hugely abstracted cloud). So it’s use is like a bell curve.

So why is this relevant to your question of “inside outside configuration”? Because in my never humble opinion: the difference answers you get are due to the where said person is in their own DevOps delivery model journey.

It sounds like you like the Chef product: move on it. Just because I’m predicting a fade in/fade out doesn’t mean it’s not a path worth taking. It starts some wonderful things in motion.

19

Very interesting conversation, but I have one question: Nobody is mentioning any price, and yet when I look at puppet, chef or ansible, I see some pretty steep price for enterprise use. Compared to DSC, which comes built in the OS so no extra cost, that’s a pretty big monetary overhead. Am I missing something?