Getting from shell to Puppet
After this (french) article, dealing with managing servers with shell scripts (what we were doing), and this (also french) one, which dealt with tools for automated deployment (what we planed to do), including Puppet, here is the article about Puppet. By doing. With blood, tears, and victories ;)
Because yes, going from servers managed by shell scripts to Puppet, when you don't know Puppet, it's not so easy.
Where do we come from
On the project we have five environments of three computers each: one web server (Apache), one app server (Tomcat), one database server (PostgreSQL). Plus one ftp server, plus two monitoring servers (Zabbix)(which host a web server and a database server) and deployment server (OCS). Plus a demo server. Plus two servers for an other project (which host the same components).
To summarize, we have 17 servers almost identical, plus 4 atypical ones.
We use shell scripts (bash) to install those servers :
- Core configuration (packages, users, DNS, environment variables, monitoring daemons, ...)
- Installation of base softwares, without their configurations: they are shipped with the applications.
And we use one other shell script to check whether everything goes well, and a little bit of glue (Makefile, home made Perl modules, lib directory for shared shell functions, meta-configuration files, ...)
To be humble, we kind of master shell. It's a pleasure to solve problems by some well crafted shell lines. For powerful they are, those install scripts hit limits. They :
- Are not idempotent: run twice, the same install script can break things.
- Do not handle software upgrades : getting from Postgresql 8.4.3 to Postresql 8.4.7 is not like installing 8.4.7 from scratch.
- Are too much project-dependent : they are built for one project. Installing a server for another project requires to modify the scripts.
- Are not dev-compliant : we want to be devops, we want the developers to install servers/VM. It's possible with scripts but all the devs do not know shell, right ? But they play with configuration files all the time, so what if installing a server/VM was just a matter of tweaking configuration files ? Dev could then install a full server/VM ;)
We have two choices:
- Continue with scripts, with the risk they might become fat and impossible to maintain.
- Go on with serious soft : Puppet/Chef/cfEngine
We choose Puppet for those reasons:
- A PoC was made, and it shows that it can fit (and provide us a bootstrap to start with)
- We ear about Puppet all around us
- We don't know Puppet :)
So we will puppetize our scripts.
Migration
Puppetizing our scripts means that we will destroy almost everything we have been working on for more than a year, and start building again.
It's an effort. Agile teaches us not to be attached to the code (if you are, refactoring will be a pain for you), but it pinches the heart.
Mourning done, it's time to learn Puppet. At this very moment, troubles begin...
The idea behind Puppet is 'convention over configuration'. It means that you specify less but you have to learn more things (the "conventions"). And there is a lot of things to learn...
Puppet works with contracts: you specify them, in its Ruby-like language, what you want, and it handles.
I insist: you specify, it makes. It means that you do not do. Which means that if you were used to code (to act to obtain something), it's not the case anymore : you specify a target, it reaches it. You don't really know how, by the way...
This way of doing leads to a different way of thinking, and it's disturbing. It's not easy to get into it when you didn't see / learn it before, which was indeed my case: I didn't find this paradigm in any other language I learned (by myself).
We started with simple things, things that Puppet is good at: creating users, playing with system files, installing packages. The web is full of documentation about how to do such things.
About the documentation... I rarely saw such a badly organized documentation. I always have to watch at least three web pages before finding what I'm looking for: is it a metaparameter? a function? a type? a fact? something else?
With time, the you'll get into the syntax of Puppet. But more because you remember it than because it's logical: a bit like memorizing a text without any sense (and for everything else there is this very well done document)
Once simple things are puppetized, we tried less simples tasks: downloading a package from Nexus, running it if it's a binary, installing it if it's an rpm, creating a database if it's doesn't exists, creating corresponding roles, updating pg_hba.conf and .pgpass. It's seems obvious that Puppet doesn't follow the Perl motto (turn complicated things into simple ones, keep simple what is it) and making Puppet do what it's not good at is a pain: we had to fight with it to make it do what we want. And to my opinion it's a failure when I have to write shell into a program to make it do what I want...
Add to this the obscures (obfuscated ?) error messages that almost never show you the root cause of the problem, and you easily understand that I was angry with Puppet.
To summarize, here is what I blame Puppet for :
- It is slow. I made my tests in a VM, and it's damn slow.
- Syntax is obscure : why keywords are sometimes in camelCase, sometimes not? Why sometimes brackets and sometimes curly brackets? Why 'owner' and not 'uid'? Why should I always have to specify the 'title'? Why do I have to sometimes put colons, sometimes not? Why sometimes ending a line with semi-colon, sometimes with a coma, sometimes with nothing? Why does Puppet sometimes accept coma at the end of the lines, sometimes not? Why 'title' and 'name' ? Why do facts and user defined variables have the same syntax? Why 'source' LOOKS LIKE a path but WITHOUT 'manifests'? Why doing a syntax that LOOKS LIKE Ruby but is not? Do I have to use classes or nodes?
- The documentation, badly organized.
- Error message are far from being clear : instead of saying that the variable is undef can't you suggest me it could come from an uninitialized plugin?
I never learned Ruby or programming by contract, the fee to Puppetland is expensive.
Some hints
A pyramid of classes
Migrating from our shell scripts to Puppet has been done gradually: we have built one module per functional component to replace: Tomcat, Postgresql, Apache, ...
We made much more functional component than we expected at first: pki, sudo, user (to create users but also to install scripts in their ~/bin), <component>db (for the database of those components), ntp, logrotate...
The documentation of Puppet tells us that the 'node' directive has to be used, but we used 'class' to pass parameters (such as password) from classes to classes. Thus, we are able to change the password of all classes in one move.
So, we have grouped those components in classes, and stacked these classes.
To Puppetize a server, we just had to assign the classes to this server, et voilà.
class basenode ($user_hash=undef, monitored=true) {
class { 'basesystem': }
class { 'puppet': }
class { 'nexus': }
class { 'hosts': }
class { 'user': hash => $user_hash }
class { 'env': }
class { 'ntp': }
class { 'ssh_keys': }
class { 'sudo': }
class { 'baseapp': }
if $monitored { class { 'monitorednode': } }
}
class dbnode ($min_sec=undef, $i_pass=undef, $mo_pass=undef, $g_pass=undef, $pgversion='x.y.z-1', $backup_retention_days=undef, $monitored=true ) {
class { 'basenode': monitored => $monitored}
class { 'pgsql': log_min_duration_statement => $min_sec, pgversion => $pgversion, backup_retention_days => $backup_retention_days }
if $monitored { class { 'pgsql::zabbix': } }
class { 'pgsql::tools': }
class { 'db': i_pass => $i_pass, mo_pass => $mo_pass, g_pass => $gestion_pass}
if $monitored { class { 'db::zabbix': } }
class { 'pshops': }
}
node 'db1' {
class { 'dbnode': min_sec => '1000', i_pass=>'hop', mo_pass=>'zou', g_pass=>'bla'}
}
In this example, db1 is a dbnode, which itself contains basenode. And some parameters are passed to specify passwords.
Whilst we built our classes and put our servers into them, we progressively deleted our installation scripts.
Wrapping
We dared wrapping some Puppet's directives to suit our needs :
define user::userfile (
$basedir = "/home/${user::username}",
$source = '',
$content = '',
$mode = '0644',
$owner = $user::username,
$group = $user::username,
$replace = ''
)
{
file { "${basedir}/${title}":
owner => $owner,
group => $group,
mode => $mode,
}
if $source { File["${basedir}/${title}"] { source => $source } }
if $content { File["${basedir}/${title}"] { content => $content } }
if $replace { File["${basedir}/${title}"] { replace => $replace } }
}
This function puts a script in the ~/bin of the user, with correct rights. It's avoid us to repeat ourselves (DRY :) )
Nexus
Some of our binaries installers are in Nexus. And guess what? Puppet didn't know how to handle such a thing. We had to teach it :
nexus/init.pp :
class nexus {
$repo = 'thirdparty'
$target_dir = ''
$user = 'user'
$pass = 'pass'
$server = 'some.server.tld'
$port = '7853'
$url = "http://${user}:${nexus::pass}@${server}:${port}/nexus/service/local/artifact/maven/content"
}
nexus/get.pp :
define nexus::get (
$groupeid,
$artifactid,
$version,
$type,
$classifier = '',
$mode = '0644'
)
{
include nexus
$full_url = "${nexus::url}?r=${nexus::repo}&g=${groupeid}&a=${artifactid}&e=${type}&v=${version}&c=$classifier"
exec { "download_via_nexus ${title}":
command => "mkdir -p \$(dirname \"$title\") ; /usr/bin/wget -q \"$full_url\" -O \"${$title}\" ; /bin/chmod ${mode} \"${$title}\"",
path => [ '/usr/bin','/bin'],
timeout => 0, unless => "test -e \"${title}\" && test `/usr/bin/wget -q \"${full_md5_url}\" -O -` = `md5sum \"${title}\" | /bin/cut -d' ' -f1`",
}
}
nexus/exec.pp :
define nexus::exec (
$groupeid,
$artifactid,
$version,
$classifier = '',
$type = '',
$command = 'echo "I should do something with \"<%= title %>\""',
$purge_after = false,
$mode = '0644',
$creates = '', $unless = '', $onlyif = '',
$path = ['/bin', '/usr/bin']
)
{
nexus::get {$title:
groupeid => $groupeid,
artifactid => $artifactid,
version => $version,
type => $type,
mode => $mode,
classifier => $classifier
}
$real_command = inline_template($command)
exec { "exec_from_nexus ${title}":
command => $real_command,
path => $path,
logoutput => true,
timeout => 0,
require => Nexus::Get["$title"]
}
if $unless { Exec["exec_from_nexus ${title}"] { unless => $unless } }
if $onlyif { Exec["exec_from_nexus ${title}"] { onlyif => $onlyif } }
if $creates { Exec["exec_from_nexus ${title}"] { creates => $creates } }
}
nexus/extract.pp :
define nexus::extract (
$repo = thirdparty,
$groupeid,
$artifactid,
$version,
$type = '',
$classifier = '',
$target_dir = '',
$owner = $user::username,
$group = $user::username,
$extract_cmd = "tar --owner ${owner} --group ${group} -C '${target_dir}' -xz",
$creates,
$nexus_url = 'http://user:pass@some.serveur.tld:7853/nexus/service/local/artifact/maven/content'
)
{
$full_url = "$nexus_url?r=$repo&g=$groupeid&a=$artifactid&e=$type&v=$version&c=$classifier"
exec { "extract_from_nexus ${title}":
command => "/usr/bin/wget -q '$full_url' -O- | ${extract_cmd}",
path => ['/bin', '/usr/bin'],
unless => "test -d '${creates}'",
timeout => 0,
require => File[$target_dir]
}
exec { "ensure correct u:g to ${creates}":
path => ['/bin', '/usr/bin'],
command => "chown -R ${owner}:${group} '${creates}'",
unless => "test `stat -c '%U:%G' '${creates}'` = '${owner}:${group}'",
require => Exec["extract_from_nexus ${title}"],
}
}
Look at the shells commands we had to use here and there to make Puppet act the way we want: do not download things twice, put things in the right directory, put the correct rights, ...
Shell
The most tricky part was to make Puppet install Postgresql. We install Postgresql from binaries (that were downloaded from Internet and put in Nexus), and we wanted Puppet to handle not only installation but also update from minor to minor (x.y.z -> x.y.z1) and from minor to major (x.y.z -> x1.y1.z1), thus with dump and restore. As Puppet is not shipped with this functionality, we had to take a weapon of mass destruction : shell.
nexus::exec { "/var/cache/puppet/pgsql-${pgsql::params::version}":
groupeid => 'org', artifactid => 'postgresql',
version => "${pgsql::params::version}",
classifier => "linux-x64",
type => 'bin',
mode => '0755',
purge_after => false,
path => [ '/bin','/sbin','/usr/bin','/usr/sbin' ],
command => $minipgsqlversion ? { # Depending on the installed version ...
# Same major as it's asked to me (x.y.z vs x.y.z1) : update without dump
$pgsql::params::miniPgVersion => "bash -c 'if test $pgsqlversion != ${pgsql::params::dirPgVersion} ; then echo -e \"\\n\\n\\n\\n\\n\\n\\n\\nn\\n\" | <%= title %> --prefix ${pgsql::params::prefix} --datadir ${pgsql::params::datadir} --locale ${pgsql::params::loc} --mode text --servicename ${pgsql::params::service} --superpassword ${pgsql::params::pass} ; else /bin/true ; fi'",
# Nothing found : installation
'notfound' => "echo -e \"\\n\\n\\n\\n\\n\\n\\n\\nn\\n\" | <%= title %> --prefix ${pgsql::params::prefix} --datadir ${pgsql::params::datadir} --locale ${pgsql::params::loc} --mode text --servicename ${pgsql::params::service} --superpassword ${pgsql::params::pass}",
# Default case (x.y.z vs x1.y1.z1) update with dump
default => "bash -c 'name=/opt/pgsql-$pgsqlversion-to-${pgsql::params::dirPgVersion}-\$(date '+%Y%m%d-%H%M%S').sql.gz ; /usr/local/postgresql-$minipgsqlversion/bin/pg_dumpall -U postgres | gzip > \$name ; service postgresql-$minipgsqlversion stop ; echo -e \"\\n\\n\\n\\n\\n\\n\\n\\nn\\n\" | <%= title %> --prefix ${pgsql::params::prefix} --datadir ${pgsql::params::datadir} --locale ${pgsql::params::loc} --mode text --servicename ${pgsql::params::service} --superpassword ${pgsql::params::pass} ; service ${pgsql::params::service} start || /bin/true ; gunzip -c \$name | ${pgsql::params::prefix}/bin/psql -U postgres '",
}
}
Again, look at the deployed arsenal.
Here is the facter that provides the Postgresql version :
Facter.add("pgsqlversion") do setcode do psql=Dir.glob("/usr/local/*/bin/psql").sort[-1] if psql %x(#{psql} --version).split("\n")[0].split()[-1] else 'notfound' end end end Facter.add("minipgsqlversion") do setcode do psql=Dir.glob("/usr/local/*/bin/psql").sort[-1] if psql %x(#{psql} --version).split("\n")[0].split()[-1].gsub(/\.\d*$/,'') else 'notfound' end end end
Once built the Puppet's configuration, we could sleep more peacefully at night.
Benefits
We didn't use Puppet from the beginning of the project because we wanted to be fast, and we didn't know Puppet. Moreover, the team was young and Puppet was not the key tools to beginning with (to be honest: it was shell)
Thus, beginning by shell scripts was a good idea, but it hits limits I talked about hereabove.
The puppetization of platforms was not painful only for me, but to the others teams too, in the good way : the psychorigidy of Puppet has shown us that some things weren't clear on our servers (the uid of our main unix user was not the same everywhere, for example). It forced us to make things clearer and more explicit.
Where we hesitated to run our installation scripts, we now ask Puppet to do it: happiness of programmation by contract is there: when something is done, it's not to be done anymore, and you don't care how it was done (except in some rare cases).
When one thing is puppetized (the installation of Tomcat, for example) we have no more doubts: it's installed in the same way everywhere, and it works.
Installing a new server takes 10 minutes: 30 seconds to install the core stuff (which basically renames the server and installs Puppet) and 9min and half to drink tea waiting for it's over. Followed by the sweet feeling that things went well.
Truly, we simplify our install and update process: everything is in one place (on the Puppet master), and one command line to type.
Conclusion
Installing servers with Puppet is a real plus compared to shell (yeah captain Obvious!). But Puppet is quite hard to learn. Our choice at the beginning of the project was to not start with it, for these reasons :
- It was important for the team to learn shell first, even if the lifetime of the scripts had to be short. According to us, adding Yet An Other Tool was not a good idea.
- The environment was small (15 servers at that time) and installed by our hoster. So installing the OSs by shell scripts was not unreasonable.
Nowadays we think we have made the good choice : learning Puppet was not the first thing to do. But the second, just after learning shell :), since we achieved all of goals with Puppet.
Some useful links :
- Puppet Cheat Code Sheet, what you always forget about the syntax of Puppet's most commons resources.
- Puppet modules. Lots of Puppet modules. In the rare cases they don't fit you, they will help you a lot.