Quelques bouts de code pour automatiser le déploiement

Automate ALL THE THINGS!

Ce billet n’est qu’un simple «link dump» pour retrouver parmi plusieurs notes éparpillés. Je compte éventuellement publier la totalité de mon travail dans des projets publics sur GitHub une fois la boucle complétée. Le tout sans fournir les données privés, évidemment.

Faire le saut vers l’automatisation demande beaucoup de préparation et je prends le temps de publier ici quelques bouts de code que j’ai écrits pour compléter la tâche.

Au final, mon projet permettra de déployer un site qui s’appuie sur un cluster MariaDB, Memcached, une stack LAMP («prefork») lorsqu’on a pas le choix, une stack [HHVM/php5-fpm, Python, nodejs] app servers pour le reste servi par un frontend NGINX. Mes scripts vont déployer une série d’applications web avec toutes les dépendances qui les adaptent géré dans leur propre «git repo» parent. Dans mon cas, ce sera: WordPress, MediaWiki, Discourse, et quelques autres.

Requis

  • Instantiation à partir de commandes nova du terminal, crée une nouvelle VM mise à jour et son nom définit son rôle dans le réseau interne
  • Les VMs sont uniquement accessible par un Jump box (i.e. réseau interne seulement)
  • Un système regarde si un répertoire clone git à eu des changements sur la branche «master», lance un événement si c’est le cas
  • Chaque machine sont construites à partir d’une VM minimale. Dans ce cas-ci; Ubuntu 14.04 LTS
  • Système doit s’assurer que TOUTES les mises à jour sont appliqués régulièrement
  • Système doit s’assurer que ses services interne sont fonctionnels
  • Dans le cas d’une situation où une VM atteint le seuil critique OOM, la VM redémarre automatiquement
  • Le nom de la VM décrit son rôle, et les scripts d’installation installent les requis qui y sont affectés
  • Les configurations utilisent les détails (e.g. adresses IP privés et publiques) de chaque pool (e.g. redis, memcache, mariadb) et ajuste automatiquement les configurations dans chaque application
  • … etc.

Bouts de code

Billets inspirants sur le sujet

Thoughts about learning in the web developer job, what managers might be missing

After reading an article titled What the ‘Learn to Code’ movement is forgetting: Existing developers, I couldn’t resist commenting. Sadly, I couldn’t add my own comment because their commenting is allowing only paying members so I am sharing it here.

Developers do not simply convert paragraphs of text requirements into “morse”. Obviously. Its a complex craft and like anything else, its all about people you are working with. There are passionate programmers and also the ones who only wants to do the bare minimum. In between, its all about leaving room for creativity.

What a programmer’s day looks like

Let’s remind ourselves what a programer is required to do has a lot of complexity and has to deal with legacy, unknowns, and delivery dates. Not everything can be foreseen: how the team members will work together, their capabilities, ego, and most importantly, whether or not each of them will be in “the zone” long enough during work hours.

The time they are at their optimal is what you want to have most from them, its that time where they can do amazing things, isn’t it. Being developer is about solving puzzles, review code or existing components they can leverage or would need to refactor. Being interrupted by numerous meetings to talk about something they are already trying to figure out doing isn’t helping them.

A good way to help them is to set in place asynchronous communication channels such as IRC, and code review practices. In some open source communities, merge to the master branch requires code review, a common practice is to have a bot to announce commits on an IRC channel. That’s the best of both worlds, you let developers be in their zone AND collaborate. As for paired-programming, they can just announce at the morning SCRUM meeting that they would like to spend time, or just ask for it on IRC.

Learning

As for learning, passionate developers already keep themselves in the loop of best practices and how to improve their craft. The experienced ones might even have ideas on how do things that you never thought of.

I’ve seen companies who allows participation to open source projects, and/or share their knowledge openly on sites such as Stack Overflow and documentation sites. This is another great way to make developers more engaged to their role, and stay in your company.

When I think of learning resources for web developers in general, there is an independent, and vendor neutral, resource where web developers can learn. The site is convened by the W3C, and sponsored by industry leaders to create just what we’ve been missing all along. The site is webplatform.org, and I feel like it’s not enough set forward by the industry. Maybe because its not complete, but its a wiki, anybody can participate.

Full disclosure; I work full time on the webplatform.org project, but note that regardless of my current position, I would participate to the site regardless of whether or not i’m hired to do so.

Photo credits: Anthony Catalano, Morse Code Straight Key J-38

Project idea: Creating a home made OpenStack cluster for development purposes

Think about it. How about using spare computers to create a homemade OpenStack cluster for development.

We can do that from our cloud provider, or create a separate project or even use Wikimedia’s OpenStack infrastructure allowance for the project.

With such setup, one could work locally with his Salt stack (or Puppet, or Ansible) deployment schemes, try them, trash VMs, rebuild.

The beauty of it would be that it could be made in a fashion that would not even modify the computer running the VMs. The cluster member running OpenStack hypervisor would be installed seeded through net boot. Not booting from the network would revert the computer back as if it never been used.

Here is what I think would require to make this happen.

Limitations

  • Not use Computer/Laptop local hard drive
  • Rely only on net boot

Material

  • 1..n Computers/laptop supporting netboot
  • 1 Storage device supporting one or more storage protocol (nfs, samba, sshfs)

Hardware requirements

  • 1 VM providing tftp, dhcp, dns to serve as net boot server that should run outide of the cluster (“Networking node”)
  • 1 VM image of OpenStack controller (“OpS controller”)
  • 1 LiveCD+persistent image with OpenStack preinstalled, configured to use storage device credentials as it’s root filesystem (“OpS Hypervisor”)

Distribution choice factors

  • Networking node can be the smallest Linux possible, on a RaspberryPI, or a modified Router or Network Attached storage device?
  • OpS Hypervisor to be among the supported OpenStack distributions (I think a Ubuntu precise 12.04 LTS or a variant such as Puppy Linux might work too)

To be continued…

I will keep you posted whenever possible on the outcome of my research.

Did you ever do this in your infra. Leave a comment.

Who else is using feature flipping thing on their web applications?

I am currently reading and collecting ideas on how to present and propose an implementation in my projects.

I want to use:

  • Continuous integration
  • Automated builds
  • Feature flipping

And make all of this quick and easy for anybody in the team.

Feature flipping

This is fairly new to me, but I like the idea. The concept is that the code declares in their own components which features they are contributing to.

This way, we can then totally hide it from sight from the users.

Source control branching

I am currently searching and preparing to introduce to my client ways to work with a few elements in our project.

The idea is that instead of managing a complex branch scheme, and skim to the essential.

A trend was to use GitFlow, then, the project grows, developers do not have all the time to manage everything, and things get out of hands.

It may then look like something similar to that:

Quoting a slide from Zach Holman about branching

It doesn’t seem bad in the first place, but even though Git gives an easy way to do so, if you want to adapt quickly, it can bring overhead.

At least, that’s what Flikr, Github, Twitter, Facebook (so I heard) does.

I’ll keep you posted on what I find on the idea soon-ish.

References