Converting a dynamic site into static HTML documents

Its been two times now that I’ve been asked to make a website that was running on a CMS and make it static.

This is an useful practice if you want to keep the site content for posterity without having to maintain the underlying CMS. It makes it easier to migrate sites since the sites that you know you won’t add content to anymore becomes simply a bunch of HTML files in a folder.

My end goal was to make an EXACT copy of what the site is like when generated by the CMS, BUT now stored as simple HTML files. When I say EXACT, I mean it, even as to keep documents at their original location from the new static files. It means that each HTML document had to keep their same value BUT that a file will exist and the web server will find it. For example, if a link points to /foo, the link in the page remain as-is, even though its now a static file at /foo.html, but the web server will serve /foo.html anyway.

Here are a few steps I made to achieve just that. Notice that your mileage may vary, I’ve done those steps and they worked for me. I’ve done it once for a WordPress blog and another on the [email protected] website that was running on ExpressionEngine.

Steps

1. Browse and get all pages you think could be lost in scraping

We want a simple file with one web page per line with its full address.
This will help the crawler to not forget pages.

  • Use a web browser developer tool Network inspector, keep it open with “preserve log”.
  • Once you browsed the site a bit, from the network inspector tool, list all documents and then export using the “Save as HAR” feature.
  • Extract urls from har file using underscore-cli

    npm install underscore-cli
    cat site.har | underscore select ‘.entries .request .url’ > workfile.txt

  • Remove first and last lines (its a JSON array and we want one document per line)

  • Remove the trailing remove hostname from each line (i.e. start by /path), in vim you can do %s/http:\/\/www\.example.org//g
  • Remove " and ", from each lines, in vim you can do %s/",$//g
  • At the last line, make sure the " is removed too because the last regex missed it
  • Remove duplicate lines, in vim you can do :sort u
  • Save this file as list.txt for the next step.

2. Let’s scrape it all

We’ll do two scrapes. First one is to get all assets it can get, then we’ll go again with different options.

The following are the commands I ran on the last successful attempt to replicate the site I was working on.
This is not a statement that this method is the most efficient technique.
Please feel free to improve the document as you see fit.

First a quick TL;DR of wget options

  • -m is the same as --mirror
  • -k is the same as --convert-links
  • -K is the same as --backup-converted which creates .orig files
  • -p is the same as --page-requisites makes a page to get ALL requirements
  • -nc ensures we dont download the same file twice and end up with duplicates (e.g. file.html AND file.1.html)
  • --cut-dirs would prevent creating directories and mix things around, do not use.

Notice that we’re sending headers as if we were a web browser. Its up to you.

wget -i list.txt -nc --random-wait --mirror -e robots=off --no-cache -k -E --page-requisites \
     --user-agent='User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36' \
     --header='Accept-Language: fr-FR,fr;q=0.8,fr-CA;q=0.6,en-US;q=0.4,en;q=0.2' \
     --header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
     http://www.example.org/

Then, another pass

wget -i list.txt --mirror -e robots=off -k -K -E --no-cache --no-parent \
     --user-agent='User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36' \
     --header='Accept-Language: fr-FR,fr;q=0.8,fr-CA;q=0.6,en-US;q=0.4,en;q=0.2' \
     --header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
     http://www.example.org/

3. Do some cleanup on the fetched files

Here are a few commands I ran to clean the files a bit

  • Remove empty lines in every .orig files. They’re the ones we’ll use in the end after all

    find . -type f -regextype posix-egrep -regex '.*\.orig$' -exec sed -i 's/\r//' {} \;
    
  • Rename the .orig file into html

    find . -name '*orig' | sed -e "p;s/orig/html/" | xargs -n2 mv
    find . -type f -name '*\.html\.html' | sed -e "p;s/\.html//" | xargs -n2 mv
    
  • Many folders might have only an index.html file in it. Let’s just make them a file without directory

    find . -type f -name 'index.html' | sed -e "p;s/\/index\.html/.html/" | xargs -n2 mv
    
  • Remove files that has a .1 (or any number in them), they are most likely duplicates anyway

    find . -type f -name '*\.1\.*' -exec rm -rf {} \;
    

How to run your own OAuth Identity provider service

Generally, we connect our application against a provider so it can share details about a user. Most of the documentation you’ll find online explains how to use their service, but very few outlines concisely how it is if you want to be your own provider and share state across applications you also manage.

Documentation would generally allow your third party app to let users use their information from another site such as Facebook, GitHub, Twitter, etc.

But what if you wanted to share information across your web applications in a similar way?

This post is a quick summary of how things works so you can get acquainted with the basics.

Whitelist

Big sites aren’t generally one big code repository but a set of separate components. A way to make each component to share your account details is most possibly by making a difference between their own infrastructure and third parties.

If your app were to use an external resource such as Google, the process would end up making Google users to be asked if they really want to share their information with you. This is why they would get a dialog similar to this.

img

While its OK to ask confirmation from a user if he wants to share his details with an external site, in the case of two components from the same site can share this information transparently.

If you are your own Identity Provider, you can configure your relying parties as “whitelisted” so that your accounts system don’t display such dialog.

Becoming your own Identity provider

In the case of WebPlatform.org we wanted to become our own Identity provider and found that we could deploy our own fork of Firefox Accounts (“FxA”) would allow us to do so.

The way its designed is that we have an OAuth protected “profile” endpoint that holds user details (email, full name, etc) as a “source of truth”. Each relying party (your own wiki, discussion forum, etc) gathers information from and ensure it has the same information locally.

In order to do so, we have to make a module/plugin for each web application so we it can query and bootstrap users locally based on the accounts system. We call those “relying parties”.

Once we have relying party adapter in place, a web app will have the ability to check by itself with the accounts server to see if there’s a session for the current browsing session. If it has, it’ll give either an already generated OAuth Bearer token, or generate one for the one for the service in question –the “SSO” behavior.

With the OAuth Bearer token in hand, a relying party (i.e. the WebPlatform.org annotation service) can go read details from the “profile” endpoint and then check locally if it already has a user available.

If the relying party doesn’t have a user, it’ll create one.

Each relying party is responsible to sync its local user and session state based on what the accounts service gives.

Quelques bouts de code pour automatiser le déploiement

Automate ALL THE THINGS!

Ce billet n’est qu’un simple «link dump» pour retrouver parmi plusieurs notes éparpillés. Je compte éventuellement publier la totalité de mon travail dans des projets publics sur GitHub une fois la boucle complétée. Le tout sans fournir les données privés, évidemment.

Faire le saut vers l’automatisation demande beaucoup de préparation et je prends le temps de publier ici quelques bouts de code que j’ai écrits pour compléter la tâche.

Au final, mon projet permettra de déployer un site qui s’appuie sur un cluster MariaDB, Memcached, une stack LAMP («prefork») lorsqu’on a pas le choix, une stack [HHVM/php5-fpm, Python, nodejs] app servers pour le reste servi par un frontend NGINX. Mes scripts vont déployer une série d’applications web avec toutes les dépendances qui les adaptent géré dans leur propre «git repo» parent. Dans mon cas, ce sera: WordPress, MediaWiki, Discourse, et quelques autres.

Requis

  • Instantiation à partir de commandes nova du terminal, crée une nouvelle VM mise à jour et son nom définit son rôle dans le réseau interne
  • Les VMs sont uniquement accessible par un Jump box (i.e. réseau interne seulement)
  • Un système regarde si un répertoire clone git à eu des changements sur la branche «master», lance un événement si c’est le cas
  • Chaque machine sont construites à partir d’une VM minimale. Dans ce cas-ci; Ubuntu 14.04 LTS
  • Système doit s’assurer que TOUTES les mises à jour sont appliqués régulièrement
  • Système doit s’assurer que ses services interne sont fonctionnels
  • Dans le cas d’une situation où une VM atteint le seuil critique OOM, la VM redémarre automatiquement
  • Le nom de la VM décrit son rôle, et les scripts d’installation installent les requis qui y sont affectés
  • Les configurations utilisent les détails (e.g. adresses IP privés et publiques) de chaque pool (e.g. redis, memcache, mariadb) et ajuste automatiquement les configurations dans chaque application
  • … etc.

Bouts de code

Billets inspirants sur le sujet

Thoughts about learning in the web developer job, what managers might be missing

After reading an article titled What the ‘Learn to Code’ movement is forgetting: Existing developers, I couldn’t resist commenting. Sadly, I couldn’t add my own comment because their commenting is allowing only paying members so I am sharing it here.

Developers do not simply convert paragraphs of text requirements into “morse”. Obviously. Its a complex craft and like anything else, its all about people you are working with. There are passionate programmers and also the ones who only wants to do the bare minimum. In between, its all about leaving room for creativity.

What a programmer’s day looks like

Let’s remind ourselves what a programer is required to do has a lot of complexity and has to deal with legacy, unknowns, and delivery dates. Not everything can be foreseen: how the team members will work together, their capabilities, ego, and most importantly, whether or not each of them will be in “the zone” long enough during work hours.

The time they are at their optimal is what you want to have most from them, its that time where they can do amazing things, isn’t it. Being developer is about solving puzzles, review code or existing components they can leverage or would need to refactor. Being interrupted by numerous meetings to talk about something they are already trying to figure out doing isn’t helping them.

A good way to help them is to set in place asynchronous communication channels such as IRC, and code review practices. In some open source communities, merge to the master branch requires code review, a common practice is to have a bot to announce commits on an IRC channel. That’s the best of both worlds, you let developers be in their zone AND collaborate. As for paired-programming, they can just announce at the morning SCRUM meeting that they would like to spend time, or just ask for it on IRC.

Learning

As for learning, passionate developers already keep themselves in the loop of best practices and how to improve their craft. The experienced ones might even have ideas on how do things that you never thought of.

I’ve seen companies who allows participation to open source projects, and/or share their knowledge openly on sites such as Stack Overflow and documentation sites. This is another great way to make developers more engaged to their role, and stay in your company.

When I think of learning resources for web developers in general, there is an independent, and vendor neutral, resource where web developers can learn. The site is convened by the W3C, and sponsored by industry leaders to create just what we’ve been missing all along. The site is webplatform.org, and I feel like it’s not enough set forward by the industry. Maybe because its not complete, but its a wiki, anybody can participate.

Full disclosure; I work full time on the webplatform.org project, but note that regardless of my current position, I would participate to the site regardless of whether or not i’m hired to do so.

Photo credits: Anthony Catalano, Morse Code Straight Key J-38

Answers I gave for an article about the impacts of Heartbleed

Credits: Codenomicon

Last spring, I had been invited to answer questions for an article about Heartbleed. It was an invitation extended by my friends at SaltStack. Since I use extensively automated deployment techniques with Salt Stack, they thought i’d have things to say.

I accepted to answer the questions on my personal time as the DevOps lead of WebPlatform.org, rather than directly as a W3C team member. And that, even though I had big help from my manager, Doug Schepers, and a few colleagues from the team. Their help was much useful to review and add stats to my arguments.

The article has been published on Dell.com’s Tech Page One blog but only quotes a few pieces of my original contribution. Here is the full answers I gave to the questions.

What made Heartbleed such a tremendous threat?

It hit more than the epicenter of our whole Internet infrastructure, it proven true every sysadmin worst nightmare and on top of that, it impacted everybody.

OpenSSL is so widely used (estimated 2/3 of the web) that many organizations were impacted. This vulnerability has been introduced as a programming mistake for about 2 years. What gives the chills is that since the person who is exploiting the fail do not need to be the “man in the middle” nor need to gain terminal access to the servers, there are no traces left behind. We therefore had no idea how much it was used and by who.

It’s important to remember that the attacker couldn’t know what data he would get either, or if he would get anything interesting at all. And if the “good guys” didn’t know about heartbleed for so long, probably not many “bad guys” did, either.

In consequence, the world had to make a major effort to update a lot of software and people to replace many passwords, but the major impact is probably more psychological: we don’t know what was stolen and we don’t know if there is another “heartbleed” hidden in our servers.

On the bright side, the old advice we give to end users: change your password often, is still valid. This was a nuisance, for sure, but at least heartbleed didn’t change anything there.

Is there anything about the current computing landscape that made the threat greater than it might have been in the past?

The time it takes to gather infrormation through such vulnerability and the set of tools available today has increased dramatically. Ther are network port scanners, for instance, that can report with high 90-percent accuracy in less than 10 minutes.

When attackers gets insight of a vulnerability, its not hard to get the information they are searching for. In the case of heartbleed, it was as simple as searching for machines listening on TCP port 443 and keep a
list of servers using a given version of OpenSSL that is known to have the mistake. Not something hard to do at all for the attackers.

Todays’ computing landscape has better tools, but so the tools to crack them. A better solution is to strenghten the infrastructure and educate end users. Otherwise, the chain will remain as strong as its weakest link.

If there is a thing that our society should do more is to continue teaching sane privacy techniques such as strong and varied passwords.

Hopefully we’ll get to a point where companies would rely on “harder to gain” information than you mother maiden name or your date of birth.

What were the most important things that companies needed to do in the face of Heartbleed?

Companies that doesn’t maintain adequate server management techniques and tools had a lot of (manual) things to do.

The priority was to upgrade to the patched version all affected parts of the infrastructure, ensure that all system that stores passwords gets new passwords as soon as possible.

Of course, the issue had to be fixed as quickly as possible, and without interrupting the services… or belating deliverables that they were already working on.

Therefore, one of the most appreciated thing a system admin has is a way to manage his fleed of servers remotely. In the old days, we only had SSH —hopefully that one doesn’t have its own “heartbleed” problem too— but we now have tools to execute remotely such as Ansible and Salt stack that can help us quickly get the an accurate picture of the whole infrastructure.

Could a Heartbleed-level/style hack happen again?

Before heartbleed, I was an optimistic person. Besides giving chills to everybody, what heartbleed did is a loud “wake up” call.

It is only a matter of time before we learn something else to arise. Computer security is still based on “hard enough to crack” secrets and an accepted set of annoyances. This is where education and better automation tools are of great help. It’d ease the justification of existence, maintenance, and deployment of such measures to everybody.

How can companies best protect themselves and their customers moving forward?

Education, and automation; in that order. Its better to have a good answer —ideally quickly— than a quick wrong answer.

I hope to see training about security practices more often. Its not the lack of training that is the problem, but the people to take time to learn the techniques. Organizations such as the OWASP, —a group who teaches common security mistakes to web developers— is educating security vulnerabilities.

There are dozens of other potential oops. One of the most common type of security breach is what we call “SQL injection”. That’s yet another programming mistake that sends user input directly to the database server without filtering.

img

In the case of Heartbleed, it was a similar kind of programming error. A way to achieve peace of mind includes among others; Testing, proof techniques, “sandboxing” (protected memory), making software Open Source all helps to catch those errors.

While its still a hard problem to find the errors in all that code, using those techniques has the merit to have more than a limited set of eyes on the code.

But all security can be completely overcome by a small human mistake answering private information to the wrong person. Education about how to validate and detect the real collaborator to the potential thief is another challenge in itself.

Many people are talking about it for ages. The recent events makes their teachings more relevant than ever.

Credits

Heartbleed graphic: Codenomicon
Comic strip: XKCD #327

Processus de création d’une VM faisant partie d’un parc géré par Salt Stack

A mon emploi actuel je gère un parc de machines virtuelles (VMs) qui est automatisé par un système de gestion de configuration appelé Salt Stack.

Salt stack est un outil permettant de décrire quel est l’état désiré d’un serveur. Comme d’autres outils avec une utilité similaire, Salt Stack peut être utilisé dans des environnements hétérogènes sous GNU/Linux, Mac OS, et Windows, FreeBSD, etc.

L’environment que nous utilisons est un serveur avec des «blades». Chaque «blade» fournit les services créant un cluster OpenStack. Dans le futur, nous risquons d’avoir plus d’un fournisseur OpenStack. Pour automatiser comme nous l’aimons, nous utilisons grandement la ligne de commande avec le paquet python-novaclient.

Chaque machine virtuelle roule une version LTS («Long Term Support») de Ubuntu.

Absolument toutes les configurations sont appliqués via Salt Stack, la seule chose qui est fait manuellement en
ce moment est de créer la nouvelle instance, et de l’ajouter au «master» de Salt Stack.

Même là, ça risque de changer lorsque nous aurons déployé Salt Cloud.

Procédure

Mise à jour Mars 2015: Un nouvel article sur le même sujet a été écrit (en anglais) et illustre comment faire une nouvelle VM avec encore moins d’étapes

  1. Boot une nouvelle node avec Nova

    [email protected]:~$ nova boot --image lucid-minion-template --flavor wpdn.large --key-name renoirb app6
    
  2. Donner un nom en fonction du type de serveur a déployer avec un numéro à la fin. Exemple: app6

    NOTE Dans mon cas, j’ai notamment: app, db, memcached, etc.

  3. Ajoute l’adresse floating dans /srv/pillar/nodes/init.sls comme les autres

    nodes:
      master:
        public:  ####IP PUBLIQUE CACHÉE####
        private: 10.0.0.1
    
      app1:
        public:  ####IP PUBLIQUE CACHÉE####
        private: 10.0.0.7
    
      memcache2:
        public:  ####IP PUBLIQUE CACHÉE####
        private: 10.0.0.4
    
      app5:
        public:  ####IP PUBLIQUE CACHÉE####
        private: 10.0.0.3
    
  4. Prend le fichier /home/ubuntu/runme de n’importe quel autre serveur et colle le dans la nouvelle machine. Puis execute (sudo /bin/bash runme)

  5. Ajouter une ligne dans le nouveau serveir dans /etc/salt/minion.d/master.conf

    id: app6
    

    … Voir les autres nodes

  6. Restart salt-minion

    [email protected]:~$ sudo service salt-minion restart
    
  7. Ajoute la clée au master

    [email protected]:~$ sudo salt-key -a app6
    

    … Le ‘-a foo’ est optionnel et tu peux lister Les nodes.

  8. Run state.highstate

    [email protected]:~$ sudo salt app6 state.highstate
    
  9. Uploader le code via rsync dans la nouvelle app node, puis re-rouler state.highstate (certains scripts prennent pour aquis que le code est déjà déployé)

    [email protected]:~$ sudo salt app6 state.sls code.node_app
    [email protected]:~$ sudo salt app6 state.highstate
    

    Comme je disais, parfois, le premier state.highstate ne marche pas a cause du code pas déployé.

  10. Rafraichir les autorisations pour storage

    [email protected]:~$ sudo salt 'storage*' state.highstate
    [email protected]:~$ sudo salt 'monitor*' state.highstate
    
  11. Updater le hosts file de quelque nodes

    [email protected]:~$ sudo salt 'app*' state.sls hosts
    [email protected]:~$ sudo salt 'db*' state.sls hosts
    [email protected]:~$ sudo salt 'memcache*' state.sls hosts
    

Project idea: Creating a home made OpenStack cluster for development purposes

Think about it. How about using spare computers to create a homemade OpenStack cluster for development.

We can do that from our cloud provider, or create a separate project or even use Wikimedia’s OpenStack infrastructure allowance for the project.

With such setup, one could work locally with his Salt stack (or Puppet, or Ansible) deployment schemes, try them, trash VMs, rebuild.

The beauty of it would be that it could be made in a fashion that would not even modify the computer running the VMs. The cluster member running OpenStack hypervisor would be installed seeded through net boot. Not booting from the network would revert the computer back as if it never been used.

Here is what I think would require to make this happen.

Limitations

  • Not use Computer/Laptop local hard drive
  • Rely only on net boot

Material

  • 1..n Computers/laptop supporting netboot
  • 1 Storage device supporting one or more storage protocol (nfs, samba, sshfs)

Hardware requirements

  • 1 VM providing tftp, dhcp, dns to serve as net boot server that should run outide of the cluster (“Networking node”)
  • 1 VM image of OpenStack controller (“OpS controller”)
  • 1 LiveCD+persistent image with OpenStack preinstalled, configured to use storage device credentials as it’s root filesystem (“OpS Hypervisor”)

Distribution choice factors

  • Networking node can be the smallest Linux possible, on a RaspberryPI, or a modified Router or Network Attached storage device?
  • OpS Hypervisor to be among the supported OpenStack distributions (I think a Ubuntu precise 12.04 LTS or a variant such as Puppy Linux might work too)

To be continued…

I will keep you posted whenever possible on the outcome of my research.

Did you ever do this in your infra. Leave a comment.

I am joining W3C to work on the WebPlatform project!

webplatform.org

In a recent post, I was explaining that I resigned my position.

The new exciting project that I was referring to is the WebPlatform project at the W3C.

For people who do not know what the W3C is, it is the main international standards organization for the World Wide Web.

The WebPlatform project is a rapidly growing web development collaborative documentation website.

I would even call it the Web developer’s missing manual for web developers to build quality web applications using the latest techniques.

As for the sponsors, it is backed by well known companies, that we refer to as stewards, such as Adobe, Google, Mozilla, and Microsoft to name a few.

I will contribute to the site as a Developer Operations engineer. My work will be basically to enhance the features and manage the server infrastructure of the site as much as continuing what I already do as a hobbyist speaker.

Roughly, it consist of everything I always did or dreamed of:

  • Work on a flagship site with heavy traffic usage, with and for respected professionals around all the world
  • Use my favourite operating system and tools in System Administration
  • Technical liaison with Open-source communities
  • Develop and maintain the community platform, with web development, ensuring application performance, and implement continuous deployment
  • Speak and participate at conferences related to web standards
  • Contribute to moving the web forward, full-time (!)

If you are living in Montreal and you would like me to share with you on how to participate in this exciting project, just send me an e-mail (my first-name AT w3 dot org) and I’ll gladly answer.

I have just resigned from my new job to start on an exciting project!

In the last months, many things happened to me.

First, I worked for almost two years in a contract at Ericsson, but it ended. Therefore, I started searching for a new job. I had some offers and I had a hard time to pick one.

However, after all, I have just received a new offer to work for the W3C; something I cannot that I could not turn down.

I am very excited to have this opportunity. I would even say more: it is a honour.

I will explain more about it in another moment.

For now, I would just like to thank TEKsystems‘ team and to mention the many good things about working there:

TEKsystems

TEKsystems is a company based in the USA and their main solution development center is located in Montreal Downtown. Most of the projects are web based business applications serving many customers in the united states.

Among the things I enjoyed most were:

  • The management team uses Agile methodologies wherever applicable;
  • The leadership is strong, and the team leads are skillful;
  • The multiple coding technologies in one place: Java, PHP, Frontend, iOS, Android, .NET,
  • The multidisciplinary team: Architecture, Business Analysis, User experience;
  • The members of the team, at the end of the project, get shuffled in the office to start a new one;
  • The internal structure to create mini-projects so we can learn and contribute from other employes;
  • The fact that the office is located in Montreal Downtown, and right beside McGill University where I am starting a Continuing Studies Courses certificate

I worked there for the last month and a half, and it is amazingly well organized company. Based on my own experience in different work contexts, I can say that working with them for web development projects was enjoyable.

If it wasn’t because of my new opportunity, I would have stayed there for a long while.

Procédure pour avoir un environnement de dévelopement local facile à configurer avec Apache

Je ne sais pour vous, mais je ne peut plus programmer sans avoir l’environement serveur localement sur ma machine. Changer ou ajouter un fichier VirtualHost pour chaque nouveau projet est assez répétitif. Il doit y avoir une façon automatique de le faire?

Oui.

Ça s’appelle VirtualDocumentRoot

J’ai ce tutoriel qui traîne dans mon Wiki personnel depuis des lustres, et c’est maintenant que je commence a migrer mes projets sous NGINX que je décide de le mettre en ligne. Il n’est jamais trop tard pour publier.

Cette méthode de configuration répond exactement au besoin précis de ne pas avoir a configurer un hôte virtuel apache pour chaque projet.

Avec cette procédure, vous n’aurez qu’a maintenir votre fichier hosts, le reste suivra tout seul.

Vous pouvez appliquer cette technique avec n’importe quelle version du serveur http “Apache”. Cette procédure peut même être faite si vous développez sous Windows ou Mac OS avec les distributions du serveur HTTP Apache sous Windows telles que MAMP, XAMPP, et EasyPHP.

Pourtant avec un serveur web local, ce type de configuration est possible depuis longtemps, il faut simplement savoir comment ça s’appelle: VirtualDocumentRoot.

Voici comment je configure mon environnement LAMP depuis quelques temps.

Procédure

Établissement du standard

Tout commence par une certaine convention. Avec celle-ci, tout devrait suivre automatiquement.

L’idée est de pouvoir accéder a un l’espace de travail du projet A du client B sur ma machine locale. L’adresse locale n’est plus localhost, mais quelque chose de plus explicite.

Ce que j’apprécie le plus de cette méthode car elle permet de conserver dans un dossier parent tout ce qui est spécifique pour le projet et le client. Le code a exécuter qui soit dans un sous-dossier ne feait que du sens.

Par exemple, un projet appelé projectname du client client serait classé dans un dossier sous le chemin /home/renoirb/workspace/client/projectname.

Le code du projet web serait accessible via le serveur web à l’adresse http://projectname.client.dev/ qui pointe vers l’adresse IP de la station de travail locale.

L’espace de travail du projet

IMPORTANT
Il faut que les noms de dossiers soient en minuscule et aucun espace, ni caractères accentués, sinon le serveur Apache risque de ne pas trouver le dossier. Principalement parce que l’adresse entrée dans le navigateur est convertie en bas de case, et que généralement un système d’exploitation qui se respecte fait une différence entre, par exemple ‘Allo’ et ‘allo’.

La convention suggérée va comme suit:

  • chaque projet est classé dans un chemin prévisible, similaire à /home/renoirb/workspace/client/projectname
  • le projet a un dossier web/
  • les autres dossiers au même niveau que web/ peuvent être n’importe quoi d’autre.

Idéalement, la logique applicative ne devrait pas être visible publiquement de toute façon. Seulement le fichier principal appelle l'”autoloader” en dehors du DocumentRoot.

De cette façon le vous pouvez classer tout vos projets du même client, et séparer par projets.

La procédure tient aussi en compte
* L’utilisateur courrant puisse écrire dans son dossier workspace/ avec Apache2 comme s’il était son propre utilisateur avec mpm-itk
* Le nom de domaine utilisé définit dans quel dossier de l’utilisateur chercher

Procédure

  • Assurer que les modules sont chargés
     sudo a2enmod vhost_alias
  • Ajouter le fichier default a la config de apache
     sudo vi /etc/apache2/ports.conf
  • Vérifier qu’il y a ceci:
    NameVirtualHost *:80 
    Listen 80 
    UseCanonicalName Off
  • Modifier le fichier de config du VirtualHost par défaut
  • Fichier de configuration par magique
    sudo vi /etc/apache2/sites-available/default
  • Verifier qu’il y a ce bloc dans <VirtualHost ...>:
    <IfModule mpm_itk_module>
        AssignUserId renoirb users
    </IfModule>
  • Remplacer la mention DocumentRoot par ce format:
    VirtualDocumentRoot /home/renoirb/workspaces/%1/%0/web

Sources

Conférence: Comment évaluer le niveau de qualité d’un site web selon les techniques d’intégration web d’actualité

Sur scène durant ma présentation donnée au Web-In 2013 quelques mois avant le WAQ

Cette semaine, je fait une conférence pour la deuxième fois de ma présentation qui décrit les pratiques en intégration web qui sont d’actualités.

La première fois que j’ai présenté c’était pendant le WebIn 2012, et cette-semaine c’est à Québec pendant la conférence Web à Québec 2013 (WAQ).

Cette fois-ci, j’aurai l’honneur de présenter tout juste avant une personne que j’admire; Jonathan Snook.

Audience

L’audience de cette présentation est principalement les gens qui travaillent avec le web, qui engagent des programmeurs/intégrateurs et qui veulent comprendre les dernières tendances et le pourquoi elles sont populaires.

Comme une valeur ajoutée, j’illustre avec des exemples concrêts et compare les outils vis à vis leur utilité.

 

Synopsis

La façon de concevoir des sites Web a beaucoup évolué au cours des deux dernières années. Concevez-vous encore vos sites comme en 2009? Trouvez-vous vos sites lents? Êtes-vous web-responsable? Imaginez si vous étiez capable de gagner du temps.

Comment pouvez-vous être certain que votre pigiste, employé ou fournisseur fait un travail de qualité?

Nous allons couvrir des techniques clé qui permettent d’optimiser le travail, comment être plus efficace avec votre équipe technique et savoir jauger le son niveau d’expérience.

L’objectif de la séance est de donner des outils pour évaluer le niveau de qualité d’un site web; des pistes pour en améliorer la performance et s’assurer de pouvoir en entretenir le code après plusieurs révisions.

 

Trois questions qui seront répondues dans la conférence

  1. Que faire pour optimiser la communication lors de l’établissement du cahier de charge
  2. Quelles techniques ont été découvertes pour éviter le gaspillage lors de la réalisation
  3. Indicateurs de qualité qu’on peut retrouver dans le HTML/CSS

 

Les «slides»


Télécharger en format PDF

 

Réception

Je ne m’attendait pas vraiment au succès de ma présentation durant la conférence, c’était une belle surprise.

Mon ami François Légaré a pris cette photo:

Il y avait foule lors de ma présentation au Web à Québec

 

Merci à

  1. @gabiviana: Gabi Viana pour les visuels
  2. @joplam: Josée Plamondon pour la révision
  3. @cybik: Renaud Lepage pour la révision
  4. @vivrass: Martin Provencher pour l’inspiration de l’angle technique à couvrir.

Choosing a framework, how I personally define what is “hot” about them, an evaluation process (part 1)

This post is all about what I think should be used to define quality in your choice of programming framework tool-set. This post is the first one of a collection of (unknown for now) number of posts.

This week, I had to bring to a team of people what I consider a good PHP framework and why I am sold. I am aware that I may sound blinded with my choice, but this post is about explaining my own guidelines about my choice. I work with them since many years and that made me try many in the PHP ecosystem.

Everything goes with PHP Sauce

Things can get messy with PHP, that’s why we have to be very cautious on how we structure and consider recommendations from other good programmers.

Since we know that PHP can be a “sauce where everything goes”. Structure is crucial. There is many frameworks, many libraries, and to be able to build solutions that will solve more usefull and important matters, we need to agree on how to collaborate.

I do not claim to be a guru, I have done stuff and thanks to people I mention here, I feel I am getting better at the craft and I thought my reflection process was worth sharing.

 

 

Continue reading “Choosing a framework, how I personally define what is “hot” about them, an evaluation process (part 1)”

Confirm form before submitting using Twitter Bootstrap in a Modal window [Snippet]

Idea

Have you ever wanted to create a way to confirm input before actually submitting the form. Automatically.

My snippet’s goal is exactly for that purpose. The idea is that it reads the form inputs and labels, and generates a modal window (like a lightbox) and asks confirmation. This project is a contribution I do to Twitter Bootstrap and jQuery-bootstrap-scripting (“dialog2”) to add on any form using Twitter Bootstrap form markup convention the capability to create a confirmation modal window.

The concept is that some forms needs to have some confirmation and requires more than “are you sure”, but also what you are submitting.

How it works

When the pages load, it executes in this order the following:

  • Whether there is a form[data-behavior=confirm], and modify the form default submit button to replace it with a link (idea being that if there is no javascript enabled, you can still submit!)
  • on a Click action on the newly created a.confirm-toggle, it fires up some html manipulation and creates a modal window using jQuery Bootstrap Scriptin’s Dialog2 plugin

Word of advice

I have to warn you though. The solution is not yet complete. It works well with input[type=text] and some select. But I need to add more field types as time goes. That’s why I created this Fiddle to give room for improvement. Feel free to collaborate. When the solution will be strong enough, i’ll contribute it to both projects in their respective syntax (dialog2 doesnt initialize the same way as Bootstrap’s modal()

Known limitations

  • Works only with select and input[type=text], and breaks bad with radio and checkbox
  • Works only with ONE form in the page, for now.
  • ¨

The code

You can play with this Fiddle in the meantime i make it ready-er

The javascript

 
    /**
     * Use form's content into a confirmation modal
     *
     * Using: http://nikku.github.com/jquery-bootstrap-scripting/
     * Requires: Twitter Bootstrap, dialog2
     *
     * @author: Renoir Boulanger
     */
    if ($('form[data-behavior=confirm]').length >= 1) {
 
        console.log('run.js: document ready : Applying confirmation, ' + $('form[data-behavior=confirm]').length + ' times');
 
        $('form[data-behavior=confirm] select').click(function() {
            var dataValue = $(this).find(':selected').text();
            $(this).attr('data-value', dataValue);
 
            // Debug console
            console.log('Adding data-value at: ' + dataValue);
        });
 
        $('form[data-behavior=confirm] .confirm-toggle').replaceWith('<input id="confirm-submit" type="submit" class="is-hidden"/>' + '<a href="#" class="confirm-toggle btn btn-primary">Continue</a>');
 
        /**
         *  Since we know javascript is executed so far, lets handle it with the confirmation.
         *
         *  That way, no javascript-enabled browsing user will be able to use the form. #progressiveEnhancement
         *
         *  Do the work.
         */
        $('form[data-behavior=confirm] .confirm-toggle').click(function(event) {
            event.preventDefault();
 
            // Get form content
            var form = $('form[data-behavior=confirm]').clone().attr('id', 'cloned'); //.appendTo('body');
            var i = 0;
            form.find(':input:not([type=hidden])').each(function() {
                var field = $(this);
 
 
                if (field.is('select')) {
                    fieldValue = $(this).attr('data-value');
                    if (fieldValue === undefined) {
                        fieldValue = field.find(':selected').html();
 
                        // Debug console
                        console.log('fieldValue was undefined, setting to : ' + fieldValue);
                    }
 
                    // Debug console
                    console.log('fieldValue is : ' + fieldValue);
                } else {
                    fieldValue = field.val();
                }
 
                // Remove undefined field (they are useless)
                if (fieldValue === '') {
                    field.parent().parent().addClass('empty-field-resolved');
                }
 
                // Debug console
                console.log('Field ' + i + ' :' + fieldValue + ' for #' + field.attr('id'));
 
                // Wrap fieldValue in a tag, Tested in IE7!!
                field.after($('<span class="value">' + fieldValue + '</span>'));
 
                // Remove the field itself, we only want to see the resulting
                field.remove();
 
                i++;
            });
 
            // Work stuff out for modal window, copying content, and building modal into the DOM
            var decorate = $("<div id=\"modal-placeholder\"><div class=\"modal-builder\"></div></div>");
            var buildup = decorate.find(".modal-builder").html(form);
 
            buildup.appendTo('body');
 
            // Debug console
            console.log('Appending #modal-placeholder in body, ready to call dialog2()');
 
            // Remove not needed anymore stuff
            $('.modal-builder .help-block, .modal-builder .input-append, .modal-builder .form-actions').remove();
 
            $('.modal-builder').dialog2({
                title: "Are you sure?",
                id: "confirm-modal",
                modalClass: "modal-wide fade in",
                closeOnOverlayClick: false,
                closeOnEscape: false,
                initialLoadText: "Loading in progress...",
                buttons: {
                    Confirm: {
                        primary: true,
                        click: function() {
                            // Debug
                            console.log('Inside dialog2() clicked Confirm');
 
                            $('#confirm-submit').click();
                        }
                    },
                    "Forgot something": {
                        click: function() {
                            // Debug
                            console.log('Inside dialog2() clicked cancel');
 
                            $(this).dialog2("close");
                            $('.modal').remove();
                            return false;
                        }
                    }
                }
            });
            // Do my own cleanup. Remove potentially bogus nodes
            $('#modal-placeholder, .modal-header .close, .control-group.empty-field-resolved').remove();
        });
    }

The CSS

Minimally…

.modal-wide {
  overflow:visible;
  width: 650px;
}
 
.modal-builder .control-label {
      padding:0 !important;
      font-weight:bold;
}
 
 /* In case you use a span.required-star
  * with title="required field", I don't want to
  * have them in modal
  **/
.modal-builder  .required-star {
    display:none;
}
 
.modal-builder .hide-in-confirm {
  display:none !important;
}

Tell me in the comments if you want the LESS block I created

The Form

Always using Twitter Bootstrap markup, A form minimally needs

What is really required here is

  • Form tag has data-behavior="confirm" attribute
  • Form submit button has at least the class name confirm-toggle

Just use any combination of Twitter Bootstrap Form patterns

<form class="form-horizontal" method="post">
<fieldset>
<div class="control-group hidden-in-confirm">
<!-- This block will be hidden -->
            <label class="control-label" for="input01">Text input</label>
<div class="controls">
<input id="input01" class="input-xlarge" type="text" />
 
In addition to freeform text, any HTML5 text-based input appears like so.</div>
</div>
<!-- ... the form ... -->
<div class="form-actions">
            <button class="btn">Cancel</button>
<input class="btn btn-primary confirm-toggle" type="submit" value="Continue" /></div></fieldset>
</form>

Possible enhancements paths

My solution fit for what I needed. A two parameters to add confirm in a modal using the project’s markup. Sadly I read after doing it that there is some plugins I could have considered such as the Form plugin

.

End word

I am going to prepare this post and propose a pull request to the jQuery Bootstrap Scripting AND Bootstrap project in a day or two.

Ressources

  • http://nikku.github.com/jquery-bootstrap-scripting/#
  • http://twitter.github.com/bootstrap/base-css.html#forms