A New WordPress Script

I’ve been working on some scripts to help with the deployment of sites to Apache lately. My original script was pretty simple. I used it to automate the creation of vhost entries when I added a new site to the server. Since one of my clients actually uses WordPress for all it’s sites, and we have been working to migrate all the sites from a multi-site installation to stand alone sites, I needed an easy way to setup a Wordpress site on the new server. Ideally I wanted to be able to pass a script the site name and it create the folder in var/ , install the latest copy of WordPress, install the base themes and base plugins and setup the proper vhost entry in etc/

Ultimately I’d like to be able to incorporate this into a website so that we can install a brand new standard base install of WordPress with the press of a button. (I don’t like how gists are embedded in jekyll so I am usinga code block + including the link to the gist)

https://gist.github.com/weatheredwatcher/14dd0d50ee3e876746ff#file-wp_install-py

wp_install.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
import os, sys, argparse, wget, tarfile
if not os.geteuid()==0:
    sys.exit("\nMust be run by root\n")

def main(argv):
    ##define options here
    server_root = '/var/www/vhosts/'
    plugin_path = '/var/resources/plugins/'
    theme_path = 'var/resources/themes/'
    domain = ''
    options = ''

    parser = argparse.ArgumentParser()
    parser.add_argument('domain')
    args = parser.parse_args(argv)
    domain = args.domain
##create folder    
    if not os.path.exists(server_root):
        print "Creating " + server_root
        os.makedirs(server_root)
    if not os.path.exists('tmp'):
        print "Creating Temp Folder"
        os.mkdirs('tmp')
##download wordpress
    download = wget.download('https://wordpress.org/latest.tar.gz')
    archive = tarfile.open(download, 'r:gz')
    archive.extractall(server_root)
    os.rename("/var/www/vhosts/wordpress", "/var/www/vhosts/" + domain)
##install base theme
##install base plugins

##create vhost file
    filename = domain + ".conf"
    target = open(filename, 'w')
    vhost = """
  <VirtualHost *:80>
          ServerName %s
          ServerAlias www.%s
          DocumentRoot /var/www/%s
          <Directory /var/www/%s>
                  Options -Indexes FollowSymLinks -MultiViews
                  AllowOverride All
          </Directory>

          CustomLog /var/log/httpd/%s-access.log combined
          ErrorLog /var/log/httpd/%s-error.log

          # Possible values include: debug, info, notice, warn, error, crit,
          # alert, emerg.
  </VirtualHost>""" % (domain, domain, domain, domain, domain, domain)
    target.write(vhost)

if __name__ == "__main__":
    main(sys.argv[1:])

My standard disclaimers apply here. If you do not know python well enough to follow the code, I am not liable for any issues that might come up trying to use this code. If you know python better then me (not a huge strectch!) please leave any comments below on your opinions on how to make this better!! Also, I am happy to help anyone out who wants to use this for something and need assistance. My python is not by any means stellar, but I can get most things done with it!

Server Stuff a Developer Needs to Know (Part 1)

Recently I have been moving more and more away from development and more into System Administration. Still, my skills and experience in programming means that I also have a good handle on developers. So I have been making a effort to move in the direction of DevOps. I do some consultation work for a startup that is currently coming out of the USC Incubator. I mostly contribute as the Architect and help provide direction to the infrastructure and coding best practices as well as run their servers. So this fits perfectly in the role of DevOps. One of the things that has really begun to frustrate me is the lack of BASIC linux skills in developers that should know better. I am not asking you to setup a fucking mail server or configure a load balancer. I can do those things just fine without your help. But there are certain things that a developer NEEDS to know how to do. So, here is the first in a series of posts on some things that in my humble opinion a developer should be able to do without me holding their goddamn hand!

Logging into and Deploying Code to a Server

Guys, this is not that hard. First of all: A secure server should be using ssh — NOT ftp. Do not ask me for FTP access to my server. I will not give it to you. No matter how many times you ask me to.

You can connect to a server using ssh like this:

ssh username@hostname

If you do not want to use a password, or if I decide not to give you one, you will need to provide me with a public key. It is fairly easy. On a Linux/Unix system, in the termianl type:

ssh-keygen -t rsa -f ~/.ssh/id_rsa

This will give you a key pair. Email me the one with the .pub extension

id_rsa.pub

Running Windows? Either install git and use git-bash or install Cygwin and run the Bash shell. The above commands will work for you too.

Then you should be able to log in without a password. Having issues?

ssh user@hostname -i ~/.ssh/id_rsa.pub

Rather do it yourself? Once you have logged in to the server with your given user name, you can add your key to the authorized_keys file.

It needs to be in a file called ~/.ssh/authorized_keys

Now on to deployment. Most of the time there is going to be some procedure in place for production deployments, but not so much on a dev server. You should be able to upload your code to the server. Aside from using git, which is just easy, I personally like secure copy or scp. It is fairly simple to use, especially if you already have a key setup on the remote server. The syntax is easy:

scp -r localfolder/ user@remotehost:/path/to/server

If you would rather use rsync:

rsync -r localfolder/ user@remotehost:/path/to/server

It’s not rocket science people! Learn some basic Linux!

DevOps: Bulding Projects With Ant

I’ve been running a build process for an application that I am working on that is rather complicated. Originally I was manually managing the build process, with a few scripts to supplement my procedures. The past few days I have finally had the time to sit down and consolidate my build into a single ant script. The company that I am contracting for is a heavy Java shop, so I am using Ant mostly because everyone will have access to ant. I am use to make so it’s been a bit of a learning curve, but Ant is a very cool thing. So, for starters, let me outline my current build process and then I will show you how I have automated it.

  1. We pull the code from the QA branch in GitHub
  2. Run Composer over the code to install all the dependencies
  3. We tag the build with a version tag
  4. We write the version tag and the date to a version file
  5. We tar up ONLY the files that are absolutely needed to run the application (we have tools, docs, ect that we do not need)
  6. We send the final archive to the server team for deployment

Actually, I started using Ant early on for the final tar build. Ant has a great task for tar. Here is the target for that.

<target name="deploy">
      <tar destfile="deploy.tar"
           basedir="build/"   
           excludes="build/**, database/**, docs/**.vagrant/**"/>

</target>

This essentially creates a file called deploy.tar in a sub-folder called build/. It excludes anything already in build/ as well as database/, docs/', and.vagrant’. Obviously I truncated my list…it is a lot longer in reality! A lot of files sit in the root like the Vagrantfile, a Makefile the files for composer.

So this is about the final step. But I still need to automate the rest.

The first step is grabbing my code from GitHub. Eclipse offers a task for any that is based on jgit. You need a few dependencies for this to work. Namely the jgit class file and the jgit-ant class file. You also need the ssh library, which I was already using for some other scripts.

You need to load them as resources, which is done like so:

<taskdef resource="org/eclipse/jgit/ant/ant-tasks.properties">
     <classpath>
       <pathelement location="resources/org.eclipse.jgit.ant-3.0.0.2013061825-r.jar"/>
       <pathelement location="resources/org.eclipse.jgit-3.0.0.2013061825-r.jar"/>
       <pathelement location="../jsch-0.1.49.jar"/>
     </classpath>
</taskdef>

Then we set up the task for cloning:

<target name="clone">
     <git-clone 
         uri="git@github.com:weatheredwatcher/weatheredwatcher.git" 
         branch="testing"
         dest="build/" />
</target>

(Note that I am using this site rather then the project that I am working on…)

The next step is running Composer. For those that do not know, Composer is a dependency management tool for PHP. We are loading several dependencies as well as some custom libs via composer so it is important that we generate the right files.

<target name="init" description="Installing Denpendencies">
  <delete file="build/composer.lock" />

    <exec executable="php" failonerror="true"
        dir="build/">
        <arg value="composer.phar" />
        <arg value="install" />
  </exec>

</target>

So what we are doing here is first we delete the lock file. Typically the devs install a few extra tools that are not needed on the QA server. So we remove the lock file and then only install the production level requirements with Composer. failonerror ensure that we get an error if anything bad happens rather then a success.

As far as tagging goes, I feel it is better to do the tagging in GitHub rather then in the build process. So we will only be writing to a version file. We need to write the current tag as well as the build date to this file. The git command for displaying the current tag is git describe --exact-match --abbrev=0. We antify this like so:

<exec executable="git" failonerror="true"
   dir="build/">
   <arg value="describe" />
   <arg value="--exact-match" />
   <arg value="--abbrev=0" />
   <redirector output="build/version" />
</exec>

The last part was hard. You cannot pass a redirect > through the exec task. Instead, we use the redirector. The date is similar, but we add an append option to the redirector to make sure we do not overwrite the file.

<exec executable="date" dir="build/">
  <redirector output="build/version" append="true"/>
</exec>  

So if we put it all together into a massive ant script, we have nearly the entire deploy build. The last part is the one where we send it along to the Server Team’s folders via a mount and a copy.

The final thing to do is to clean up.

<target name="clean">
    <delete dir="build"/>
</target>

My next step will be taking this process and integrating it into a Hudson build for Continuous Integration or CI. Obviously, Hudson will be able to take on a lot of this functionality without any ant scripts…but I also can just have Hudson run the ant script if I want. We will see. Until then, happy coding!!

Using Zend Style Config Files Everywhere

It’s been a while since my last blog entry, I know!! I feel bad, so here is a bit of php goodness to make us all feel better!

It’s always a good security practice to remove configuration from your web-app. One way to do this is to use a configuration file. Now, Zend has a very cool way of doing this,(Zend\Config\Reader) but the client that I am with right now is using a solution based on Codeigniter. However, since they are planning on eventually moving to Zend anyway, I figured I would implement a solution based on the Zend config solution.

First comes the file, which I called environment.ini and placed in /etc.

The entries are in the following format:

cg.database.name=name
cg.database.username=username
cg.database.password=password
cg.database.hostname=hostname
cg.services.name=dev
cg.services.port=8080

To utilize this for the database, for example, lets create a helper with the following function:

function get_environment(){
$config = array();
foreach( file( '/etc/sitename/environment.ini') as $line) {
    list( $keys, $value) = explode( '=', $line);

    $temp =& $config;
    foreach( explode( '.', $keys) as $key)
    {           
        $temp =& $temp[$key];
    }
    $temp = trim( $value);

}

return $config;

}

This will return an array like this:

array 
  'cg' => 
    array 
     'database' => 
        array 
          'name' => string 'name'
          'hostname' => string 'hostname'
          'username' => string 'username'
          'password' => string 'password' 
      'services' => 
        array (size=5)
          'url' => string 'dev'
          'port' => string '8080'

The next part of this is making use of the data in your application. In the database config file for Codeigniter, for example:

$dbconfig = get_environment();

$db = $dbconfig['cg'][database];

$db['default']['hostname'] = $db['hostname'];
$db['default']['username'] = $db['username'];
$db['default']['password'] = $db['password'];
$db['default']['database'] = $db['name'];

And there you have it! Of course, I based this on CI but you should be able to use this code for any framework..or even no framework.

Preparing a Dev Environment With Puppet

For starts, I now have markup installed in my blog, so no more typing html!! Yea!\n Today we are going to talk about Puppet. No, not Pinochio, or those Punch and Judy dolls. This is Puppet as in the server provisioning tool.

At work I am setting up a development environment for our dev team. Since most of them are just learning php, and for over all consistency I am using Vagrant to build a standard dev vm for everyone to work off of.

The general requirements are simple:

  1. We must run Zend Server
  2. We must load the php drivers MS Sql
  3. We must install subversion

With these requirements in mind, I set out to build my first puppet script.

The first class that we define is our services class. I need to make sure that Apache is running. Also, I found out that Cent Os turns iptables on by default. That interferes with the dev box, as well as being unnecessary! So we make sure that iptables is off.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class services {
  #we want apache
  service {
    'httpd':
      ensure => running,
      enable => true
  }

service {
  'iptables':
    ensure => stopped,
    enable => false
 }
}

The next two classes work in tandem. The repos class defines our Zend Server repo and packages install the required packages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
class packages {
  package {
    "httpd":                      ensure => "present"; # Apache
    "subversion":                 ensure => "present"; # Subversion
    "zend-server-ce-php-5.3":     ensure => "present"; # Zend Server (CE)
    "php-5.3-mssql-zend-server":  ensure => "present"; # MSSQL Extenstion - provided by Zend
  }
}


class repos {
  #lets install some repos
  file { "/etc/yum.repos.d/zend.repo":
    content => "[Zend]
    name=Zend Server
    baseurl=http://repos.zend.com/zend-server/rpm/x86_64
    enabled=1
    gpgcheck=1
    gpgkey=http://repos.zend.com/zend.key

    [Zend_noarch]
    name=Zend Server - noarch
    baseurl=http://repos.zend.com/zend-server/rpm/noarch
    enabled=1
    gpgcheck=1
    gpgkey=http://repos.zend.com/zend.key
    "
  }

}

If anyone wants to see the entire file, here it is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
stage {

  'users':      before => Stage['repos'];
  'repos':      before => Stage['packages'];
  'packages':   before => Stage['configure'];
  'configure':  before => Stage['services'];
  'services':   before => Stage['main'];

}

class services {
  #we want apache
  service {
    'httpd':
      ensure => running,
      enable => true
  }

  service {
    'iptables':
      ensure => stopped,
      enable => false
  }
}

class configure {

  # symlinking the code from /home/vagrant/public to var/www/public
  exec { "public simlink":
    command => "/bin/ln -s /home/vagrant/public /var/www/",
    unless  => "/usr/bin/test -L /var/www/",
  }
  file {"/var/www/index.html":
    ensure => "absent"

  }
}

class packages {
  package {
    "httpd":                      ensure => "present"; # Apache
    "subversion":                 ensure => "present"; # Subversion
    "zend-server-ce-php-5.3":     ensure => "present"; # Zend Server (CE)
    "php-5.3-mssql-zend-server":  ensure => "present"; # MSSQL Extenstion - provided by Zend
  }
}

class repos {

  file { "/etc/yum.repos.d/zend.repo":
    content => "[Zend]
name=Zend Server
baseurl=http://repos.zend.com/zend-server/rpm/x86_64
enabled=1
gpgcheck=1
gpgkey=http://repos.zend.com/zend.key

[Zend_noarch]
name=Zend Server - noarch
baseurl=http://repos.zend.com/zend-server/rpm/noarch
enabled=1
gpgcheck=1
gpgkey=http://repos.zend.com/zend.key
    "
  }

}

class users
{
  group { "puppet":
    ensure => "present",
  }
  user { "vagrant":
    ensure => "present",

  }
}

class {
  users:      stage => "users";
  repos:      stage => "repos";
  packages:   stage => "packages";
  configure:  stage => "configure";
  services:   stage => "services";

  }