hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20130409] NetBSD is part of Google's Summer of Code 2013
News is out that NetBSD is part of Google's Summer of Code 2013 (GSoC) again. GSoC is about students doing work for Open Source projects over the summer, and getting paid while doing so. By Google. For projects proposed by both students and the Open Source projects.

Click on the above link for more information on GSoC in general, there is also a list of proposed projects for this year in NetBSD.

Next steps are:

  • April 9 - 21: Would-be student participants discuss application ideas with mentoring organizations.
  • April 22, 19:00 UTC: Student application period opens.
  • May 3, 19:00 UTC: Student application deadline.
  • Interim Period: Mentoring organizations review and rank student proposals; where necessary, mentoring organizations may request further proposal detail from the student applicant.
  • May 6: Mentoring organizations should have requested slots via their profile in Melange by this point.
  • May 8: Slot allocations published to mentoring organizations From there, students work on their projects with the help of their mentors. There's a "midterm" report due with a first part of the money paid, the rest is paid if the project is finished successfully.

    During the project, students are encouraged to publish news about their process to the world in blogs and other ways found appropriate by their mentoring organizations.

    Past NetBSD projects can be found on SourceForce.

    Interested? Act now!

    [Tags: ]


    • [20130324] Ansible & EC2 - Playbooks for orchestrating NetBSD into the cloud
    As follower of my blog you have seen the steps towards getting NetBSD instances started in Amazon's EC2 cloud with a simple web application deployed on one EC2 instance and the database on another one.

    These blog articles were very detailed on purpose, to have full logfiles available just in case needed. I have used these logs to prepare my pkgsrcCon 2013 talk about Ansible and Amazon's EC2, so things can be looked at without actually running anything. As it turns out this was good, because the 32bit NetBSD instances that I've used during my pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation was a bit more on the theoretical side than I originally planned.

    Now after pkgsrcCon is over, I would like to publish the presentation slides with all the details, and especially the playbooks and all other files to look at - enjoy!

    [Tags: , , , , , ]


    [20130321] Ansible, EC2 and NetBSD final milestone 4 reached: Web and DB on separate VMs in the cloud
    In the fourth and last step on my journey to use Ansible to bring a non-trivial system of a Web server and a DB server into Amazon's EC2 cloud, this is the final step. After starting out with a local VMware VM and making first steps with Ansible and EC2, the previous step was to push a single system into the cloud. Now, the final step is to setup two distinct VMs, one for the database and one for the webserver, and then make them known to each other.

    The single steps are:

    1. Prepare the two VMs
    2. Basic setup for all systems
    3. Install the database server
    4. Install the webserver
    5. Connect database and webserver
    Again, here are all the steps in detail:
    1. As before, ensure local time is correct when talking to Amazon, and also make sure the SSH agent has the proper key loaded. 
      % date
Thu Mar 21 00:45:37 CET 2013
% ssh-add -l
2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
    2. Make sure security groups are setup properly. We use one group for the database server, and one for the webserver. This defines the access permissions from the internet, and also allows to identify systems for their individual configuration and also for connecting them in the final step: 
      % euca-describe-groups
...
GROUP   sg-ae54b3c5     749335780469    ec2-dbservers   Database servers
PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     3306    3306    FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-dbservers   ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
    3. Now, run our playbook to setup the two VMs. This uses the single playbook from the previous milestone, and just runs it twice with different security groups: 
      % ansible-playbook -i hosts-HF config-ec2-prepare-db+web-vm.yml

PLAY [localhost] ********************* 

TASK: [ec2-webservers | Launch new EC2 instance] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Give the system 30 seconds to boot up] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})

TASK: [ec2-webservers | Install pkgin via /usr/bootstrap.sh] ********************* 
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})

TASK: [ec2-webservers | Copy over Ansible binary package] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Install Ansible dependencies] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Install Ansible package (manually)] ********************* 
changed: [127.0.0.1]

TASK: [ec2-webservers | Setup lame /usr/bin/python symlink] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Launch new EC2 instance] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Give the system 30 seconds to boot up] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})

TASK: [ec2-dbservers | Install pkgin via /usr/bootstrap.sh] ********************* 
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})

TASK: [ec2-dbservers | Copy over Ansible binary package] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Install Ansible dependencies] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Install Ansible package (manually)] ********************* 
changed: [127.0.0.1]

TASK: [ec2-dbservers | Setup lame /usr/bin/python symlink] ********************* 
changed: [127.0.0.1]

PLAY RECAP ********************* 
127.0.0.1                      : ok=18   changed=18   unreachable=0    failed=0
    4. Just to make sure, check that the two instances run properly, and are in the right security groups, ec2-webservers and ec2-dbservers: 
      % euca-describe-instances
RESERVATION     r-a419f9d9      749335780469    ec2-webservers
INSTANCE        i-21b7c441      ami-5d0f8034    ...
RESERVATION     r-641efe19      749335780469    ec2-dbservers
INSTANCE        i-54a2ab3e      ami-5d0f8034    ...
    5. Next, bring the two freshly setup systems (which are already capable of acting as ansible targets) up to our basic system setup: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml

PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 

TASK: [ping] ********************* 
ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
ok: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Install tcsh] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Add user feyrer] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Create ~feyrer/.ssh directory] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable ssh login with ssh-key] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Install sudo] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Disable ssh logins as root] ********************* 
ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
ok: [ec2-54-234-139-151.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-54-234-139-151.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0
    6. Check: 
      % ssh ec2-54-234-139-151.compute-1.amazonaws.com id
uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
% 
% ssh ec2-54-235-44-118.compute-1.amazonaws.com id
uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
    7. Now that the two machines run with our basline configuration, install their individual software and settings. First the database server: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml

PLAY [security_group_ec2-dbservers] ********************* 

TASK: [Install mysql] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Install MySQL rc.d script] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Start MySQL service] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Setup DB] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Add db-user] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Copy over DB template] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Import DB data] ********************* 
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0    

      

    8.  Check and see if the database works as expected:

      % ssh -t ec2-54-235-44-118.compute-1.amazonaws.com mysql -u webapp -p webapp
Enter password: ****
...
mysql> show tables;
+------------------+
| Tables_in_webapp |
+------------------+
| names            |
+------------------+
1 row in set (0.01 sec)

mysql> select * from names;
+----+--------+------+
| id | first  | last |
+----+--------+------+
|  1 | Donald | Duck |
|  2 | Daisy  | Duck |
+----+--------+------+
2 rows in set (0.00 sec)

mysql> bye

      


    9.  Excellent. Now setup the webserver, too:

        
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml

PLAY [security_group_ec2-webservers] ********************* 

TASK: [Installing ap24-php53 package and dependencies] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Install Apache rc.d script] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable and start Apache service] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable PHP in Apache config file] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})

TASK: [Make Apache read index.php] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Install phpmyadmin] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable phpmyadmin in Apache config] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Fix Apache access control for phpmyadmin] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Enable PHP modules in PHP config file] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})

TASK: [Create directory for webapp] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Deploy example webapp] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

TASK: [Create webapp symlink for easy access] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

NOTIFIED: [restart apache] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-54-234-139-151.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0    

      


    10.  Again, test:

      % links -dump ec2-54-234-139-151.compute-1.amazonaws.com/
                                   It works!
%
% links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/phptest.php | head
   PHP Logo                                                                   
                                                                              
                               PHP Version 5.3.17                             

   System          NetBSD ip-10-80-61-33.ec2.internal 6.0.1 NetBSD 6.0.1      
                   (XEN3PAE_DOMU) i386                                        
   Build Date      Dec 14 2012 10:31:13                                       
                   './configure' '--with-config-file-path=/usr/pkg/etc'       
                   '--with-config-file-scan-dir=/usr/pkg/etc/php.d'           
                   '--sysconfdir=/usr/pkg/etc' '--localstatedir=/var'         
% 
% links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
   Showing table hf.names:

   Cannot connect to database: Can't connect to local MySQL server through
   socket '/tmp/mysql.sock' (2)(2002)

      


    11.  Close to optimum, but the last error is actually expectet: In
  order for proper operation, the Database needs to grant the
  webserver access, and the web server needs to know where the
  database server is. So let's connect them!
  
      

  This step is done by preparing a shell script on both systems, which
  will then be ran to  - depending on the system's security group - perform the
  proper steps:

      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-connections.yml

PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 

TASK: [Collect EC2 host information] ********************* 
ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
ok: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Prepare connection-script in /tmp/do-connect-vms.sh] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

TASK: [Run connection-script] ********************* 
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
changed: [ec2-54-235-44-118.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-54-234-139-151.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    
ec2-54-235-44-118.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    

      


    12.  With that final step, our test web application works, and the
  webserver can access the database properly:

      % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
   Showing table hf.names:

   +--------------------+
   | id | first  | last |
   |----+--------+------|
   | 1  | Donald | Duck |
   |----+--------+------|
   | 2  | Daisy  | Duck |
   +--------------------+

     ----------------------------------------------------------------------

   Enter new values:

   first:     _____________________ 
   last:      _____________________ 
   [ Submit ]
    So much for this exercise. I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like, or stay tuned to find my presentation and all the data after pkgsrcCon 2013.

    [Tags: , , , ]


    [20130321] Happy 20th Birthday, NetBSD! (Update)

    20 years back from today, NetBSD was initially checked into CVS. Revision 1.1 of src/Makefile was committed on March 21st 1993 on 09:45:37 by Chris Demetriou (cgd@): 

    % cvs log -Nr1.1 Makefile
...
revision 1.1
date: 1993/03/21 09:45:37;  author: cgd;  state: Exp;
branches:  1.1.1;
Initial revision
    NetBSD was started as successor to the Berkeley System Distribution (BSD) Unix with a focus on multiplatform support.

    Personally, I've followed NetBSD since the day in 1993 when the Amiga port popped up, which was the first platform that the newly forked operating system was ported to after its separation from BSD.

    Many things have happened in the past 20 years, and a lot could be shown and told for the history books at this point. But I guess that can be done later - I'd be happy to help out with such a project if someone wants to start it, though :)

    For today I'm very happy that NetBSD is available on a wide range on platforms, runs the software that I want and gives me the assurrance it will be around tomorrow and hopefully for the next 20 years, too.

    Cheers, NetBSD!

    Update: Jeremy Reed pointed me at his BSDnewletter posting, which gives a number of details of NetBSD's history. Recommended reading!

    [Tags: , ]


    [20130321] Ansible, EC2 and NetBSD milestone 3 reached: Web and database in the cloud
    With the previous work on setting up a local VM as database and web server and setting up a Xen VM in Amazon's EC2 cloud combined, it is pretty straight forward to setup a EC2 instance that has all the software to serve a simple web application from the cloud.

    The single steps are:

    1. Prepare the environment with proper time, SSH agent and EC2 firewall groups
    2. Setup EC2 instance with pkgin and ansible
    3. Do basic preparations to meet our standards for logins, shells and general usability and security
    4. Setup database server with DB software, user and import of data
    5. Setup web server with all the software and some demo application
    The following details show all the commands can and their output in more detail:

    1. Make sure time is set properly - needed when talking to Amazon EC2: 
      % sudo sh /etc/rc.d/ntpd stop
ntpd not running? (check /var/run/ntpd.pid).
% sudo sh /etc/rc.d/ntpdate restart
Setting date via ntp.
% sudo sh /etc/rc.d/ntpd start
Starting ntpd.
% date
Sat Mar 16 16:46:19 CET 2013
    2. Teach our EC2 SSH key to SSH agent, so we don't have to type a password (which we don't know anyways - EC2 only works with SSH keys): 
      % ssh-add -l
Could not open a connection to your authentication agent.
% 
% eval `ssh-agent`
Agent pid 10467
% ssh-add -l
The agent has no identities.
% ssh-add ../../euca2ools/key-eucaHF.pem
Identity added: ../../euca2ools/key-eucaHF.pem (../../euca2ools/key-eucaHF.pem)
% ssh-add -l
2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
    3. Check security (firewall) groups - those are stored in EC2, and we have previously set them up: 
      % euca-describe-groups
...
GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
    4. See if there are any EC2 instances running: 
      % euca-describe-instances
%
      No - that's fine, we are about to change that!

    5. Run first playbook to launch EC2 instance and prepare it for using with ansible: 
      % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml

PLAY [localhost] ********************* 

TASK: [Launch new EC2 instance] ********************* 
changed: [127.0.0.1]

TASK: [Give the system 30 seconds to boot up] ********************* 
changed: [127.0.0.1]

TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
changed: [127.0.0.1]

TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})

TASK: [Install pkgin via /usr/bootstrap.sh] ********************* 
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})

TASK: [Copy over Ansible binary package] ********************* 
changed: [127.0.0.1]

TASK: [Install Ansible dependencies] ********************* 
changed: [127.0.0.1]

TASK: [Install Ansible package (manually)] ********************* 
changed: [127.0.0.1]

TASK: [Setup lame /usr/bin/python symlink] ********************* 
changed: [127.0.0.1]

PLAY RECAP ********************* 
127.0.0.1                      : ok=9    changed=9    unreachable=0    failed=0
      We now have a EC2 instance running that has Ansible installed: 
      % euca-describe-instances
RESERVATION     r-d77272ad      749335780469    ec2-webservers
INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com ...
    6. With this EC2 instance, we can do some basic preparations for our standards, e.g. a login without requiring root (and while there, actually disable allowing as root), setup sudo and a proper shell: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml

PLAY [security_group_ec2-webservers] ********************* 

TASK: [ping] ********************* 
ok: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install tcsh] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Add user feyrer] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Create ~feyrer/.ssh directory] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable ssh login with ssh-key] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install sudo] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Disable ssh logins as root] ********************* 
ok: [ec2-107-22-69-112.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0
      Let's have a look if things actually work: 
      % ssh 107.22.69.112 id
uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
% ssh ec2-107-22-69-112.compute-1.amazonaws.com id
uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
% ssh ec2-107-22-69-112.compute-1.amazonaws.com sudo id
uid=0(root) gid=0(wheel) groups=0(wheel),2(kmem),3(sys),4(tty),5(operator),20(staff),31(guest)
    7. Next, install database software and import our demo database, just as we did in out local VM: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml

PLAY [security_group_ec2-webservers] ********************* 

TASK: [Install mysql] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install MySQL rc.d script] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Start MySQL service] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Setup DB] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Add db-user] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Copy over DB template] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Import DB data] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0
      Again, let's see if everything works as expected: 
      % ssh ec2-107-22-69-112.compute-1.amazonaws.com
...
ip-10-202-65-196: {1} mysql -u webapp -p webapp
Enter password: ******
...
mysql> show tables;
+------------------+
| Tables_in_webapp |
+------------------+
| names            |
+------------------+
1 row in set (0.00 sec)

mysql> select * from names;
+----+--------+------+
| id | first  | last |
+----+--------+------+
|  1 | Donald | Duck |
|  2 | Daisy  | Duck |
+----+--------+------+
2 rows in set (0.00 sec)

mysql> exit
Bye
ip-10-202-65-196: {2} exit
logout
Connection to ec2-107-22-69-112.compute-1.amazonaws.com closed.
    8. Last, add Apache+PHP and our small demo web-application: 
        
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml

PLAY [security_group_ec2-webservers] ********************* 

TASK: [Installing ap24-php53 package and dependencies] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install Apache rc.d script] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable and start Apache service] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable PHP in Apache config file] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})

TASK: [Make Apache read index.php] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Install phpmyadmin] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable phpmyadmin in Apache config] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Fix Apache access control for phpmyadmin] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Enable PHP modules in PHP config file] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})

TASK: [Create directory for webapp] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Deploy example webapp] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

TASK: [Create webapp symlink for easy access] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

NOTIFIED: [restart apache] ********************* 
changed: [ec2-107-22-69-112.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-107-22-69-112.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0
    9. Test! 
      % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/
                                   It works!

% links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/phptest.php
   PHP Logo                                                                   
                                                                              
                               PHP Version 5.3.17                             

   System          NetBSD ip-10-202-65-196.ec2.internal 6.0.1 NetBSD 6.0.1    
                   (XEN3PAE_DOMU) i386                                        
   Build Date      Dec 14 2012 10:31:13                                       
...

% links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/webapp/
   Showing table hf.names:

   +--------------------+
   | id | first  | last |
   |----+--------+------|
   | 1  | Donald | Duck |
   |----+--------+------|
   | 2  | Daisy  | Duck |
   +--------------------+

     ----------------------------------------------------------------------

   Enter new values:

   first:     _____________________ 
   last:      _____________________ 
   [ Submit ]
    10. At this point, everything is setup and can be enjoyed. If the instance is needed no longer, it can be terminated: 
      % euca-describe-instances
RESERVATION     r-d77272ad      749335780469    ec2-webservers
INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com       ...
% euca-terminate-instances i-9fafc2f2
INSTANCE        i-9fafc2f2
% euca-describe-instances
RESERVATION     r-d77272ad      749335780469    ec2-webservers
INSTANCE        i-9fafc2f2      ami-5d0f8034                    terminated      eucaHF  ...
    What's next on my journey is to put database and webserver into separate VMs. First one of each, and then see if I find the nerve to look into a setup with more redundancy.

    Shameless plug: I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like!

    [Tags: , , , , , , ]


    [20130314] Ansible, EC2 and NetBSD milestone 2 reached: Instance preparation and communication
    On my quest to use Ansible to get a NetBSD virtual machine into Amazon's EC2 cloud, I've previously described how I use ansible to prepare a local machine. Working from a basic NetBSD setup, the system is setup for basic operation, the configured as both a database server and a Web/PHP server to serve a small demo application.

    Now the next step is to replace the VM with an Amazon EC2 instance. I have previously written about how to manage Amazon/EC2 NetBSD instances, and here are the steps that I make to first prepare an EC2 instance with NetBSD and Ansible, and then use a regular Ansible playbook to talk to all my EC2 instances. Note that the connection between the machines setup via euca2ools and ansible is in the security group names. In this case, the security group "ec2-webservers" is assumed to exist.

    1. Make sure SSH agent runs and has the EC2 SSH-key added: 
      % ssh-add -l
Could not open a connection to your authentication agent.
% eval `ssh-agent`
Agent pid 9304
% ssh-add -l
The agent has no identities.
% ssh-add .../key-ec2HF.pem 
Identity added: ../../euca2ools/key-ec2HF.pem (../../euca2ools/key-ec2HF.pem)
% ssh-add -l
2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-ec2HF.pem (RSA)
%
    2. When using a VM to talk to EC2, pay special attention that it has the correct time, else funny things will happen: 
      % date
Sun Mar 10 14:42:33 CET 2013
    3. Setup the ec2-webservers security (firewall) group. This is used both when creating the EC2 instances, and when accessing them. It's the link between EC2 and Ansible's ec2.py script. 
      % euca-add-group -d 'Web servers' ec2-webservers
% euca-authorize -P tcp -p 80-80 -s 0.0.0.0/0 ec2-webservers
% euca-authorize -P tcp -p 22-22 -s 0.0.0.0/0 ec2-webservers
% euca-authorize -P icmp -s 0.0.0.0/0 ec2-webservers
% 
% euca-describe-groups
GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22     FROM     CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80     FROM     CIDR    0.0.0.0/0
PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1     FROM     CIDR    0.0.0.0/0
    4. List out EC2 instances: 
      % euca-describe-instances
%
      None so far.

    5. Let's use our playbook to prepare our first EC2 instance: 
      % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml

PLAY [localhost] ********************* 

TASK: [Launch new EC2 instance] ********************* 
changed: [127.0.0.1]

TASK: [Give the system 30 seconds to boot up] ********************* 
changed: [127.0.0.1]

TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
changed: [127.0.0.1]

TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," </usr/bootstrap.sh.orig >/usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})

TASK: [Install pkgin via /usr/bootstrap.sh] ********************* 
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})

TASK: [Copy over Ansible binary package] ********************* 
changed: [127.0.0.1]

TASK: [Install Ansible dependencies] ********************* 
changed: [127.0.0.1]

TASK: [Install Ansible package (manually)] ********************* 
changed: [127.0.0.1]

TASK: [Setup lame /usr/bin/python symlink] ********************* 
changed: [127.0.0.1]

PLAY RECAP ********************* 
127.0.0.1                      : ok=9    changed=9    unreachable=0    failed=0    


%
    6. There we go. Let's list it: 
      % euca-describe-instances
RESERVATION     r-bb3b6ac1      749335780469    ec2-webservers
INSTANCE        i-2cb9a45f      ami-a754dbce    ec2-54-234-59-5.compute-1.amazonaws.com \
	ip-10-243-150-74.ec2.internal   running ec2HF  0               t1.micro        \
	2013-03-10T13:47:32.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     54.234.59.5     10.243.150.74                   ebs                                                                     
%
    7. That worked - excellent! Let's add a few more, just for kicks: 
      % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & 
% ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & 
% ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & 
% ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & 
% ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & 
% 

<...wait...>

% euca-describe-instances
RESERVATION     r-bb3b6ac1      749335780469    ec2-webservers
INSTANCE        i-2cb9a45f      ami-a754dbce    ec2-54-234-59-5.compute-1.amazonaws.com \
	ip-10-243-150-74.ec2.internal   running ec2HF  0               t1.micro        \
	2013-03-10T13:47:32.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     54.234.59.5     10.243.150.74                   ebs                                                                     
RESERVATION     r-8b3c6df1      749335780469    ec2-webservers
INSTANCE        i-7cb5a80f      ami-a754dbce    ec2-23-20-42-71.compute-1.amazonaws.com \
	ip-10-203-73-195.ec2.internal   running ec2HF  0               t1.micro        \
	2013-03-10T13:50:48.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     23.20.42.71     10.203.73.195                   ebs                                                                     
RESERVATION     r-733f6e09      749335780469    ec2-webservers
INSTANCE        i-42b5a831      ami-a754dbce    ec2-23-20-87-176.compute-1.amazonaws.com        \
	ip-10-116-37-145.ec2.internal   running ec2HF  0               t1.micro        \
	2013-03-10T13:50:54.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     23.20.87.176    10.116.37.145                   ebs                                                                     
RESERVATION     r-713f6e0b      749335780469    ec2-webservers
INSTANCE        i-40b5a833      ami-a754dbce    ec2-54-242-254-237.compute-1.amazonaws.com      \
	ip-10-195-47-153.ec2.internal   running ec2HF  0               t1.micro        \
	2013-03-10T13:50:54.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     54.242.254.237  10.195.47.153                   ebs                                                                     
RESERVATION     r-773f6e0d      749335780469    ec2-webservers
INSTANCE        i-46b5a835      ami-a754dbce    ec2-54-235-232-227.compute-1.amazonaws.com      \
	ip-10-194-7-72.ec2.internal     running ec2HF  0               t1.micro        \
	2013-03-10T13:50:54.000Z        us-east-1a      aki-825ea7eb                    \
	monitoring-disabled     54.235.232.227  10.194.7.72                     ebs                                                                     
RESERVATION     r-b72475cd      749335780469    ec2-webservers
INSTANCE        i-b2adb0c1      ami-a754dbce    ec2-50-16-129-62.compute-1.amazonaws.com        \
	domU-12-31-39-14-C6-CB.compute-1.internal       running ec2HF  0               t1.micro        \
	2013-03-10T13:55:24.000Z        us-east-1d      aki-825ea7eb                    \
	monitoring-disabled     50.16.129.62    10.206.197.53                   ebs                                                                     
%
    8. Let's talk to our EC2 instances now. For that, we use the ec2.py script, which enumerates all instances: 
      % ./ec2.py --list
{
  "i-2cb9a45f": [
    "ec2-54-234-59-5.compute-1.amazonaws.com"
  ], 
  "i-40b5a833": [
    "ec2-54-242-254-237.compute-1.amazonaws.com"
  ], 
  "i-42b5a831": [
    "ec2-23-20-87-176.compute-1.amazonaws.com"
  ], 
  "i-46b5a835": [
    "ec2-54-235-232-227.compute-1.amazonaws.com"
  ], 
  "i-7cb5a80f": [
    "ec2-23-20-42-71.compute-1.amazonaws.com"
  ], 
  "i-b2adb0c1": [
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ], 
  "key_ec2HF": [
    "ec2-54-234-59-5.compute-1.amazonaws.com", 
    "ec2-23-20-42-71.compute-1.amazonaws.com", 
    "ec2-23-20-87-176.compute-1.amazonaws.com", 
    "ec2-54-242-254-237.compute-1.amazonaws.com", 
    "ec2-54-235-232-227.compute-1.amazonaws.com", 
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ], 
  "security_group_ec2-webservers": [
    "ec2-54-234-59-5.compute-1.amazonaws.com", 
    "ec2-23-20-42-71.compute-1.amazonaws.com", 
    "ec2-23-20-87-176.compute-1.amazonaws.com", 
    "ec2-54-242-254-237.compute-1.amazonaws.com", 
    "ec2-54-235-232-227.compute-1.amazonaws.com", 
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ], 
  "type_t1_micro": [
    "ec2-54-234-59-5.compute-1.amazonaws.com", 
    "ec2-23-20-42-71.compute-1.amazonaws.com", 
    "ec2-23-20-87-176.compute-1.amazonaws.com", 
    "ec2-54-242-254-237.compute-1.amazonaws.com", 
    "ec2-54-235-232-227.compute-1.amazonaws.com", 
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ], 
  "us-east-1": [
    "ec2-54-234-59-5.compute-1.amazonaws.com", 
    "ec2-23-20-42-71.compute-1.amazonaws.com", 
    "ec2-23-20-87-176.compute-1.amazonaws.com", 
    "ec2-54-242-254-237.compute-1.amazonaws.com", 
    "ec2-54-235-232-227.compute-1.amazonaws.com", 
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ], 
  "us-east-1a": [
    "ec2-54-234-59-5.compute-1.amazonaws.com", 
    "ec2-23-20-42-71.compute-1.amazonaws.com", 
    "ec2-23-20-87-176.compute-1.amazonaws.com", 
    "ec2-54-242-254-237.compute-1.amazonaws.com", 
    "ec2-54-235-232-227.compute-1.amazonaws.com"
  ], 
  "us-east-1d": [
    "ec2-50-16-129-62.compute-1.amazonaws.com"
  ]
}
    9. ec2.py can also give us information about one instance: 
      % ./ec2.py --host ec2-54-234-59-5.compute-1.amazonaws.com
{
  "ec2__in_monitoring_element": false, 
  "ec2_ami_launch_index": "0", 
  "ec2_architecture": "x86_64", 
  "ec2_client_token": "", 
  "ec2_dns_name": "ec2-54-234-59-5.compute-1.amazonaws.com", 
  "ec2_eventsSet": "", 
  "ec2_group_name": "", 
  "ec2_hypervisor": "xen", 
  "ec2_id": "i-2cb9a45f", 
  "ec2_image_id": "ami-a754dbce", 
  "ec2_instanceState": "", 
  "ec2_instance_type": "t1.micro", 
  "ec2_ip_address": "54.234.59.5", 
  "ec2_item": "", 
  "ec2_kernel": "aki-825ea7eb", 
  "ec2_key_name": "ec2HF", 
  "ec2_launch_time": "2013-03-10T13:47:32.000Z", 
  "ec2_monitored": false, 
  "ec2_monitoring": "", 
  "ec2_networkInterfaceSet": "", 
  "ec2_persistent": false, 
  "ec2_placement": "us-east-1a", 
  "ec2_platform": "", 
  "ec2_previous_state": "", 
  "ec2_private_dns_name": "ip-10-243-150-74.ec2.internal", 
  "ec2_private_ip_address": "10.243.150.74", 
  "ec2_public_dns_name": "ec2-54-234-59-5.compute-1.amazonaws.com", 
  "ec2_ramdisk": "", 
  "ec2_reason": "", 
  "ec2_region": "us-east-1", 
  "ec2_requester_id": "", 
  "ec2_root_device_name": "/dev/sda1", 
  "ec2_root_device_type": "ebs", 
  "ec2_security_group_ids": "sg-a854b3c3", 
  "ec2_security_group_names": "ec2-webservers", 
  "ec2_shutdown_state": "", 
  "ec2_spot_instance_request_id": "", 
  "ec2_state": "running", 
  "ec2_state_code": 16, 
  "ec2_state_reason": "", 
  "ec2_subnet_id": "", 
  "ec2_tenancy": "default", 
  "ec2_virtualization_type": "paravirtual", 
  "ec2_vpc_id": ""
}
    10. Now let's use a regular playbook with the ec2.py script to get a list of all instances in the 'ec2-webservers' group and then use ansible's ping module on all of them: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml

PLAY [security_group_ec2-webservers] ********************* 

GATHERING FACTS ********************* 
ok: [ec2-50-16-129-62.compute-1.amazonaws.com]
ok: [ec2-54-235-232-227.compute-1.amazonaws.com]
ok: [ec2-23-20-42-71.compute-1.amazonaws.com]
ok: [ec2-23-20-87-176.compute-1.amazonaws.com]
ok: [ec2-54-242-254-237.compute-1.amazonaws.com]
ok: [ec2-54-234-59-5.compute-1.amazonaws.com]

TASK: [ping] ********************* 
ok: [ec2-54-235-232-227.compute-1.amazonaws.com]
ok: [ec2-50-16-129-62.compute-1.amazonaws.com]
ok: [ec2-23-20-87-176.compute-1.amazonaws.com]
ok: [ec2-23-20-42-71.compute-1.amazonaws.com]
ok: [ec2-54-234-59-5.compute-1.amazonaws.com]
ok: [ec2-54-242-254-237.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-23-20-42-71.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-23-20-87-176.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-50-16-129-62.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-54-234-59-5.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-54-235-232-227.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-54-242-254-237.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    


%
    11. Finally, clean up and use euca-terminate-instance to delete all instances: 
      % euca-describe-instances | grep INSTANCE | awk '{print $2}' | xargs -n 1 euca-terminate-instances
INSTANCE        i-60829f13
INSTANCE        i-2cb9a45f
INSTANCE        i-7cb5a80f
INSTANCE        i-42b5a831
INSTANCE        i-40b5a833
INSTANCE        i-46b5a835
INSTANCE        i-b2adb0c1
% euca-describe-instances
RESERVATION     r-bb3b6ac1      749335780469    ec2-webservers
INSTANCE        i-2cb9a45f      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:47:32.000Z        us-east-1a      \
	aki-825ea7eb                    monitoring-disabled                                     ebs                                                                     
RESERVATION     r-8b3c6df1      749335780469    ec2-webservers
INSTANCE        i-7cb5a80f      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:50:48.000Z        us-east-1a      \
	aki-825ea7eb                    monitoring-disabled                                     ebs                                                                     
RESERVATION     r-733f6e09      749335780469    ec2-webservers
INSTANCE        i-42b5a831      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:50:54.000Z        us-east-1a      \
	aki-825ea7eb                   monitoring-disabled                                     ebs                                                                     
RESERVATION     r-713f6e0b      749335780469    ec2-webservers
INSTANCE        i-40b5a833      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:50:54.000Z        us-east-1a      \
	aki-825ea7eb                    monitoring-disabled                                     ebs                                                                     
RESERVATION     r-773f6e0d      749335780469    ec2-webservers
INSTANCE        i-46b5a835      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:50:54.000Z        us-east-1a      \
	aki-825ea7eb                    monitoring-disabled                                     ebs                                                                     
RESERVATION     r-b72475cd      749335780469    ec2-webservers
INSTANCE        i-b2adb0c1      ami-a754dbce                    terminated      ec2HF  \
	0               t1.micro        2013-03-10T13:55:24.000Z        us-east-1d      \
	aki-825ea7eb                    monitoring-disabled                                     ebs                                                             
%
    12. The terminated instances will be removed by EC2 eventually, and you can start all over.
    With the above steps and the previous work to use Ansible to setup a NetBSD system with basic configuration as database- and webserver the next step is to put those two things together, and get a (single) NetBSD machine into the Amazon cloud that serves as both database and webserver.

    Let's stay tune for this to happen!

    Shameless plug: I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like!

    References: CapsUnlock blog post, CentOS Wiki.

    [Tags: , , ]


    [20130314] NPF documentation
    Mindaugas Rasiukevicius has worked on NetBSD's new packet filter "npf" for quite some time now. In order to make things easier for new users, he now has also put up documentation for it.

    The documentation covers an introduction with a brief note on NPF's design followed by elements of a NPF configuration file. This is followed by instructions for dynamic rules, stateful filtering and network address translation (NAT). Further paragraphs describe the extension API, troubleshooting and appendixes with more information.

    [Tags: , ]


    [20130310] Talking to the cloud
    After some more hacking, I have a basic understanding of how to start Amazon NetBSD EC2 instances using Ansible, fix the instances so they can be used as targets for further Ansible commands, and then actually talking to my herd of happy instances.

    Here's a teaser:

    1. Start EC2 instances, put them into ec2-webservers group. Repeat the following command for more than one instance: 
      % ansible -i hosts-HF localhost -m ec2 -a 'image=ami-a754dbce instance_type=t1.micro \
key_name=eucaHF group=ec2-webservers'
    2. Prepare instances for Ansible (omitted - needs cleanup & automation)

    3. Use Ansible to ping all servers in the ec2-webservers group: 
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml

PLAY [security_group_ec2-webservers] ********************* 

GATHERING FACTS ********************* 
ok: [ec2-23-23-15-202.compute-1.amazonaws.com]
ok: [ec2-54-235-230-206.compute-1.amazonaws.com]

TASK: [ping] ********************* 
ok: [ec2-23-23-15-202.compute-1.amazonaws.com]
ok: [ec2-54-235-230-206.compute-1.amazonaws.com]

PLAY RECAP ********************* 
ec2-23-23-15-202.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0    
ec2-54-235-230-206.compute-1.amazonaws.com : ok=2    changed=0    unreachable=0    failed=0


    [Tags: , , ]


    [20130309] pkgsrcCon 2013 schedule
    Julian Fagir has posted pkgsrcCon 2013's schedule to the pkgsrc-users list. The event is on March 23rd in Berlin Moabit. Here are a bunch of reasons to get there:
    • pkgsrc release engineering
    • pkgsrc on SmartOS
    • Mancoosi tools for the analysis and quality assurance of FOSS distributions
    • Go On NetBSD
    • Rehabilitating pkglint
    • DeforaOS and pkgsrc (presentation with workshop)
    Register now!

    [Tags: , ]


    [20130228] Ansible and NetBSD milestone 1 reached: playbooks for system config, web+db servers
    In my quest to play with Ansible, I've reached my first milestone: I now have playbooks that take a basic NetBSD installation, configure it into a usable base installation, and then add a MySQL database, Apache and PHP to use it as webserver, and then deploy a simple web application.

    The playbooks are too emberassing to publish, but here are the steps to get things going:

    1. Setup NetBSD 6.0 with "base" and "etc" set, also add "pkgin" from menu
    2. Allow root logins via ssh (for a start, will be changed later)
    3. Install ansible-1.0nb1 binary package with all its depends
    4. From a management station, run: ansible-playbook -k -i hosts-HF config-netbsd-basic.yml
    5. Then, run: ansible-playbook -i hosts-HF config-netbsd-dbserver.yml
    6. Last, run: ansible-playbook -i hosts-HF config-netbsd-webserver.yml
    After that, a simple "phptest()" page, phpmyadmin and my simple PHP-based web application can be run. Administration of the system is via SSH and sudo, root logins were disabled in the first ansible playbook.

    Now to tweak the ansible playbooks to look less ugly, use variables, and then separate database and webserver into two separate machines - all in preparation to move them into the Amazon EC2 cloud. Stay tuned!

    For the record, here's a log of the three ansible playbooks above, starting from my basic NetBSD installation that already has pkgin and ansible: 

    % ansible-playbook -k -i hosts-HF config-netbsd-basic.yml
SSH password: 

PLAY [netbsd] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Install tcsh] ********************* 
changed: [10.0.0.181]

TASK: [Add user feyrer] ********************* 
changed: [10.0.0.181]

TASK: [Create ~feyrer/.ssh directory] ********************* 
changed: [10.0.0.181]

TASK: [Enable ssh login with ssh-key] ********************* 
changed: [10.0.0.181]

TASK: [Install sudo] ********************* 
changed: [10.0.0.181]

TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
changed: [10.0.0.181]

TASK: [Disable ssh logins as root] ********************* 
changed: [10.0.0.181]

NOTIFIED: [restart sshd] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=9    changed=8    unreachable=0    failed=0    


% ansible-playbook    -i hosts-HF config-netbsd-dbserver.yml

PLAY [dbservers] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Install mysql] ********************* 
changed: [10.0.0.181]

TASK: [Install MySQL rc.d script] ********************* 
changed: [10.0.0.181]

TASK: [Start MySQL service] ********************* 
changed: [10.0.0.181]

TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
changed: [10.0.0.181]

TASK: [Setup DB] ********************* 
changed: [10.0.0.181]

TASK: [Add db-user] ********************* 
changed: [10.0.0.181]

TASK: [Copy over DB template] ********************* 
changed: [10.0.0.181]

TASK: [Import DB data] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=9    changed=8    unreachable=0    failed=0    


%% ansible-playbook    -i hosts-HF config-netbsd-webserver.yml

PLAY [webservers] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Installing ap24-php53 package and dependencies] ********************* 
changed: [10.0.0.181]

TASK: [Install Apache rc.d script] ********************* 
changed: [10.0.0.181]

TASK: [Enable and start Apache service] ********************* 
changed: [10.0.0.181]

TASK: [Enable PHP in Apache config file] ********************* 
changed: [10.0.0.181] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
changed: [10.0.0.181] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})

TASK: [Make Apache read index.php] ********************* 
changed: [10.0.0.181]

TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
changed: [10.0.0.181]

TASK: [Install phpmyadmin] ********************* 
changed: [10.0.0.181]

TASK: [Enable phpmyadmin in Apache config] ********************* 
changed: [10.0.0.181]

TASK: [Enable PHP modules in PHP config file] ********************* 
changed: [10.0.0.181] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})

TASK: [Fix Apache access control] ********************* 
changed: [10.0.0.181]

TASK: [Create directory for webapp] ********************* 
changed: [10.0.0.181]

TASK: [Deploy example webapp] ********************* 
changed: [10.0.0.181]

TASK: [Create webapp symlink for easy access] ********************* 
changed: [10.0.0.181]

NOTIFIED: [restart apache] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=15   changed=14   unreachable=0    failed=0    


% links -dump http://10.0.0.181/webapp/
   Showing table hf.names:

   +--------------------+
   | id | first  | last |
   |----+--------+------|
   | 1  | Donald | Duck |
   |----+--------+------|
   | 2  | Daisy  | Duck |
   +--------------------+

     ----------------------------------------------------------------------

   Enter new values:

   first:     _____________________ 
   last:      _____________________ 
   [ Submit ] 

%


    [Tags:     , , , , , ]


    More recent 10 entriesPrevious 10 entries
    Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.
    Access count: 35200416
    Copyright (c) Hubert Feyrer