hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20161105] NetBSD 7.0/xen scheduling mystery, and how to fix it with processor sets
Today I had a need to do some number crunching using a home-brewn C program. In order to do some manual load balancing, I was firing up some Amazon AWS instances (which is Xen) with NetBSD 7.0. In this case, the system was assigned two CPUs, from dmesg:
    # dmesg | grep cpu
    vcpu0 at hypervisor0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, id 0x306e4
    vcpu1 at hypervisor0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, id 0x306e4
I started two instances of my program, with the intent to have each one use one CPU. Which is not what happened! Here is what I observed, and how I fixed things for now.

I was looking at top(1) to see that everything was running fine, and noticed funny WCPU and CPU values:

      PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
      2791 root      25    0  8816K  964K RUN/0     16:10 54.20% 54.20% myprog
      2845 root      26    0  8816K  964K RUN/0     17:10 47.90% 47.90% myprog
I expected something like WCPU and CPU being around 100%, assuming that each process was bound to its own CPU. The values I actually saw (and listed above) suggested that both programs were fighting for the same CPU. Huh?!

top's CPU state shows:

    load averages:  2.15,  2.07,  1.82;               up 0+00:45:19        18:00:55
    27 processes: 2 runnable, 23 sleeping, 2 on CPU
    CPU states: 50.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 50.0% idle
    Memory: 119M Act, 7940K Exec, 101M File, 3546M Free
Which is not too useful. Typing "1" in top(1) lists the actual per-CPU usage instead:
    load averages:  2.14,  2.08,  1.83;               up 0+00:45:56        18:01:32
    27 processes: 4 runnable, 21 sleeping, 2 on CPU
    CPU0 states:  100% user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle
    CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    Memory: 119M Act, 7940K Exec, 101M File, 3546M Free
This confirmed my suspicion that both processes were bound to one CPU, and that the other one was idling. Bad! But how to fix?

One option is to kick your operating system out of the window, but I still like NetBSD, so here's another solution: NetBSD allows to create "processor sets", assign CPU(s) to them and then assign processes to the processor sets. Let's have a look!

Processor sets are manipulated using the psrset(8) utility. By default all CPUs are in the same (system) processor set:

    # psrset
    system processor set 0: processor(s) 0 1
First step is to create a new processor set:
    # psrset -c
    1
    # psrset
    system processor set 0: processor(s) 0 1
    user processor set 1: empty
Next, assign one CPU to the new set:
    # psrset -a 1 1
    # psrset
    system processor set 0: processor(s) 0
    user processor set 1: processor(s) 1
Last, find out what the process IDs of my two (running) processes are, and assign them to the two processor sets:
    # ps -u 
    USER  PID %CPU %MEM   VSZ  RSS TTY     STAT STARTED     TIME COMMAND
    root 2791 52.0  0.0  8816  964 pts/4   R+    5:28PM 22:57.80 myprog
    root 2845 50.0  0.0  8816  964 pts/2   R+    5:26PM 23:33.97 myprog
    #
    # psrset -b 0 2791
    # psrset -b 1 2845
Note that this was done with the two processes running, there is no need to stop and restart them! The effect of the commands is imediate, as can be seen in top(1):
    load averages:  2.02,  2.05,  1.94;               up 0+00:59:32        18:15:08
    27 processes: 1 runnable, 24 sleeping, 2 on CPU
    CPU0 states:  100% user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle
    CPU1 states:  100% user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle
    Memory: 119M Act, 7940K Exec, 101M File, 3546M Free
    Swap:

      PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
     2845 root      25    0  8816K  964K CPU/1     26:14   100%   100% myprog
     2791 root      25    0  8816K  964K RUN/0     25:40   100%   100% myprog
Things are as expected now, with each program being bound to its own CPU.

Now why this didn't happen by default is left as an exercise to the reader. Hints that may help:

    # uname -a
    NetBSD foo.eu-west-1.compute.internal 7.0 NetBSD 7.0 (XEN3_DOMU.201509250726Z) amd64
    # dmesg
    ...
    hypervisor0 at mainbus0: Xen version 4.2.amazon
    VIRQ_DEBUG interrupt using event channel 3
    vcpu0 at hypervisor0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, id 0x306e4
    vcpu1 at hypervisor0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, id 0x306e4 
AWS Instance type: c3.large
AMI ID: NetBSD-x86_64-7.0-201511211930Z-20151121-1142 (ami-ac983ddf)

[Tags: , , , , , ]


[20150225] NetBSD on IBM's SoftLayer and Microsoft's Azure and more
Xen is a common and popular virtualization platform today. NetBSD was ported to it prety early. Due to that, NetBSD cam be ran on any "cloud" infrastructure that uses Xen, with Amazon's EC2 cloud probably the most prominent and largest one.

As the whole "cloud" and "platform as a service" (PaaS) is lifting off, other virtualization platforms get popular, which need attention from NetBSD. The first one to note is IBM's SoftLayer, which uses its own virtualization technique that supports many operating systems but strange enough not NetBSD. To still get things going, Emile "iMil" Heitor has investigated the situation, and wrote how to install NetBSD (or any PV-capable system) on IBM's SoftLayer.

Another major platform to look for is Microsoft's Azure. It uses Microsoft's ohn Hyper-V technique, which there is no NetBSD support yet! There is a project description "NetBSD/azure -- Bringing NetBSD to Microsoft Azure" available that got some pretty useful links this week - any takers? As starting point, there's code for Running FreeBSD in Azure.

Last but not least, the last big virtualization platfor amiss is KVM, which is used e.g. in Google's cloud platform. While there are some mentions that NetBSD runs as guest operating system, I am not sure what the latest state is. Anyone in for a comparison? :)

[Tags: , , , , , , , , ]



[20130324] Ansible & EC2 - Playbooks for orchestrating NetBSD into the cloud
As follower of my blog you have seen the steps towards getting NetBSD instances started in Amazon's EC2 cloud with a simple web application deployed on one EC2 instance and the database on another one.

These blog articles were very detailed on purpose, to have full logfiles available just in case needed. I have used these logs to prepare my pkgsrcCon 2013 talk about Ansible and Amazon's EC2, so things can be looked at without actually running anything. As it turns out this was good, because the 32bit NetBSD instances that I've used during my pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation was a bit more on the theoretical side than I originally planned.

Now after pkgsrcCon is over, I would like to publish the presentation slides with all the details, and especially the playbooks and all other files to look at - enjoy!

[Tags: , , , , , ]



[20130321] Ansible, EC2 and NetBSD final milestone 4 reached: Web and DB on separate VMs in the cloud
In the fourth and last step on my journey to use Ansible to bring a non-trivial system of a Web server and a DB server into Amazon's EC2 cloud, this is the final step. After starting out with a local VMware VM and making first steps with Ansible and EC2, the previous step was to push a single system into the cloud. Now, the final step is to setup two distinct VMs, one for the database and one for the webserver, and then make them known to each other.

The single steps are:

  1. Prepare the two VMs
  2. Basic setup for all systems
  3. Install the database server
  4. Install the webserver
  5. Connect database and webserver
Again, here are all the steps in detail:
  1. As before, ensure local time is correct when talking to Amazon, and also make sure the SSH agent has the proper key loaded.
    % date
    Thu Mar 21 00:45:37 CET 2013
    % ssh-add -l
    2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
    
  2. Make sure security groups are setup properly. We use one group for the database server, and one for the webserver. This defines the access permissions from the internet, and also allows to identify systems for their individual configuration and also for connecting them in the final step:
    % euca-describe-groups
    ...
    GROUP   sg-ae54b3c5     749335780469    ec2-dbservers   Database servers
    PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     3306    3306    FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-dbservers   ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
    GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
    PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
    
  3. Now, run our playbook to setup the two VMs. This uses the single playbook from the previous milestone, and just runs it twice with different security groups:
    % ansible-playbook -i hosts-HF config-ec2-prepare-db+web-vm.yml
    
    PLAY [localhost] ********************* 
    
    TASK: [ec2-webservers | Launch new EC2 instance] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Give the system 30 seconds to boot up] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
    changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
    
    TASK: [ec2-webservers | Install pkgin via /usr/bootstrap.sh] ********************* 
    changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
    
    TASK: [ec2-webservers | Copy over Ansible binary package] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Install Ansible dependencies] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Install Ansible package (manually)] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-webservers | Setup lame /usr/bin/python symlink] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Launch new EC2 instance] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Give the system 30 seconds to boot up] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
    changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
    
    TASK: [ec2-dbservers | Install pkgin via /usr/bootstrap.sh] ********************* 
    changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
    
    TASK: [ec2-dbservers | Copy over Ansible binary package] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Install Ansible dependencies] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Install Ansible package (manually)] ********************* 
    changed: [127.0.0.1]
    
    TASK: [ec2-dbservers | Setup lame /usr/bin/python symlink] ********************* 
    changed: [127.0.0.1]
    
    PLAY RECAP ********************* 
    127.0.0.1                      : ok=18   changed=18   unreachable=0    failed=0    
    
  4. Just to make sure, check that the two instances run properly, and are in the right security groups, ec2-webservers and ec2-dbservers:
    % euca-describe-instances
    RESERVATION     r-a419f9d9      749335780469    ec2-webservers
    INSTANCE        i-21b7c441      ami-5d0f8034    ...
    RESERVATION     r-641efe19      749335780469    ec2-dbservers
    INSTANCE        i-54a2ab3e      ami-5d0f8034    ...
    
  5. Next, bring the two freshly setup systems (which are already capable of acting as ansible targets) up to our basic system setup:
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml
    
    PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 
    
    TASK: [ping] ********************* 
    ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
    ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Install tcsh] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Add user feyrer] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Create ~feyrer/.ssh directory] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable ssh login with ssh-key] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Install sudo] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Disable ssh logins as root] ********************* 
    ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
    ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-54-234-139-151.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
    ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
    
  6. Check:
    % ssh ec2-54-234-139-151.compute-1.amazonaws.com id
    uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
    % 
    % ssh ec2-54-235-44-118.compute-1.amazonaws.com id
    uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
    
  7. Now that the two machines run with our basline configuration, install their individual software and settings. First the database server:
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml
    
    PLAY [security_group_ec2-dbservers] ********************* 
    
    TASK: [Install mysql] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Install MySQL rc.d script] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Start MySQL service] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Setup DB] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Add db-user] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Copy over DB template] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Import DB data] ********************* 
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0    
    
    
    
  8. Check and see if the database works as expected:
    % ssh -t ec2-54-235-44-118.compute-1.amazonaws.com mysql -u webapp -p webapp
    Enter password: ****
    ...
    mysql> show tables;
    +------------------+
    | Tables_in_webapp |
    +------------------+
    | names            |
    +------------------+
    1 row in set (0.01 sec)
    
    mysql> select * from names;
    +----+--------+------+
    | id | first  | last |
    +----+--------+------+
    |  1 | Donald | Duck |
    |  2 | Daisy  | Duck |
    +----+--------+------+
    2 rows in set (0.00 sec)
    
    mysql> bye
    
  9. Excellent. Now setup the webserver, too:
      
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml
    
    PLAY [security_group_ec2-webservers] ********************* 
    
    TASK: [Installing ap24-php53 package and dependencies] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Install Apache rc.d script] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable and start Apache service] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable PHP in Apache config file] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})
    
    TASK: [Make Apache read index.php] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Install phpmyadmin] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable phpmyadmin in Apache config] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Fix Apache access control for phpmyadmin] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Enable PHP modules in PHP config file] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})
    
    TASK: [Create directory for webapp] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Deploy example webapp] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    TASK: [Create webapp symlink for easy access] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    NOTIFIED: [restart apache] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-54-234-139-151.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0    
    
  10. Again, test:
    % links -dump ec2-54-234-139-151.compute-1.amazonaws.com/
                                       It works!
    %
    % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/phptest.php | head
       PHP Logo                                                                   
                                                                                  
                                   PHP Version 5.3.17                             
    
       System          NetBSD ip-10-80-61-33.ec2.internal 6.0.1 NetBSD 6.0.1      
                       (XEN3PAE_DOMU) i386                                        
       Build Date      Dec 14 2012 10:31:13                                       
                       './configure' '--with-config-file-path=/usr/pkg/etc'       
                       '--with-config-file-scan-dir=/usr/pkg/etc/php.d'           
                       '--sysconfdir=/usr/pkg/etc' '--localstatedir=/var'         
    % 
    % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
       Showing table hf.names:
    
       Cannot connect to database: Can't connect to local MySQL server through
       socket '/tmp/mysql.sock' (2)(2002)
    
  11. Close to optimum, but the last error is actually expectet: In order for proper operation, the Database needs to grant the webserver access, and the web server needs to know where the database server is. So let's connect them!

    This step is done by preparing a shell script on both systems, which will then be ran to - depending on the system's security group - perform the proper steps:

    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-connections.yml
    
    PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 
    
    TASK: [Collect EC2 host information] ********************* 
    ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
    ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Prepare connection-script in /tmp/do-connect-vms.sh] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    TASK: [Run connection-script] ********************* 
    changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
    changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-54-234-139-151.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    
    ec2-54-235-44-118.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    
    
  12. With that final step, our test web application works, and the webserver can access the database properly:
    % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
       Showing table hf.names:
    
       +--------------------+
       | id | first  | last |
       |----+--------+------|
       | 1  | Donald | Duck |
       |----+--------+------|
       | 2  | Daisy  | Duck |
       +--------------------+
    
         ----------------------------------------------------------------------
    
       Enter new values:
    
       first:     _____________________ 
       last:      _____________________ 
       [ Submit ] 
    
So much for this exercise. I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like, or stay tuned to find my presentation and all the data after pkgsrcCon 2013.

[Tags: , , , ]


[20130321] Ansible, EC2 and NetBSD milestone 3 reached: Web and database in the cloud
With the previous work on setting up a local VM as database and web server and setting up a Xen VM in Amazon's EC2 cloud combined, it is pretty straight forward to setup a EC2 instance that has all the software to serve a simple web application from the cloud.

The single steps are:

  1. Prepare the environment with proper time, SSH agent and EC2 firewall groups
  2. Setup EC2 instance with pkgin and ansible
  3. Do basic preparations to meet our standards for logins, shells and general usability and security
  4. Setup database server with DB software, user and import of data
  5. Setup web server with all the software and some demo application
The following details show all the commands can and their output in more detail:

  1. Make sure time is set properly - needed when talking to Amazon EC2:
    % sudo sh /etc/rc.d/ntpd stop
    ntpd not running? (check /var/run/ntpd.pid).
    % sudo sh /etc/rc.d/ntpdate restart
    Setting date via ntp.
    % sudo sh /etc/rc.d/ntpd start
    Starting ntpd.
    % date
    Sat Mar 16 16:46:19 CET 2013
    
  2. Teach our EC2 SSH key to SSH agent, so we don't have to type a password (which we don't know anyways - EC2 only works with SSH keys):
    % ssh-add -l
    Could not open a connection to your authentication agent.
    % 
    % eval `ssh-agent`
    Agent pid 10467
    % ssh-add -l
    The agent has no identities.
    % ssh-add ../../euca2ools/key-eucaHF.pem
    Identity added: ../../euca2ools/key-eucaHF.pem (../../euca2ools/key-eucaHF.pem)
    % ssh-add -l
    2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
    
  3. Check security (firewall) groups - those are stored in EC2, and we have previously set them up:
    % euca-describe-groups
    ...
    GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
    PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
    PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
    
  4. See if there are any EC2 instances running:
    % euca-describe-instances
    %
    
    No - that's fine, we are about to change that!

  5. Run first playbook to launch EC2 instance and prepare it for using with ansible:
    % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml
    
    PLAY [localhost] ********************* 
    
    TASK: [Launch new EC2 instance] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Give the system 30 seconds to boot up] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
    changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
    changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
    
    TASK: [Install pkgin via /usr/bootstrap.sh] ********************* 
    changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
    
    TASK: [Copy over Ansible binary package] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Install Ansible dependencies] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Install Ansible package (manually)] ********************* 
    changed: [127.0.0.1]
    
    TASK: [Setup lame /usr/bin/python symlink] ********************* 
    changed: [127.0.0.1]
    
    PLAY RECAP ********************* 
    127.0.0.1                      : ok=9    changed=9    unreachable=0    failed=0    
    
    We now have a EC2 instance running that has Ansible installed:
    % euca-describe-instances
    RESERVATION     r-d77272ad      749335780469    ec2-webservers
    INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com ...
    
  6. With this EC2 instance, we can do some basic preparations for our standards, e.g. a login without requiring root (and while there, actually disable allowing as root), setup sudo and a proper shell:
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml
    
    PLAY [security_group_ec2-webservers] ********************* 
    
    TASK: [ping] ********************* 
    ok: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install tcsh] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Add user feyrer] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Create ~feyrer/.ssh directory] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable ssh login with ssh-key] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install sudo] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Disable ssh logins as root] ********************* 
    ok: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
    
    Let's have a look if things actually work:
    % ssh 107.22.69.112 id
    uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
    % ssh ec2-107-22-69-112.compute-1.amazonaws.com id
    uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
    % ssh ec2-107-22-69-112.compute-1.amazonaws.com sudo id
    uid=0(root) gid=0(wheel) groups=0(wheel),2(kmem),3(sys),4(tty),5(operator),20(staff),31(guest)
    
  7. Next, install database software and import our demo database, just as we did in out local VM:
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml
    
    PLAY [security_group_ec2-webservers] ********************* 
    
    TASK: [Install mysql] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install MySQL rc.d script] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Start MySQL service] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Setup DB] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Add db-user] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Copy over DB template] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Import DB data] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0    
    
    Again, let's see if everything works as expected:
    % ssh ec2-107-22-69-112.compute-1.amazonaws.com
    ...
    ip-10-202-65-196: {1} mysql -u webapp -p webapp
    Enter password: ******
    ...
    mysql> show tables;
    +------------------+
    | Tables_in_webapp |
    +------------------+
    | names            |
    +------------------+
    1 row in set (0.00 sec)
    
    mysql> select * from names;
    +----+--------+------+
    | id | first  | last |
    +----+--------+------+
    |  1 | Donald | Duck |
    |  2 | Daisy  | Duck |
    +----+--------+------+
    2 rows in set (0.00 sec)
    
    mysql> exit
    Bye
    ip-10-202-65-196: {2} exit
    logout
    Connection to ec2-107-22-69-112.compute-1.amazonaws.com closed.
    
  8. Last, add Apache+PHP and our small demo web-application:
      
    % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml
    
    PLAY [security_group_ec2-webservers] ********************* 
    
    TASK: [Installing ap24-php53 package and dependencies] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install Apache rc.d script] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable and start Apache service] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable PHP in Apache config file] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})
    
    TASK: [Make Apache read index.php] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Install phpmyadmin] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable phpmyadmin in Apache config] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Fix Apache access control for phpmyadmin] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Enable PHP modules in PHP config file] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})
    
    TASK: [Create directory for webapp] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Deploy example webapp] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    TASK: [Create webapp symlink for easy access] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    NOTIFIED: [restart apache] ********************* 
    changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
    
    PLAY RECAP ********************* 
    ec2-107-22-69-112.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0    
    
  9. Test!
    % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/
                                       It works!
    
    % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/phptest.php
       PHP Logo                                                                   
                                                                                  
                                   PHP Version 5.3.17                             
    
       System          NetBSD ip-10-202-65-196.ec2.internal 6.0.1 NetBSD 6.0.1    
                       (XEN3PAE_DOMU) i386                                        
       Build Date      Dec 14 2012 10:31:13                                       
    ...
    
    % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/webapp/
       Showing table hf.names:
    
       +--------------------+
       | id | first  | last |
       |----+--------+------|
       | 1  | Donald | Duck |
       |----+--------+------|
       | 2  | Daisy  | Duck |
       +--------------------+
    
         ----------------------------------------------------------------------
    
       Enter new values:
    
       first:     _____________________ 
       last:      _____________________ 
       [ Submit ] 
    
  10. At this point, everything is setup and can be enjoyed. If the instance is needed no longer, it can be terminated:
    % euca-describe-instances
    RESERVATION     r-d77272ad      749335780469    ec2-webservers
    INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com       ...
    % euca-terminate-instances i-9fafc2f2
    INSTANCE        i-9fafc2f2
    % euca-describe-instances
    RESERVATION     r-d77272ad      749335780469    ec2-webservers
    INSTANCE        i-9fafc2f2      ami-5d0f8034                    terminated      eucaHF  ...
    
What's next on my journey is to put database and webserver into separate VMs. First one of each, and then see if I find the nerve to look into a setup with more redundancy.

Shameless plug: I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like!

[Tags: , , , , , , ]



[20130228] Ansible and NetBSD milestone 1 reached: playbooks for system config, web+db servers
In my quest to play with Ansible, I've reached my first milestone: I now have playbooks that take a basic NetBSD installation, configure it into a usable base installation, and then add a MySQL database, Apache and PHP to use it as webserver, and then deploy a simple web application.

The playbooks are too emberassing to publish, but here are the steps to get things going:

  1. Setup NetBSD 6.0 with "base" and "etc" set, also add "pkgin" from menu
  2. Allow root logins via ssh (for a start, will be changed later)
  3. Install ansible-1.0nb1 binary package with all its depends
  4. From a management station, run: ansible-playbook -k -i hosts-HF config-netbsd-basic.yml
  5. Then, run: ansible-playbook -i hosts-HF config-netbsd-dbserver.yml
  6. Last, run: ansible-playbook -i hosts-HF config-netbsd-webserver.yml
After that, a simple "phptest()" page, phpmyadmin and my simple PHP-based web application can be run. Administration of the system is via SSH and sudo, root logins were disabled in the first ansible playbook.

Now to tweak the ansible playbooks to look less ugly, use variables, and then separate database and webserver into two separate machines - all in preparation to move them into the Amazon EC2 cloud. Stay tuned!

For the record, here's a log of the three ansible playbooks above, starting from my basic NetBSD installation that already has pkgin and ansible:

% ansible-playbook -k -i hosts-HF config-netbsd-basic.yml
SSH password: 

PLAY [netbsd] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Install tcsh] ********************* 
changed: [10.0.0.181]

TASK: [Add user feyrer] ********************* 
changed: [10.0.0.181]

TASK: [Create ~feyrer/.ssh directory] ********************* 
changed: [10.0.0.181]

TASK: [Enable ssh login with ssh-key] ********************* 
changed: [10.0.0.181]

TASK: [Install sudo] ********************* 
changed: [10.0.0.181]

TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
changed: [10.0.0.181]

TASK: [Disable ssh logins as root] ********************* 
changed: [10.0.0.181]

NOTIFIED: [restart sshd] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=9    changed=8    unreachable=0    failed=0    


% ansible-playbook    -i hosts-HF config-netbsd-dbserver.yml

PLAY [dbservers] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Install mysql] ********************* 
changed: [10.0.0.181]

TASK: [Install MySQL rc.d script] ********************* 
changed: [10.0.0.181]

TASK: [Start MySQL service] ********************* 
changed: [10.0.0.181]

TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
changed: [10.0.0.181]

TASK: [Setup DB] ********************* 
changed: [10.0.0.181]

TASK: [Add db-user] ********************* 
changed: [10.0.0.181]

TASK: [Copy over DB template] ********************* 
changed: [10.0.0.181]

TASK: [Import DB data] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=9    changed=8    unreachable=0    failed=0    


%% ansible-playbook    -i hosts-HF config-netbsd-webserver.yml

PLAY [webservers] ********************* 

GATHERING FACTS ********************* 
ok: [10.0.0.181]

TASK: [Installing ap24-php53 package and dependencies] ********************* 
changed: [10.0.0.181]

TASK: [Install Apache rc.d script] ********************* 
changed: [10.0.0.181]

TASK: [Enable and start Apache service] ********************* 
changed: [10.0.0.181]

TASK: [Enable PHP in Apache config file] ********************* 
changed: [10.0.0.181] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
changed: [10.0.0.181] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})

TASK: [Make Apache read index.php] ********************* 
changed: [10.0.0.181]

TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
changed: [10.0.0.181]

TASK: [Install phpmyadmin] ********************* 
changed: [10.0.0.181]

TASK: [Enable phpmyadmin in Apache config] ********************* 
changed: [10.0.0.181]

TASK: [Enable PHP modules in PHP config file] ********************* 
changed: [10.0.0.181] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
changed: [10.0.0.181] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})

TASK: [Fix Apache access control] ********************* 
changed: [10.0.0.181]

TASK: [Create directory for webapp] ********************* 
changed: [10.0.0.181]

TASK: [Deploy example webapp] ********************* 
changed: [10.0.0.181]

TASK: [Create webapp symlink for easy access] ********************* 
changed: [10.0.0.181]

NOTIFIED: [restart apache] ********************* 
changed: [10.0.0.181]

PLAY RECAP ********************* 
10.0.0.181                     : ok=15   changed=14   unreachable=0    failed=0    


% links -dump http://10.0.0.181/webapp/
   Showing table hf.names:

   +--------------------+
   | id | first  | last |
   |----+--------+------|
   | 1  | Donald | Duck |
   |----+--------+------|
   | 2  | Daisy  | Duck |
   +--------------------+

     ----------------------------------------------------------------------

   Enter new values:

   first:     _____________________ 
   last:      _____________________ 
   [ Submit ] 

% 


[Tags:
, , , , , ]


[20130203] Managing Amazon/EC2 NetBSD instances with euca2ools
What and Why

Playing with ansible, its "ec2" module came to my attention: it is intended to manage virtual machines in Amazon's EC2 cloud. The idea is that you describe a system with the property "needs to run in Amazon's cloud", and ansible then starts the machine if it isn't there already. In order to get to the point where this can be played with, a working version of the euca2ools package was required first.

Packaging was mostly a no-brainer, and a package is currently under review and will end up in pkgsrc eventually. The more interesting part was to verify if the pkg actually worked as expected. This proved tricky for two reasons: 1) my overall lack of how to use the Amazon AWS command line tools (ec2-ami-tools, ec2-api-tools), and 2) the fact that euca2ools is mostly written for the Eucalyptus Cloud infrastructure, which just happens to be compatible with Amazon AWS. To give future parties something to google, here are the steps that to fire up a NetBSD machine in the Amazon cloud.

How - Prerequirements

A login for Amazon Web Services (AWS) is required, of which the Elastic Cloud Computing (EC2) Xen infrastructure is a part of. I won't go into details of this, please see the NetBSD wiki or my article ``NetBSD in der Cloud'' in the German FreeX 5/2012 magazine, pages 58-63, for details.

Before starting, a few environment variables have to be filled with authentication information. Log into the Amazon AWS Console, click on your name in the upper right corner to get to the "Security Credentials" page, and create an access key if not already present. Get the acces key ID and the secret key, and put them into environment variables EC2_ACCESS_KEY and EC2_SECRET_KEY:

% setenv EC2_ACCESS_KEY "AKxxxxxxxxxxxxxxxxxx"
% setenv EC2_SECRET_KEY "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Next create and download a X.509 certificate - make sure to get both the file with the private key (pk-XXXX.pem) as well as the file with the public key (cert-XXX.pem). Set the environment variables EC2_CERT and EC2_PRIVATE_KEY to thos files, respectively:
% setenv EC2_CERT         .../cert-XXX.pem
% setenv EC2_PRIVATE_KEY  .../pk-XXX.pem
Last, euca2ools want to know what cloud infrastructure to use for virtual machines (EC2) and storage (S3). Coming from the Eucalyptus project, the tools can talk to cloud servers running Eucalyptus, OpenStack and Amazon AWS. Communication is via HTTP, and the environment variables EC2_URL and S3_URL have to be set accordingly:
% setenv EC2_URL          http://ec2.amazonaws.com
% setenv S3_URL           http://s3.amazonaws.com
Last, make sure your system's time is somewhat in sync with reality, else you will get funny error messages!

So much for the preparations, let's dive into euca2ools.

List Regions and Availability Zones

Amazon's service offers is spread across many data centers across different regions of the world. The list of regions is available via the "euca-describe-regions" command:

% euca-describe-regions
REGION  eu-west-1       ec2.eu-west-1.amazonaws.com
REGION  sa-east-1       ec2.sa-east-1.amazonaws.com
REGION  us-east-1       ec2.us-east-1.amazonaws.com
REGION  ap-northeast-1  ec2.ap-northeast-1.amazonaws.com
REGION  us-west-2       ec2.us-west-2.amazonaws.com
REGION  us-west-1       ec2.us-west-1.amazonaws.com
REGION  ap-southeast-1  ec2.ap-southeast-1.amazonaws.com
REGION  ap-southeast-2  ec2.ap-southeast-2.amazonaws.com
Inside one region, systems are grouped together in "availability zones" - usually data centers or separate security zones within (refer to the Amazon documentation for details). To list the availability zones in one region, use the "euca-describe-availability-zones" command:

% euca-describe-availability-zones
AVAILABILITYZONE        us-east-1a      available
AVAILABILITYZONE        us-east-1b      available
AVAILABILITYZONE        us-east-1c      available
AVAILABILITYZONE        us-east-1d      available
To specify what region to talk to there are two ways. The first is to specify the region name on any of the following commands with the "--region" option (yuck):

% euca-describe-availability-zones --region eu-west-1
AVAILABILITYZONE        eu-west-1a      available
AVAILABILITYZONE        eu-west-1b      available
AVAILABILITYZONE        eu-west-1c      available
The second one is to adjust your EC2_URL to point to your preferred region directly:
% setenv EC2_URL http://ec2.eu-west-1.amazonaws.com
% euca-describe-availability-zones --region eu-west-1
AVAILABILITYZONE        eu-west-1a      available
AVAILABILITYZONE        eu-west-1b      available
AVAILABILITYZONE        eu-west-1c      available
Systems and Listing Available Machine Images

Now that we have a basic overview of the cloud infrastructure with its regions and availability zones, the next questions are what hardware is available for running virtual machine instances on, and what operating systems can be put on.

Amazon lists available hardware configurations on their "instance types" web sites. Sizes range from Micro Instances with 613MB RAM, up to two CPU cores and no local harddisk (t1.micro) to Extra Large (XL) Instances with 15GB RAM, 8 CPU cores and 1.690 GB local harddisk. Many more configurations are available for situations that require much memory, much CPU, much IO, or do cluster computing with CPU and GPU.

As for the operating system and software to put on those virtual machine instances, there is a VERY wide choice available. The "euca-describe-images --all" command lists all available optione:

% euca-describe-images --all
...
IMAGE   ami-abd0d0df    101367081206/NetBSD-i386-6.0-20121015-1054 \
  101367081206    available       public          i386    machine \
  aki-64695810                    ebs
IMAGE   ami-7fc3c30b    101367081206/NetBSD-x86_64-6.0-20121014-1007 \
  101367081206    available       public          x86_64  machine \
  aki-62695816                    ebs
...
In the output, the configuration is identified by the Amazon Machine Identifier (AMI), e.g. "ami-7fc3c30b" for a NetBSD 6.0/amd64 instance. This image ID is required when defining what virtual machine instance to start.

Note that the "euca-describe-images" command depends on the region setting, so you will get (and need) different output depending on the region that you intend your instances to run in.

Setup SSH Access

When starting a NetBSD AMI, access will be via SSH to the root account. For that, a SSH key pair needs to be created with the "euca-add-keypair" command. The command can write the private key to a local file, be sure to protect it properly - it will be the only way of access to the system! Other interesting commands when managing SSH keys are "euca-describe-keypairs" and "euca-delete-keypair":

% euca-describe-keypairs
% euca-add-keypair -f key-eucaHF.pem eucaHF
KEYPAIR eucaHF  b8:e9:05:7e:3a:df:c7:8e:eb:6e:8d:72:ff:77:68:01:e2:03:7e:3e
% euca-describe-keypairs
KEYPAIR eucaHF  b8:e9:05:7e:3a:df:c7:8e:eb:6e:8d:72:ff:77:68:01:e2:03:7e:3e
% euca-delete-keypair eucaHF
KEYPAIR eucaHF
% euca-describe-keypairs
%
Of course we want to keep a key for logging with it, so let's re-run the important part:
% euca-add-keypair -f key-eucaHF.pem eucaHF
KEYPAIR eucaHF  9b:d4:15:09:bc:51:b1:76:5c:db:a3:93:52:f0:d8:08:87:a4:80:c7
% cat key-eucaHF.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAn8rCLhqLyfke+NqeOkqb6BUIbfwBFm/9ddG8ghVt9CUmyKUMRrKFSyaTRreO
...
wA5a3XZuEFw83HdGrhaRgom2ZJ1SEk2889FpAA+yrhveKhDJIe6Zc2rM+crqUWBfnvs=
-----END RSA PRIVATE KEY-----
Manage Virtual Machine Instances

Now that everything is prepared, telling the cloud infrastructure to find physical hardware, put our preferred operating system on it, and start it is done with the "euca-run-instance" command:

% euca-run-instances -t t1.micro -k eucaHF ami-7fc3c30b
RESERVATION     r-2182506a      749335780469    default
INSTANCE        i-2ed60264      ami-7fc3c30b                    pending \
  eucaHF  0               t1.micro        2013-02-03T15:51:49.000Z        \
  us-east-1b      aki-62695816                    monitoring-disabled \
  ebs 
That's actually as complicate as it gets - one command that tells what hardware to use (t1.micro - can be omitted, a useful default will be chosen), what SSH key to use for the root account, and what machine image (AMI) to use are all used here. In return, the command prints a number of information from the freshly created instance. The one used in the following commands is the "instance id", "i-2ed60264" in this example.

When the above command was started, this is a good time to go back to the Amazon AWS console and have a look at your instances - you will find the one listed above there now, too! Instead of the web-based console, the "euca-describe-instances" command can be used:

% euca-describe-instances
RESERVATION     r-2182506a      749335780469    default
INSTANCE        i-2ed60264      ami-7fc3c30b    ec2-54-228-22-143.compute.amazonaws.com       \
  ip-10-226-194-20.compute.internal     running eucaHF  0   \
  t1.micro        2013-02-03T15:51:49.000Z        us-east-1b \
  aki-62695816                    monitoring-disabled     \
  54.228.22.143   10.226.194.20                   ebs
Now this is all nice and dandy, but we have just created a NetBSD machine in the Amazon cloud. Let's log in!!!1!

To do so, we need the private key file created with the "euca-add-keypair" command, and the host name. The latter is available in the list of instances - be sure to use the one within the "compute.anazonaws.com" domain:

% ssh -i key-eucaHF.pem -l root ec2-54-228-22-143.compute.amazonaws.com
The authenticity of host 'ec2-54-228-22-143.compute.amazonaws.com (54.228.22.143)'
can't be established.
ECDSA key fingerprint is f7:a9:f6:21:fc:d2:0e:46:03:41:f8:d5:c1:72:92:28.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-54-228-22-143.compute.amazonaws.com,54.228.22.143' (ECDSA)
to the list of known hosts.
NetBSD 6.0 (XEN3_DOMU)
Welcome to NetBSD - Amazon EC2 image!

This system is running a snapshot of a stable branch of the NetBSD
operating system, adapted for running on the Amazon EC2 infrastructure.

The environment is very similar to one provided within a typical Xen domU
installation. It contains a small, autonomous environment (including a
compiler toolchain) that you can run to build your own system.

The file system is lightly populated so you have plenty of space to play with.
Should you need a src or pkgsrc tree, please use the "bootstrap" script found
under /usr to download them.  You can also use the script to set up
binary packages using "pkgin":

                /usr/bootstrap.sh [src|pkgsrc|binpkg]

This AMI sends email to the maintainer on first boot, to help get
an idea of what is in use at any given time.

You are encouraged to test this image as thoroughly as possible.  Should you
encounter any problem, please report it back to the development team using the
send-pr(1) utility (requires a working MTA).  If yours is not properly set up,
use the web interface at: http://www.NetBSD.org/support/send-pr.html

Thank you for helping us test and improve NetBSD's quality!
Terminal type is vt220.
We recommend that you create a non-root account and use su(1) for root access.
ip-10-226-194-20# uname -a
NetBSD ip-10-226-194-20.compute.internal 6.0 NetBSD 6.0 (XEN3_DOMU) amd64
ip-10-226-194-20# exit
From here, you are on your own - it's a NetBSD machine, after all.

One word of warning at this point: Amazon AWS is not for free (as you should be aware from the Preparations step). If you do not need machines any more, be sure to remove them from the cluster, else this may drive up your bill for nothing! You can use the "euca-terminate-instances" command to do just that:

% euca-terminate-instances i-2ed60264
INSTANCE        i-2ed60264
When you look at the output of "euca-describe-instances" now, you will see that the machine's state goes from "running" first to "shuting-down" then to "terminated" - the cloud infrastructure will eventually be cleaned up to not list the stale machines any more.

What's next?

As stated above, the whole goal of this exercise is to manage Amazon EC2 images from ansible. Weekend's mostly over and we will see where this journey is going. For the time being, I'm happy to hear about any comments of you using NetBSD on Amazon's EC2, and of my euca2ools package.

Appendix: euca2ools Cheat Sheet

Environment variables:
  setenv EC2_ACCESS_KEY "AKxxxxxxxxxxxxxxxxxx"
  setenv EC2_SECRET_KEY "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  setenv EC2_CERT         .../cert-XXX.pem
  setenv EC2_PRIVATE_KEY  .../pk-XXX.pem
  setenv EC2_URL          http://ec2.amazonaws.com
  setenv S3_URL           http://s3.amazonaws.com

Regions & availability zones:
  euca-describe-regions
  euca-describe-availability-zones
  euca-describe-availability-zones --region eu-west-1
Change default region:
  setenv EC2_URL		http://ec2.eu-west-1.amazonaws.com

AMIs:
  euca-describe-images --all

SSH Keypairs:
  euca-add-keypair		-f key-eucaHF.pem eucaHF
  euca-describe-keypairs	
  euca-delete-keypair		eucaHF

Instances:
  euca-run-instances -k eucaHF ami-7fc3c30b
  euca-describe-instances
  euca-describe-instances i-96a773dc
  ssh -i key-eucaHF.pem ec2-54-328-43-220.compute.amazonaws.com -l root
  euca-terminate-instances i-96a773dc


[Tags: , , , , ]


[20120414] Playing with Amazon's EC2 and NetBSD - top(1) pr0n
I've played with NetBSD on Amazon's EC2 service recently, and here's a small teaser for ressources that one can get with a few mouse clicks - operating system is NetBSD 6.0_BETA/amd64, note number of CPU cores and RAM:



[Tags: , , ]



[20120307] NetBSD/xen available for Multi-Processor machines
Manuel Bouyer announces that NetBSD/xen is now available for Multi-Processor machines. Citing from the release announcement:

``The NetBSD Foundation is pleased to announce completion of Multiprocessing Support for the port of its Open Source Operating System to the Xen hypervisor.

The NetBSD Fundation started the Xen MP project 8 month ago; the goal was to add SMP support to NetBSD/Xen domU kernels. This project has officially completed, and after a few bug fixes in the pmap(9) code it is now considered stable on both i386 and amd64. NetBSD 6.0 will ship with option MULTIPROCESSOR enabled by default for Xen domU kernels.

The availability of Xen MP support in NetBSD allows to run the NetBSD Open Source Operating Systems on a range of available infrastructure providers' systems. Amazon's Web Services with their Elastic Cloud Computing is a prominent examples here.

Xen is a virtualization software that enables several independent operating system instances ("domains") to run concurrently on the same computer hardware. The hardware is managed by the first domain (dom0), and further guest/user domains (domU) are spawned and managed by dom0. Operating systems available for running as dom0 and domU guests include Microsoft Windows, Solaris and Linux besides NetBSD.

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license. NetBSD is developed and supported by a large and vivid international community. Many applications are readily available through pkgsrc, the NetBSD Packages Collection.

NetBSD has been available for the Xen hypervisor since Xen 1 and NetBSD 2.0, released in 2004 , but until now only a single processor was supported in each NetBSD/xen domain.''

[Tags: , , , ]



[20120205] Automatic building of Amazon EC2 images from NetBSD
NetBSD/xen is available for some time now to work on Amazon's Xen-based Elastic Cloud Computing (EC2) platform, as previously announced in the NetBSD blog, and the NetBSD wiki has instructions on how to subscribe to Amazon, launching and rebuilding the "AMI" images.

This work is continued by Jean-Yves Migeon, who is working on build scripts for Amazon EC2, so the "AMI" images can be provided easily, with the eventual goal to include them into the NetBSD build process by Jeff Rizzo, so EC2 images can be automatically generated easily, e.g. by NetBSD's build cluster.

[Tags: , , ]



Previous 3 entries

Tags: , 2bsd, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, hardware, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrcCon, pkgsrccon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, Products, products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, Releases, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, Security, security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 23619425
Copyright (c) Hubert Feyrer