hubertf's NetBSD blog

hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!

[20130324]	Ansible & EC2 - Playbooks for orchestrating NetBSD into the cloud As follower of my blog you have seen the steps towards getting NetBSD instances started in Amazon's EC2 cloud with a simple web application deployed on one EC2 instance and the database on another one. These blog articles were very detailed on purpose, to have full logfiles available just in case needed. I have used these logs to prepare my pkgsrcCon 2013 talk about Ansible and Amazon's EC2, so things can be looked at without actually running anything. As it turns out this was good, because the 32bit NetBSD instances that I've used during my pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation was a bit more on the theoretical side than I originally planned. Now after pkgsrcCon is over, I would like to publish the presentation slides with all the details, and especially the playbooks and all other files to look at - enjoy! [Tags: amazon, ansible, aws, ec2, pkgsrc, xen]
[20130321]	Ansible, EC2 and NetBSD final milestone 4 reached: Web and DB on separate VMs in the cloud In the fourth and last step on my journey to use Ansible to bring a non-trivial system of a Web server and a DB server into Amazon's EC2 cloud, this is the final step. After starting out with a local VMware VM and making first steps with Ansible and EC2, the previous step was to push a single system into the cloud. Now, the final step is to setup two distinct VMs, one for the database and one for the webserver, and then make them known to each other. The single steps are: Prepare the two VMs Basic setup for all systems Install the database server Install the webserver Connect database and webserver Again, here are all the steps in detail: As before, ensure local time is correct when talking to Amazon, and also make sure the SSH agent has the proper key loaded. % date Thu Mar 21 00:45:37 CET 2013 % ssh-add -l 2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA) Make sure security groups are setup properly. We use one group for the database server, and one for the webserver. This defines the access permissions from the internet, and also allows to identify systems for their individual configuration and also for connecting them in the final step: % euca-describe-groups ... GROUP sg-ae54b3c5 749335780469 ec2-dbservers Database servers PERMISSION 749335780469 ec2-dbservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-dbservers ALLOWS tcp 3306 3306 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-dbservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 GROUP sg-a854b3c3 749335780469 ec2-webservers Web servers PERMISSION 749335780469 ec2-webservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 Now, run our playbook to setup the two VMs. This uses the single playbook from the previous milestone, and just runs it twice with different security groups: % ansible-playbook -i hosts-HF config-ec2-prepare-db+web-vm.yml PLAY [localhost] ******************* TASK: [ec2-webservers \| Launch new EC2 instance] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Give the system 30 seconds to boot up] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Fix /usr/bootstrap.sh to run pkgin with -y] ***************** changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'}) changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'}) TASK: [ec2-webservers \| Install pkgin via /usr/bootstrap.sh] ***************** changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'}) TASK: [ec2-webservers \| Copy over Ansible binary package] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Install Ansible dependencies] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Install Ansible package (manually)] ***************** changed: [127.0.0.1] TASK: [ec2-webservers \| Setup lame /usr/bin/python symlink] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Launch new EC2 instance] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Give the system 30 seconds to boot up] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Fix /usr/bootstrap.sh to run pkgin with -y] ***************** changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'}) changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'}) TASK: [ec2-dbservers \| Install pkgin via /usr/bootstrap.sh] ***************** changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'}) TASK: [ec2-dbservers \| Copy over Ansible binary package] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Install Ansible dependencies] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Install Ansible package (manually)] ***************** changed: [127.0.0.1] TASK: [ec2-dbservers \| Setup lame /usr/bin/python symlink] ***************** changed: [127.0.0.1] PLAY RECAP ***************** 127.0.0.1 : ok=18 changed=18 unreachable=0 failed=0 Just to make sure, check that the two instances run properly, and are in the right security groups, ec2-webservers and ec2-dbservers: % euca-describe-instances RESERVATION r-a419f9d9 749335780469 ec2-webservers INSTANCE i-21b7c441 ami-5d0f8034 ... RESERVATION r-641efe19 749335780469 ec2-dbservers INSTANCE i-54a2ab3e ami-5d0f8034 ... Next, bring the two freshly setup systems (which are already capable of acting as ansible targets) up to our basic system setup: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ***************** TASK: [ping] ***************** ok: [ec2-54-235-44-118.compute-1.amazonaws.com] ok: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install tcsh] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Add user feyrer] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Create ~feyrer/.ssh directory] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable ssh login with ssh-key] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install sudo] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Disable ssh logins as root] ***************** ok: [ec2-54-235-44-118.compute-1.amazonaws.com] ok: [ec2-54-234-139-151.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-54-234-139-151.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0 ec2-54-235-44-118.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0 Check: % ssh ec2-54-234-139-151.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % % ssh ec2-54-235-44-118.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) Now that the two machines run with our basline configuration, install their individual software and settings. First the database server: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml PLAY [security_group_ec2-dbservers] ***************** TASK: [Install mysql] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Install MySQL rc.d script] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Start MySQL service] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Install python-mysqldb (for mysql_user module)] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Setup DB] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Add db-user] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Copy over DB template] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Import DB data] ***************** changed: [ec2-54-235-44-118.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-54-235-44-118.compute-1.amazonaws.com : ok=8 changed=8 unreachable=0 failed=0 Check and see if the database works as expected: % ssh -t ec2-54-235-44-118.compute-1.amazonaws.com mysql -u webapp -p webapp Enter password: **** ... mysql> show tables; +------------------+ \| Tables_in_webapp \| +------------------+ \| names \| +------------------+ 1 row in set (0.01 sec) mysql> select * from names; +----+--------+------+ \| id \| first \| last \| +----+--------+------+ \| 1 \| Donald \| Duck \| \| 2 \| Daisy \| Duck \| +----+--------+------+ 2 rows in set (0.00 sec) mysql> bye Excellent. Now setup the webserver, too: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml PLAY [security_group_ec2-webservers] ***************** TASK: [Installing ap24-php53 package and dependencies] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install Apache rc.d script] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable and start Apache service] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable PHP in Apache config file] ******************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'LoadModule.mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'AddHandler.x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'}) TASK: [Make Apache read index.php] ******************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install phpmyadmin] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable phpmyadmin in Apache config] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Fix Apache access control for phpmyadmin] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable PHP modules in PHP config file] ******************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.zlib.so', 'l': 'extension=zlib.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.zip.so', 'l': 'extension=zip.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.mysqli.so', 'l': 'extension=mysqli.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.mysql.so', 'l': 'extension=mysql.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.mcrypt.so', 'l': 'extension=mcrypt.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.mbstring.so', 'l': 'extension=mbstring.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.json.so', 'l': 'extension=json.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.gd.so', 'l': 'extension=gd.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.gettext.so', 'l': 'extension=gettext.so'}) changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.bz2.so', 'l': 'extension=bz2.so'}) TASK: [Create directory for webapp] ******************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Deploy example webapp] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Create webapp symlink for easy access] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] NOTIFIED: [restart apache] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-54-234-139-151.compute-1.amazonaws.com : ok=14 changed=14 unreachable=0 failed=0 Again, test: % links -dump ec2-54-234-139-151.compute-1.amazonaws.com/ It works! % % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/phptest.php \| head PHP Logo PHP Version 5.3.17 System NetBSD ip-10-80-61-33.ec2.internal 6.0.1 NetBSD 6.0.1 (XEN3PAE_DOMU) i386 Build Date Dec 14 2012 10:31:13 './configure' '--with-config-file-path=/usr/pkg/etc' '--with-config-file-scan-dir=/usr/pkg/etc/php.d' '--sysconfdir=/usr/pkg/etc' '--localstatedir=/var' % % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/ Showing table hf.names: Cannot connect to database: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)(2002) Close to optimum, but the last error is actually expectet: In order for proper operation, the Database needs to grant the webserver access, and the web server needs to know where the database server is. So let's connect them! This step is done by preparing a shell script on both systems, which will then be ran to - depending on the system's security group - perform the proper steps: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-connections.yml PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ***************** TASK: [Collect EC2 host information] ***************** ok: [ec2-54-234-139-151.compute-1.amazonaws.com] ok: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Prepare connection-script in /tmp/do-connect-vms.sh] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Run connection-script] ***************** changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-54-234-139-151.compute-1.amazonaws.com : ok=3 changed=2 unreachable=0 failed=0 ec2-54-235-44-118.compute-1.amazonaws.com : ok=3 changed=2 unreachable=0 failed=0 With that final step, our test web application works, and the webserver can access the database properly: % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/** Showing table hf.names: +--------------------+ \| id \| first \| last \| \|----+--------+------\| \| 1 \| Donald \| Duck \| \|----+--------+------\| \| 2 \| Daisy \| Duck \| +--------------------+ ---------------------------------------------------------------------- Enter new values: first: _____________________ last: _____________________ [ Submit ] So much for this exercise. I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like, or stay tuned to find my presentation and all the data after pkgsrcCon 2013. [Tags: amazon, ansible, ec2, xen]
[20130321]	Ansible, EC2 and NetBSD milestone 3 reached: Web and database in the cloud With the previous work on setting up a local VM as database and web server and setting up a Xen VM in Amazon's EC2 cloud combined, it is pretty straight forward to setup a EC2 instance that has all the software to serve a simple web application from the cloud. The single steps are: Prepare the environment with proper time, SSH agent and EC2 firewall groups Setup EC2 instance with pkgin and ansible Do basic preparations to meet our standards for logins, shells and general usability and security Setup database server with DB software, user and import of data Setup web server with all the software and some demo application The following details show all the commands can and their output in more detail: Make sure time is set properly - needed when talking to Amazon EC2: % sudo sh /etc/rc.d/ntpd stop ntpd not running? (check /var/run/ntpd.pid). % sudo sh /etc/rc.d/ntpdate restart Setting date via ntp. % sudo sh /etc/rc.d/ntpd start Starting ntpd. % date Sat Mar 16 16:46:19 CET 2013 Teach our EC2 SSH key to SSH agent, so we don't have to type a password (which we don't know anyways - EC2 only works with SSH keys): % ssh-add -l Could not open a connection to your authentication agent. % % eval `ssh-agent` Agent pid 10467 % ssh-add -l The agent has no identities. % ssh-add ../../euca2ools/key-eucaHF.pem Identity added: ../../euca2ools/key-eucaHF.pem (../../euca2ools/key-eucaHF.pem) % ssh-add -l 2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA) Check security (firewall) groups - those are stored in EC2, and we have previously set them up: % euca-describe-groups ... GROUP sg-a854b3c3 749335780469 ec2-webservers Web servers PERMISSION 749335780469 ec2-webservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 See if there are any EC2 instances running: % euca-describe-instances % No - that's fine, we are about to change that! Run first playbook to launch EC2 instance and prepare it for using with ansible: % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml PLAY [localhost] ******************* TASK: [Launch new EC2 instance] ***************** changed: [127.0.0.1] TASK: [Give the system 30 seconds to boot up] ***************** changed: [127.0.0.1] TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ***************** changed: [127.0.0.1] TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ***************** changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'}) changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'}) TASK: [Install pkgin via /usr/bootstrap.sh] ***************** changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'}) TASK: [Copy over Ansible binary package] ***************** changed: [127.0.0.1] TASK: [Install Ansible dependencies] ***************** changed: [127.0.0.1] TASK: [Install Ansible package (manually)] ***************** changed: [127.0.0.1] TASK: [Setup lame /usr/bin/python symlink] ***************** changed: [127.0.0.1] PLAY RECAP ***************** 127.0.0.1 : ok=9 changed=9 unreachable=0 failed=0 We now have a EC2 instance running that has Ansible installed: % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 ec2-107-22-69-112.compute-1.amazonaws.com ... With this EC2 instance, we can do some basic preparations for our standards, e.g. a login without requiring root (and while there, actually disable allowing as root), setup sudo and a proper shell: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers] ***************** TASK: [ping] ***************** ok: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install tcsh] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add user feyrer] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Create ~feyrer/.ssh directory] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable ssh login with ssh-key] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install sudo] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Disable ssh logins as root] ***************** ok: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0 Let's have a look if things actually work: % ssh 107.22.69.112 id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % ssh ec2-107-22-69-112.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % ssh ec2-107-22-69-112.compute-1.amazonaws.com sudo id uid=0(root) gid=0(wheel) groups=0(wheel),2(kmem),3(sys),4(tty),5(operator),20(staff),31(guest) Next, install database software and import our demo database, just as we did in out local VM: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml PLAY [security_group_ec2-webservers] ***************** TASK: [Install mysql] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install MySQL rc.d script] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Start MySQL service] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install python-mysqldb (for mysql_user module)] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Setup DB] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add db-user] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Copy over DB template] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Import DB data] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=8 changed=8 unreachable=0 failed=0 Again, let's see if everything works as expected: % ssh ec2-107-22-69-112.compute-1.amazonaws.com ... ip-10-202-65-196: {1} mysql -u webapp -p webapp Enter password: ** ... mysql> show tables; +------------------+ \| Tables_in_webapp \| +------------------+ \| names \| +------------------+ 1 row in set (0.00 sec) mysql> select * from names; +----+--------+------+ \| id \| first \| last \| +----+--------+------+ \| 1 \| Donald \| Duck \| \| 2 \| Daisy \| Duck \| +----+--------+------+ 2 rows in set (0.00 sec) mysql> exit Bye ip-10-202-65-196: {2} exit logout Connection to ec2-107-22-69-112.compute-1.amazonaws.com closed. Last, add Apache+PHP and our small demo web-application: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml PLAY [security_group_ec2-webservers] ***************** TASK: [Installing ap24-php53 package and dependencies] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install Apache rc.d script] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable and start Apache service] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PHP in Apache config file] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'LoadModule.mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'AddHandler.x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'}) TASK: [Make Apache read index.php] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Install phpmyadmin] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable phpmyadmin in Apache config] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Fix Apache access control for phpmyadmin] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Enable PHP modules in PHP config file] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.zlib.so', 'l': 'extension=zlib.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.zip.so', 'l': 'extension=zip.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mysqli.so', 'l': 'extension=mysqli.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mysql.so', 'l': 'extension=mysql.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mcrypt.so', 'l': 'extension=mcrypt.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.mbstring.so', 'l': 'extension=mbstring.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.json.so', 'l': 'extension=json.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.gd.so', 'l': 'extension=gd.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.gettext.so', 'l': 'extension=gettext.so'}) changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.bz2.so', 'l': 'extension=bz2.so'}) TASK: [Create directory for webapp] ******************* changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Deploy example webapp] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] TASK: [Create webapp symlink for easy access] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] NOTIFIED: [restart apache] ***************** changed: [ec2-107-22-69-112.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-107-22-69-112.compute-1.amazonaws.com : ok=14 changed=14 unreachable=0 failed=0 Test! % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/ It works! % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/phptest.php PHP Logo PHP Version 5.3.17 System NetBSD ip-10-202-65-196.ec2.internal 6.0.1 NetBSD 6.0.1 (XEN3PAE_DOMU) i386 Build Date Dec 14 2012 10:31:13 ... % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/webapp/ Showing table hf.names: +--------------------+ \| id \| first \| last \| \|----+--------+------\| \| 1 \| Donald \| Duck \| \|----+--------+------\| \| 2 \| Daisy \| Duck \| +--------------------+ ---------------------------------------------------------------------- Enter new values: first: _____________________ last: _____________________ [ Submit ] At this point, everything is setup and can be enjoyed. If the instance is needed no longer, it can be terminated: % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 ec2-107-22-69-112.compute-1.amazonaws.com ... % euca-terminate-instances i-9fafc2f2 INSTANCE i-9fafc2f2 % euca-describe-instances RESERVATION r-d77272ad 749335780469 ec2-webservers INSTANCE i-9fafc2f2 ami-5d0f8034 terminated eucaHF ... What's next on my journey is to put database and webserver into separate VMs. First one of each, and then see if I find the nerve to look into a setup with more redundancy. Shameless plug:** I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like! [Tags: amazon, ansible, apache, ec2, mysql, php, xen]
[20130314]	Ansible, EC2 and NetBSD milestone 2 reached: Instance preparation and communication On my quest to use Ansible to get a NetBSD virtual machine into Amazon's EC2 cloud, I've previously described how I use ansible to prepare a local machine. Working from a basic NetBSD setup, the system is setup for basic operation, the configured as both a database server and a Web/PHP server to serve a small demo application. Now the next step is to replace the VM with an Amazon EC2 instance. I have previously written about how to manage Amazon/EC2 NetBSD instances, and here are the steps that I make to first prepare an EC2 instance with NetBSD and Ansible, and then use a regular Ansible playbook to talk to all my EC2 instances. Note that the connection between the machines setup via euca2ools and ansible is in the security group names. In this case, the security group "ec2-webservers" is assumed to exist. Make sure SSH agent runs and has the EC2 SSH-key added: % ssh-add -l Could not open a connection to your authentication agent. % eval `ssh-agent` Agent pid 9304 % ssh-add -l The agent has no identities. % ssh-add .../key-ec2HF.pem Identity added: ../../euca2ools/key-ec2HF.pem (../../euca2ools/key-ec2HF.pem) % ssh-add -l 2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-ec2HF.pem (RSA) % When using a VM to talk to EC2, pay special attention that it has the correct time, else funny things will happen: % date Sun Mar 10 14:42:33 CET 2013 Setup the ec2-webservers security (firewall) group. This is used both when creating the EC2 instances, and when accessing them. It's the link between EC2 and Ansible's ec2.py script. % euca-add-group -d 'Web servers' ec2-webservers % euca-authorize -P tcp -p 80-80 -s 0.0.0.0/0 ec2-webservers % euca-authorize -P tcp -p 22-22 -s 0.0.0.0/0 ec2-webservers % euca-authorize -P icmp -s 0.0.0.0/0 ec2-webservers % % euca-describe-groups GROUP sg-a854b3c3 749335780469 ec2-webservers Web servers PERMISSION 749335780469 ec2-webservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 List out EC2 instances: % euca-describe-instances % None so far. Let's use our playbook to prepare our first EC2 instance: % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml PLAY [localhost] ******************* TASK: [Launch new EC2 instance] ***************** changed: [127.0.0.1] TASK: [Give the system 30 seconds to boot up] ***************** changed: [127.0.0.1] TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ***************** changed: [127.0.0.1] TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ***************** changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'}) changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," </usr/bootstrap.sh.orig >/usr/bootstrap.sh'}) changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'}) TASK: [Install pkgin via /usr/bootstrap.sh] ***************** changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'}) TASK: [Copy over Ansible binary package] ***************** changed: [127.0.0.1] TASK: [Install Ansible dependencies] ***************** changed: [127.0.0.1] TASK: [Install Ansible package (manually)] ***************** changed: [127.0.0.1] TASK: [Setup lame /usr/bin/python symlink] ***************** changed: [127.0.0.1] PLAY RECAP ***************** 127.0.0.1 : ok=9 changed=9 unreachable=0 failed=0 % There we go. Let's list it: % euca-describe-instances RESERVATION r-bb3b6ac1 749335780469 ec2-webservers INSTANCE i-2cb9a45f ami-a754dbce ec2-54-234-59-5.compute-1.amazonaws.com \ ip-10-243-150-74.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:47:32.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 54.234.59.5 10.243.150.74 ebs % That worked - excellent! Let's add a few more, just for kicks: % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml >&/dev/null & % <...wait...> % euca-describe-instances RESERVATION r-bb3b6ac1 749335780469 ec2-webservers INSTANCE i-2cb9a45f ami-a754dbce ec2-54-234-59-5.compute-1.amazonaws.com \ ip-10-243-150-74.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:47:32.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 54.234.59.5 10.243.150.74 ebs RESERVATION r-8b3c6df1 749335780469 ec2-webservers INSTANCE i-7cb5a80f ami-a754dbce ec2-23-20-42-71.compute-1.amazonaws.com \ ip-10-203-73-195.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:50:48.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 23.20.42.71 10.203.73.195 ebs RESERVATION r-733f6e09 749335780469 ec2-webservers INSTANCE i-42b5a831 ami-a754dbce ec2-23-20-87-176.compute-1.amazonaws.com \ ip-10-116-37-145.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:50:54.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 23.20.87.176 10.116.37.145 ebs RESERVATION r-713f6e0b 749335780469 ec2-webservers INSTANCE i-40b5a833 ami-a754dbce ec2-54-242-254-237.compute-1.amazonaws.com \ ip-10-195-47-153.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:50:54.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 54.242.254.237 10.195.47.153 ebs RESERVATION r-773f6e0d 749335780469 ec2-webservers INSTANCE i-46b5a835 ami-a754dbce ec2-54-235-232-227.compute-1.amazonaws.com \ ip-10-194-7-72.ec2.internal running ec2HF 0 t1.micro \ 2013-03-10T13:50:54.000Z us-east-1a aki-825ea7eb \ monitoring-disabled 54.235.232.227 10.194.7.72 ebs RESERVATION r-b72475cd 749335780469 ec2-webservers INSTANCE i-b2adb0c1 ami-a754dbce ec2-50-16-129-62.compute-1.amazonaws.com \ domU-12-31-39-14-C6-CB.compute-1.internal running ec2HF 0 t1.micro \ 2013-03-10T13:55:24.000Z us-east-1d aki-825ea7eb \ monitoring-disabled 50.16.129.62 10.206.197.53 ebs % Let's talk to our EC2 instances now. For that, we use the ec2.py script, which enumerates all instances: % ./ec2.py --list { "i-2cb9a45f": [ "ec2-54-234-59-5.compute-1.amazonaws.com" ], "i-40b5a833": [ "ec2-54-242-254-237.compute-1.amazonaws.com" ], "i-42b5a831": [ "ec2-23-20-87-176.compute-1.amazonaws.com" ], "i-46b5a835": [ "ec2-54-235-232-227.compute-1.amazonaws.com" ], "i-7cb5a80f": [ "ec2-23-20-42-71.compute-1.amazonaws.com" ], "i-b2adb0c1": [ "ec2-50-16-129-62.compute-1.amazonaws.com" ], "key_ec2HF": [ "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2-23-20-42-71.compute-1.amazonaws.com", "ec2-23-20-87-176.compute-1.amazonaws.com", "ec2-54-242-254-237.compute-1.amazonaws.com", "ec2-54-235-232-227.compute-1.amazonaws.com", "ec2-50-16-129-62.compute-1.amazonaws.com" ], "security_group_ec2-webservers": [ "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2-23-20-42-71.compute-1.amazonaws.com", "ec2-23-20-87-176.compute-1.amazonaws.com", "ec2-54-242-254-237.compute-1.amazonaws.com", "ec2-54-235-232-227.compute-1.amazonaws.com", "ec2-50-16-129-62.compute-1.amazonaws.com" ], "type_t1_micro": [ "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2-23-20-42-71.compute-1.amazonaws.com", "ec2-23-20-87-176.compute-1.amazonaws.com", "ec2-54-242-254-237.compute-1.amazonaws.com", "ec2-54-235-232-227.compute-1.amazonaws.com", "ec2-50-16-129-62.compute-1.amazonaws.com" ], "us-east-1": [ "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2-23-20-42-71.compute-1.amazonaws.com", "ec2-23-20-87-176.compute-1.amazonaws.com", "ec2-54-242-254-237.compute-1.amazonaws.com", "ec2-54-235-232-227.compute-1.amazonaws.com", "ec2-50-16-129-62.compute-1.amazonaws.com" ], "us-east-1a": [ "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2-23-20-42-71.compute-1.amazonaws.com", "ec2-23-20-87-176.compute-1.amazonaws.com", "ec2-54-242-254-237.compute-1.amazonaws.com", "ec2-54-235-232-227.compute-1.amazonaws.com" ], "us-east-1d": [ "ec2-50-16-129-62.compute-1.amazonaws.com" ] } ec2.py can also give us information about one instance: % ./ec2.py --host ec2-54-234-59-5.compute-1.amazonaws.com { "ec2__in_monitoring_element": false, "ec2_ami_launch_index": "0", "ec2_architecture": "x86_64", "ec2_client_token": "", "ec2_dns_name": "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2_eventsSet": "", "ec2_group_name": "", "ec2_hypervisor": "xen", "ec2_id": "i-2cb9a45f", "ec2_image_id": "ami-a754dbce", "ec2_instanceState": "", "ec2_instance_type": "t1.micro", "ec2_ip_address": "54.234.59.5", "ec2_item": "", "ec2_kernel": "aki-825ea7eb", "ec2_key_name": "ec2HF", "ec2_launch_time": "2013-03-10T13:47:32.000Z", "ec2_monitored": false, "ec2_monitoring": "", "ec2_networkInterfaceSet": "", "ec2_persistent": false, "ec2_placement": "us-east-1a", "ec2_platform": "", "ec2_previous_state": "", "ec2_private_dns_name": "ip-10-243-150-74.ec2.internal", "ec2_private_ip_address": "10.243.150.74", "ec2_public_dns_name": "ec2-54-234-59-5.compute-1.amazonaws.com", "ec2_ramdisk": "", "ec2_reason": "", "ec2_region": "us-east-1", "ec2_requester_id": "", "ec2_root_device_name": "/dev/sda1", "ec2_root_device_type": "ebs", "ec2_security_group_ids": "sg-a854b3c3", "ec2_security_group_names": "ec2-webservers", "ec2_shutdown_state": "", "ec2_spot_instance_request_id": "", "ec2_state": "running", "ec2_state_code": 16, "ec2_state_reason": "", "ec2_subnet_id": "", "ec2_tenancy": "default", "ec2_virtualization_type": "paravirtual", "ec2_vpc_id": "" } Now let's use a regular playbook with the ec2.py script to get a list of all instances in the 'ec2-webservers' group and then use ansible's ping module on all of them: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers] ***************** GATHERING FACTS ***************** ok: [ec2-50-16-129-62.compute-1.amazonaws.com] ok: [ec2-54-235-232-227.compute-1.amazonaws.com] ok: [ec2-23-20-42-71.compute-1.amazonaws.com] ok: [ec2-23-20-87-176.compute-1.amazonaws.com] ok: [ec2-54-242-254-237.compute-1.amazonaws.com] ok: [ec2-54-234-59-5.compute-1.amazonaws.com] TASK: [ping] ***************** ok: [ec2-54-235-232-227.compute-1.amazonaws.com] ok: [ec2-50-16-129-62.compute-1.amazonaws.com] ok: [ec2-23-20-87-176.compute-1.amazonaws.com] ok: [ec2-23-20-42-71.compute-1.amazonaws.com] ok: [ec2-54-234-59-5.compute-1.amazonaws.com] ok: [ec2-54-242-254-237.compute-1.amazonaws.com] PLAY RECAP ***************** ec2-23-20-42-71.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-23-20-87-176.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-50-16-129-62.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-54-234-59-5.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-54-235-232-227.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-54-242-254-237.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 % Finally, clean up and use euca-terminate-instance to delete all instances: % euca-describe-instances \| grep INSTANCE \| awk '{print $2}' \| xargs -n 1 euca-terminate-instances INSTANCE i-60829f13 INSTANCE i-2cb9a45f INSTANCE i-7cb5a80f INSTANCE i-42b5a831 INSTANCE i-40b5a833 INSTANCE i-46b5a835 INSTANCE i-b2adb0c1 % euca-describe-instances RESERVATION r-bb3b6ac1 749335780469 ec2-webservers INSTANCE i-2cb9a45f ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:47:32.000Z us-east-1a \ aki-825ea7eb monitoring-disabled ebs RESERVATION r-8b3c6df1 749335780469 ec2-webservers INSTANCE i-7cb5a80f ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:50:48.000Z us-east-1a \ aki-825ea7eb monitoring-disabled ebs RESERVATION r-733f6e09 749335780469 ec2-webservers INSTANCE i-42b5a831 ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:50:54.000Z us-east-1a \ aki-825ea7eb monitoring-disabled ebs RESERVATION r-713f6e0b 749335780469 ec2-webservers INSTANCE i-40b5a833 ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:50:54.000Z us-east-1a \ aki-825ea7eb monitoring-disabled ebs RESERVATION r-773f6e0d 749335780469 ec2-webservers INSTANCE i-46b5a835 ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:50:54.000Z us-east-1a \ aki-825ea7eb monitoring-disabled ebs RESERVATION r-b72475cd 749335780469 ec2-webservers INSTANCE i-b2adb0c1 ami-a754dbce terminated ec2HF \ 0 t1.micro 2013-03-10T13:55:24.000Z us-east-1d \ aki-825ea7eb monitoring-disabled ebs % The terminated instances will be removed by EC2 eventually, and you can start all over. With the above steps and the previous work to use Ansible to setup a NetBSD system with basic configuration as database- and webserver the next step is to put those two things together, and get a (single) NetBSD machine into the Amazon cloud that serves as both database and webserver. Let's stay tune for this to happen! Shameless plug:** I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like! References: CapsUnlock blog post, CentOS Wiki. [Tags: ansible, ec2, xen]
[20130310]	Talking to the cloud After some more hacking, I have a basic understanding of how to start Amazon NetBSD EC2 instances using Ansible, fix the instances so they can be used as targets for further Ansible commands, and then actually talking to my herd of happy instances. Here's a teaser: Start EC2 instances, put them into ec2-webservers group. Repeat the following command for more than one instance: % ansible -i hosts-HF localhost -m ec2 -a 'image=ami-a754dbce instance_type=t1.micro \ key_name=eucaHF group=ec2-webservers' Prepare instances for Ansible (omitted - needs cleanup & automation) Use Ansible to ping all servers in the ec2-webservers group: % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers] ******************* GATHERING FACTS ***************** ok: [ec2-23-23-15-202.compute-1.amazonaws.com] ok: [ec2-54-235-230-206.compute-1.amazonaws.com] TASK: [ping] ***************** ok: [ec2-23-23-15-202.compute-1.amazonaws.com] ok: [ec2-54-235-230-206.compute-1.amazonaws.com] PLAY RECAP ******************* ec2-23-23-15-202.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 ec2-54-235-230-206.compute-1.amazonaws.com : ok=2 changed=0 unreachable=0 failed=0 [Tags: ansible, ec2, xen]
[20120420]	More dmesg pr0n: NetBDS/Xen with 128 (virtual) CPUs There was discussion about raising the number of CPU(core)s supported by NetBSD the other day, as the current limit of 32 isn't the sky any more in 2012. In the process, Xen-hacker Manuel Bouyer suggested using booting NetBSD ins a Xen DomU, as you can assign up to 128 (virtual) cores to a DomU. Here's the dmesg output, and I'm sure this is a lot faster than simulating 128 CPUs in qemu. So, how to go beyone 128 CPUs for testing? Anyone played with Qemu recently, or even have some decent hardware at hand? If so, be sure to post dmesg output (and CC: me)! [Tags: dmesg, xen]
[20120307]	NetBSD/xen available for Multi-Processor machines Manuel Bouyer announces that NetBSD/xen is now available for Multi-Processor machines. Citing from the release announcement: ``The NetBSD Foundation is pleased to announce completion of Multiprocessing Support for the port of its Open Source Operating System to the Xen hypervisor. The NetBSD Fundation started the Xen MP project 8 month ago; the goal was to add SMP support to NetBSD/Xen domU kernels. This project has officially completed, and after a few bug fixes in the pmap(9) code it is now considered stable on both i386 and amd64. NetBSD 6.0 will ship with option MULTIPROCESSOR enabled by default for Xen domU kernels. The availability of Xen MP support in NetBSD allows to run the NetBSD Open Source Operating Systems on a range of available infrastructure providers' systems. Amazon's Web Services with their Elastic Cloud Computing is a prominent examples here. Xen is a virtualization software that enables several independent operating system instances ("domains") to run concurrently on the same computer hardware. The hardware is managed by the first domain (dom0), and further guest/user domains (domU) are spawned and managed by dom0. Operating systems available for running as dom0 and domU guests include Microsoft Windows, Solaris and Linux besides NetBSD. NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license. NetBSD is developed and supported by a large and vivid international community. Many applications are readily available through pkgsrc, the NetBSD Packages Collection. NetBSD has been available for the Xen hypervisor since Xen 1 and NetBSD 2.0, released in 2004 , but until now only a single processor was supported in each NetBSD/xen domain.'' [Tags: amazon, ec2, smp, xen]
[20120205]	Automatic building of Amazon EC2 images from NetBSD NetBSD/xen is available for some time now to work on Amazon's Xen-based Elastic Cloud Computing (EC2) platform, as previously announced in the NetBSD blog, and the NetBSD wiki has instructions on how to subscribe to Amazon, launching and rebuilding the "AMI" images. This work is continued by Jean-Yves Migeon, who is working on build scripts for Amazon EC2, so the "AMI" images can be provided easily, with the eventual goal to include them into the NetBSD build process by Jeff Rizzo, so EC2 images can be automatically generated easily, e.g. by NetBSD's build cluster. [Tags: amazon, ec2, xen]
[20110126]	NetBSD/cloud - running NetBSD on Amazon's EC2 Citing the EC2 website, ``Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers. Amazon EC2's simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon's proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios. '' Internally, EC2 is based on Xen. And NetBSD also runs on Xen. So what's nearer than running NetBSD on EC2? Well, apparently it took some time and fiddling to get things together, but Alistair Crooks has posted a shell script on how to start working with NetBSD on Amazon's EC2 now. Note that pkgs for using EC2 are currently under development by Jean-Yves Migeon, and that there may be some manual fiddling required. Updated documentation (wiki, anyone?) are appreciated - who's first to post a dmesg(8) output to port-xen? :) [Tags: amazon, cloud, ec2, xen]
[20100908]	Funded project "Xen Paravirtualized Multi Processor Support", AKA what's missing to get full NetBSD support for Amazon's EC2 cloud Matthias Scheler writes on behalf of NetBSD's technical (core@) and administrative (board@) project leadership that ``the i386 and amd64 ports of NetBSD both offer extensive support for the Xen Hypervisor. This includes running a NetBSD kernel in paravirtualised (PV) mode as the dom0 (the virtual machine instance controlling the hardware of the host system) and domU (a normal virtual instance). Support for paravirtualization is mandatory for dom0 and greatly improves performance of a domU as no hardware emulation is required. Paravirtualization also allows using Xen on CPUs without hardware virtualization support. At the moment NetBSD doesn't support using multiple CPUs (or CPU cores) when it runs in paravirtualised mode. This is not only a severe performance limitation on modern x86 hardware it also makes NetBSD a suboptimal platform for use on Amazon EC2 service (which uses images of paravirtualized Xen domU systems). The Core team and the Board of the NetBSD project have created the attached specification for funded development to add Xen Paravirtualized Multi Processor Support to NetBSD. If you are interested in realizing this project send a proposal to core@ and board@, please.'' In case it's not clear: the project is open to everyone, not only to the current NetBSD developers community. All interested parties can submit project proposals! This new concept of "funded projects" can be considered similar to the projects student work on during Google's "Summer of Code", only that payment is done via The NetBSD Foundation, and with some other minor changes in rules. [Tags: amazon, ec2, xen, xmp]
[20100131]	Unfilling my inbox: NetBSD news from the past few weeks - ACPI, NUMA, Xen, and more Herre are some more things that I've caught in my inbox for too long, and I'm finally finding some time to sum them up here: NetBSD's "let's move kernel parts to the userland" RUMP project is still under heavy development, and in order to make testing of compatibility after kernel changes easier, a new command "rumptest" was added to build.sh: ``Basically you say: ./build.sh ${yourargs}�tools ; ./build.sh ${yourargs} rumptest Where yourargs are what have you, e.g. '-U -u -o -O /objs'. The latter builds only the rump kernel libs and uses some ld+awk magic to figure out if things go right or not. This is to avoid having to install headers and build libs (which is too slow since a full build is too slow). The magic is not a substitute for a full build, but it is n+1 times faster and works probably 99.9% of the time. The scheme uses a number of predefined component sets (e.g. tmpfs+vfs+rumpkern) to test linkage. They are currently listed in build.sh. This area probably needs some work in the future. It would be nice to autogenerate the combinations somehow. If things go well, you get something like this: ===> Rump build&link tests successful ===> build.sh ended: Wed Nov 18 20:10:59 EET 2009 '' See Antti's Antti's mail to tech-kern: on how to tell if things didn't go so well, and what to do in that case. According to Wikipedia, ``Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.'' Supporting NUMA in a contemporary (i.e.: Intel centric) SMP-enabled operating system requires following a bunch of standards, two of which are parsing of two tables, the System Resource Affinity Table (SRAT) and the System Locality Information Table (SLIT). Both tables are accessible via the Advanced Configuration and Power Interface (ACPI), and according to the German-language Wikipedia, the SRAT is used to assign local memory to local threads to boost their performance, and the SLIT defines the "distance" of the nodes among themselves, which is used to determine the "nearest" memory if local memory is not enough. Now, Christop Egger has posted patches to add an ACPI SLIT parser and an ACPI SRAT parser. See the two postings for dmesg pr0n from his tests on an 8-node system. Staying with ACPI and Christoph Egger, he found that even though the ACPI spec defines an ACPI device for fans, BIOS vendors and OEMs do their own thing. To accommodate things like the fan sensor found in the ACPI Thermal Zone in his HP Pavillion DV9700 laptop he has proposed a driver to extend the acpitz(4) driver with fan information. That way, envstat(8) can be used to display the ran's RPMs: [acpitz0] Processor Thermal Zone: 56.000 95.000 degC fan: 2840 RPM Staying with driver games, iMil writes me that there's documentation on getting DRI, AIGLX, Composite and Compiz going with NetBSD 5.0 available in the O(ther)NetBSD Wiki now. The documentation covers how to enable the Direct Rendering Manager (DRI), setting up and configuring Modular X.org, assuring that everything's in place, and how to get Compitz going. Mmm, wobbly windows at last! :-) While we're talking funky desktop stuff: Marc Balmer has submitted a patch to get touchpanel support for ums(4). ums(4) is for USB mice, and in contrast to mice, touch panels need to deal with absolute numbers, not relative numbers. Back to the guts of the kernel, another patch suggested by Christop Egger was for adding x2apic. What is x2apic? X2APIC is ``an Intel-only feature but can also be found in virtual environments with support for CPU apic id's > 0xff. I.e. Xen 4.0 (not yet released) supports 128 CPUs in HVM guests with the CPUs enumerated with even apic id's. That means you need x2apic for the 128th CPU :) '' While speaking of Xen: Xen 4.0 is coming soon, and there's a call to help testing it on NetBSD! Install Mercurial, check out latest Xen sources, apply a bunch of patches, build and install. Examples of commands are given, in addition to changes required for /boot.cfg etc. Report your findings to port-xen! Last one for today: Michal Gladecki, Editor-in-Chief of BSD Magazine writes: ``We are happy to announce that BSD Magazine is transforming into a free monthly online publication. The online version of BSD Magazine will stay in the same quality and form. It will look like the BSD magazine one is familiar and comfortable with. Please sign up to our newsletter at www.bsdmag.org and get every issue straight to your inbox. Also, you can now download any of the previous issues from our website. The first online issue -- 2/2010 -- is coming out in February. Please spread the word about BSD Magazine. '' Click! So much for today. I still have a bunch of news items in my inbox for next time, but let's call it good for today. Unrelated, I've been playing with git a bit over the past few days, and wile I have a number of questions building up (which will be subject to tech-repository or so), what I can say today is that the speed of "git pull" with NetBSD's git repository and my 1MBit DSL line reminds me a lot of the times when I used SUP with my 56k modem - it took forever, too. :-( [Tags: acpi, acpitz, aiglx, Articles, bsdmac, compiz, dmesg, dri, numa, rump, slit, touchpanel, ums, x2apic, xen]
[20090724]	Work in progress: ACLs, Xen Baloon Driver, GPIO, Raidframe detachment No big announcements in NetBSD land for a bit now, but there's lots of stuff brewing in moist dark places. I've assembled a few over the past few weeks, and I think it's time to mention them so they don't get lost: NetBSD implements traditional Unix file access control, which is based on permissions for reading, writing and executing for any of a file's owner, its group, and the rest of the world. Concepts in the form of Access Control Lists (ACLs) exist for more fine grained control, but they are not available with NetBSD. Yet. Elad Efrat is still workin on the kauth(9) framework, and as a side-product, he has implemented Access Control Lists on top of kauth(9).. The code is not fit for production use yet, but we can stay tuned to see more of this. When you have a machine running virtualization, you usually dedicate a portion of the machine's RAM to each of the VMs. You (usually) cannot spend more RAM for VMs than you have RAM in the host, obviously... until you use some sort of virtual memory for the VMs themseolves. Which is what the Xen "balloon" driver does, inflating a Xen VM's RAM as needed. Those interested in a driver can find a balloon driver for Xen3 dom0 by Cherry G. Mathew now, who's looking forward to your comments! Coming newly to NetBSD, developer Marc Balmer writes: ``NetBSD has had support for General Purpose Input/Output devices since the 4.0 release, when the GPIO framework from OpenBSD 3.6 was imported. Since the import of the GPIO framework into NetBSD, I have reworked larger parts of that subsystem in OpenBSD to address some problems and drawbacks''. More details on his motivation and details can be found on his homepage, and Marc has posted about his recent work on updating NetBSD's GPIO framework. See Marc's posting for details on changes in the API, prominent changes, security aspects and more. When you run a Xen DomU which has its file system on a vnd(4) disk and which has a number of disk images which are again put together into a raidframe(4) volume which may in turn contain further images for vnd(4), raidframe(4), cgd(4) and possibly others, tearing down the whole stack on system shutdown can get hairy. The situation is known, and David Young has put some work into this area. For now, he can properly detach raid units. See his posting for an example session. So much for the latest projects that are "work in progress" on the NetBSD front. Stay tuned for them to hit NetBSD-current! [Tags: acls, gpio, raidframe, xen]
[20090325]	Jibbed 5 rc3 - now with Xen! Zafer Aydogan's latest release candidat of his Jibbed Live CD now also offers booting a Xen3 Dom0 kernel from the CD, and allows starting further DomUs. Other news in RC3 are that packages can now be installed with pkg_add for from pkgsrc (enough RAM assumed :), and that sysinst plus official sets from NetBSD's FTP server are on the CD, so NetBSD can be installed on a harddisk from Jibbed, too. All this goes with the usual contents of the CD, i.e. a NetBSD based lice CD with a XFCE desktop. [Tags: jibbed, xen]
[20090309]	Catching up - various items (and not source-changes, this time) Many things have happened in NetBSD-land in the past few weeks, and as I've been slacking^Wbusy again, here's just a digest of things that I haven't seen mentioned elsewhere so far, in random order: BSD-related radio-show "bsdtalk" has published an interview with NetBSD's Andrew Doran in its March 2009 issue. Besides covering Andrews work, the upcoming NetBSD 5.0 release is also discussed. Available as mp3 and ogg. Cross-compiling pkgsrc packages is a long-standing dream, and it's yet waiting for someone to do it. For the time being, Jared McNeill has come up with an HowTo on how to build 32bit packages on amd64 (and probably other 64bit systems). Jared McNeill's been hacking on more stuff recently, and one thing includes changes to the framebuffer console support on x86 (i.e. both i386 and amd64). In short, the recent changes are just a stop on the way to move the splashscreen code and esp. image data from the kernel to userland. I.e. that you can put something like menu=Boot NetBSD:vesa 1280x800;splash /logo.bmp;boot netbsd into your /boot.conf in the future. But we'll see a separate announcement when that part is done. Let's stay tuned! :) Martti Kumparinen has tackled generating a UFS file system on a "large" (~5.5TB) disk. As the process is not straight forward, he has posted a howto that may help in the future. Any takers for adding comments and integrating this into The NetBSD Guide? :) Manpages are a major component of every Unix system. If you have ever tried to write such a manpage, you 'll have learned that they are in a funny text-based format similar to LaTeX and HTML, with its own processor - roff. There are several roff implementations, and the one used in NetBSD currently is the GNU implementation. To provide an alternative here is good for both removing GPL'd code from the NetBSD codebase, and also because groff is written in C++, which is slow to compile, and - well - requires a C++ compiler. A change for that situation may arise eventually, as Kristaps Dzonsons has been working on a groff replacement to format Unix manpages recently. See his posting and his homepage for further information. I've talked about Xen support for PCI passthrough recently, and Manuel Bouyer has finished his work to get full support for passing in access to specific PCI devices from the Xen Dom0 to DomUs. See his posting to port-xen for more details! Staying at Xen for a moment, David Brownlee has written instructions on Installing Windows XP in Xen under NetBSD. Just in case anyone needs to run a legacy system... :) The NetBSD operating system supports many different hardware and CPU platforms. For a specific platform, binaries are compiled with a specific compiler, and there is a set of binaries for each platform. This results in a rather big number of different sets of binaries - currently about 50. A different approach with historic precedence is to have one binary work on may hardware platforms, so-called "fat" binaries. Gregory McGarry has posted suggestions on how to modify NetBSD's toolchain to produce fat binaries. An interesting concept which would solve a number of problems (think: support, updates, pkgsrc!) Qt is a user-interface library found in widespread use in the Unix/Linux world. It's not exactly small, and its prerequirement of the X Window System doesn't it make a #1 choice for embedded systems at the first look. A Qt variant - Qt/Embedded - can be ran without X, though, and which thus avoids all the configuration and hardware support trouble of X in one go. On NetBSD, Qt/Embedded could talk to the wscons driver directly, and Valeriy 'uwe' Ushakov has posted about his work on patches to adopt Qt/Embedded to wscons. Who's first to post some screenshots? Enjoy! [Tags: 64bit, bsdtalk, fatbinary, groff, pkgsrc, qt, toolchain, vesa, windows, xen]
[20090215]	Xen news: Dom0 with PAE, PCI passthrough, migration I have outlined the need for PAE with Xen in a previous posting. In the recent past, PAE has become the default in Linux Xen kernels, and as a result, running Linux on a NetBSD-Dom0 was not possible due to lack of PAE in Dom0. This has changed now, Manuel Bouyer has added PAE support for NetBSD/Xen's Dom0 support with a XEN3PAE_DOM0 kernel config now. Manuel's posting to port-xen outlines the PAE work for Dom0. Other work that's underway is PCI passthrought: This feature that was available with Xen2 on NetBSD was changed in Xen3, and kernel support for DomUs to access specific devices on a PCI bus is underway. Currently, NetBSD DomUs need a Linux Dom0 to pass access to the PCI bus, but this is expected to change soon. First steps to pass single PCI devices from a NetBSD Dom0 to a DomU are made, and we can look forward for more of this to come. Again, see posting to port-xen for more details. The last item is just a quick heads-up in reply to Manuel's posting: Jean-Yges Migeon is currently working to get saving and restoring of DomUs as well as the associated migration of domains between several Dom0s -- i.e. to move a running DomU from one machine to another -- working. [Tags: pae, xen]
[20080909]	Booting Xen without grub So far, if you wanted to boot the Xen hypervisor, you had to use the GRUB bootloader. IIRC the reason for this was the different way of passing parameters to the kernel -- NetBSD's boot(8) doesn't use the "multiboot" protocol. This was added to the NetBSD kernel in order to boot NetBSD with grub, but that's not a help for Xen. As installing a different bootloader is suboptimal, possibly dangerous, and has (had?) issues on 64bit platforms, grub was not such a nice option. Now, Robert Swindells has finished making changes to NetBSD's native boot loader to allow it to load the Xen hypervisor by itself, without needing grub any more! This makes installation of NetBSD/Xen even easier: Just drop in a Xen-enabled kernel, add an entry to the boot.cfg file, and reboot. I can even envision that Xen could be installed in future versions of NetBSD/i386 and NetBSD/amd64 by default. Then all that's really needed would be to add the management tools after installing NetBSD. Any takers? :-) [Tags: grub, xen]
[20080610]	Yet another NetBSD/Xen guide I've been slacking on this one, but here it is: Michael Dexter has posted about his NetBSD Xen Guide. It describes setup of NetBSD/Xen, including the required packages and kernels, how to setup the GRUB bootloader, and how to configure the privileged Dom0 as well as an unprivileged domU. Further topics coviered include how a domU can be given special privileges, and a section for the adventuresome among us tells how FreeBSD and OpenBSD can be setup as domUs. [Tags: Docs, xen]
[20080208]	Creating an unprivileged Solaris domain (DomU) with NetBSD/Xen as Dom0 NetBSD/Xen works fine for both NetBSD as guest domain (domU) and as privileged domain (dom0). Those interested to run NetBSD as dom0 and Solaris as domU may be happy about Tobias Nygren's posting, which hints at his update to the NetBSD/Xen HowTo, which now also describes creating an unprivileged Solaris domain (DomU). [Tags: Docs, solaris, xen]
[20080131]	Article: Waving the flag: NetBSD developers speak about version 4.0 Federico Biancuzzi has collected interviews from more than twenty NetBSD developers in an multiple-page article which talks about what's new in the NetBSD 4.0 release: Introduction Release engineering, Sendmail, and kauth PaX, fileassoc, and Veriexec Linux compat, XFree86, pkgsrc, proplib, and Xen Filesystems iSCSI and optical disc support Bluetooth, mobile devices, agr Google Summer of Code Hackathons and funding If you have any comments, there's also a page for comments and discussion available. [Tags: Articles, bluetooth, google-soc, hackathon, iscsi, kauth, pkgsrc, Release, udf, veriexec, xen]
[20080124]	NetBSD Xen/i386 got PAE support (and why this is good) A 32bit CPU can normally address 2^32 bytes, which is about 4GB RAM. As that's not enough for everyone any more today, one possible solution is to move to more address bits, which is what 64bit CPUs are for. Another alternative is to add some fancy magic to do paging/banking, which 16-bit Intel CPUs used "back then" when Protected Mode was still a topic. Today, address space beyond 4GB RAM can be addressed using the Physical Address Extension (PAE) on Intel (and compatible) CPUs. So far, NetBSD never got PAE-support, as its amd64-port worked early and very well, so people with demand for more than 4GB ram usually moved to an appropriate hardware platform, and called it good. One case where PAE support was an issue is Xen, though: Linux usually has PAE enabled these days with Xen3 Dom0s, and DomUs running on a PAE-enabled Dom0 also have to support PAE. As a consequence, it was not easily possible to use NetBSD DomUs with contemporary Linux/Xen distributions. Thanks to long-term NetBSD/Xen-hacker Manuel Bouyer, this has changed now: Manuel has added support for PAE to the NetBSD/Xen port, and he has merged the corresponding CVS branch to NetBSD-current now. There are separate kernels, XEN3PAE_DOMU and INSTALL_XEN3PAE_DOMU, which have PAE-support enabled, and which are expected to show up in daily builds soonish. While this work does not add PAE support for NetBSD/i386 in general, it brings it a big step forward, and it may just be a matter of time until NetBSD/i386 also gets PAE support. Implementation details can be found in src/sys/arch/i386/include/pte.h and src/sys/arch/i386/include/pmap.h, btw -- see Manuel's mails to port-xen for more information. [Tags: pae, xen]
[20071118]	Xen/amd64 update Manuel has posted an update on his work on the bouyer-xenamd64 branch: ``a amd64 domU kernel runs stable, and a dom0 kernel boots and can start a domU''. Manuel also outlined his future plans: ``Once bouyer-xenamd64 is merged, I'll start a new branch to work on switching xeni386 to the new x86 pmap and xenamd64 bootstrap codes, and merge back code from xen/i386 to i386/i386. I'll also look at x86pae support in Xen, as using the x86 pmap should make this much easier.'' Esp. the bit about PAE is very desirable, as it currently stops NetBSD from working as domU with a Linux dom0 that has PAE enabled (which is the default nowadays). Every Linux user that wants to try NetBSD in a Xen domU currently has to make sure he has a non-PAE kernel first, which is suboptimal. [Tags: amd64, xen]
[20071022]	Status: NetBSD on Xen/amd64 Manuel Bouyer has worked on making NetBSD working on Xen on the amd64 platform, and he has it has made some substantial progress, see his status mail. A dmesg output is also available. [Tags: amd64, dmesg, xen]
[20070508]	Screenshots: Xen with HVM and Windows In case you always wondered what NetBSD with Xen and hardware virtualization (HVM) looks like, check out these pictures showing installation of Windows XP and Windows 2003 on a system running a NetBSD 4.0_BETA2 Dom0 with Xen 3.0.4. [Tags: images, xen]
[20070405]	Setting up Linux/Xen with a NetBSD Dom0 Harold Gutch dropped me a note that he wrote some documentation on how to setup a Debian/Linux DomU with a NetBSD Dom0 setup. [Tags: Docs, xen]
[20070218]	Hosting: Xen and 'regular' The question of NetBSD/Xen hosting came up again, and a list of 'regular' and Xen provieders was posted that I think may be of interest at some point. (mmm, abusing my blog as knowledge base - sorry! :-) [Tags: hosting, xen]
[20070103]	Article: An overview of virtualization methods, architectures, and implementations There's an article actually titles Virtual Linux over at IBM's developerworks, that has "An overview of virtualization methods, architectures, and implementations" as subtitle: ``Virtualization means many things to many people. A big focus of virtualization currently is server virtualization, or the hosting of multiple independent operating systems on a single host computer. This article explores the ideas behind virtualization and then discusses some of the many ways to implement virtualization. We also look at some of the other virtualization technologies out there, such as operating system virtualization on Linux.'' Too much focus on Linux and too little focus on NetBSD, but it may still serve as useful material when trying to tell your boss/buddy/whatever what Xen is and why it is yet another reason for going NetBSD. (While there: anyone up for providing patches to bring the NetBSD/Xen HowTo up to speed for Xen3 from the start? Send your patches to www [at] NetBSD.org!) [Tags: Articles, xen]
[20061109]	More NetBSD/Xen hosting: GPLhost.com and Panix.com OK, another round of free advertizement, from the "companies hosting NetBSD/Xen" department: If you're in need of a hosted NetBSD/Xen domain (that's 'domain' as in Xen, not as in DNS), there are a bunch of solutions today, and the most recent ones I came across are the NetBSD/Xen hosting at GPLhost.com as well as Panix' V-Colo Virtual Colocated Servers. Just for the record (and to hit the spot of this blog :-): Panix, who is one of the oldes ISPs in business ever, has a longstanding relationship with NetBSD: They use NetBSD for many of their services, and current and former Panix employees are active NetBSD developers. [Tags: hosting, xen]
[20061020]	Xen with HVM and hardware virtualization: Booting Windows XP in a DomU Manuel Bouyer has continued his work on the NetBSD/Xen port, and his latest results include support to use the Hardware Virtual Machine (HVM) interface for Intel and AMD CPUs that support virtualization as well as the Qemu Device Model (Qemu-dm), which is a modified Qemu running in Dom0 that is used to emulate some hardware devices, see Manuel's first posting to the port-xen list and the thread starting from it. In related news, Manuel has updated the packages for the latest Xen release (3.0.3) and a new one to support HVM (which needs X, in order to emulate a VGA console via Qemu, see above). With that package, it was possible to boot systems that were not specifically prepared for use with Xen, e.g. NetBSD/i386, Linux and Windows XP! See Manuel's second posting and the followups. [Tags: qemu, xen]
[20061011]	More NetBSD/xen vhosting in Germany: x\|encon (Updated) x\|encon is a hosting provider located in Hannover, Germany. Besides offering Linux vServer based hosting, they also offer Xen hosting, and their x\|bsd prominently mentions NetBSD as the base for their products. They offer two different configurations with 20/40GB harddisk and 325/650MB RAM for a monthly fee for 29/49EUR and no setup fees. The contract includes flat billing of network traffic, dynamically allocated swap space (probably swap file instead of a dedicated swap partition), a public IP address and of course full root access. (And no, I'm not sponsored by them; Rainer Brinkmöller pointed me at them) Update: The x\|encon Support Team told me that what they offer is a plain blockdevice which can be setup wit a NetBSD XEN3_DOMU_INSTALL kernel then to everyones' likes. [Tags: hosting, xen]
[20060928]	Xen3 progress: acpi/mpbios/ioapic support NetBSD/Xen-maintainer Manuel Bouyer has continued his fine work to make NetBSD work with Xen. After Xen2 DomU and Dom0 and Xen3 DomU and Dom0 support, he is now working on support for running Xen3 on SMP hardware, with all the related changes to ACPI, MPBIOS and IOAPIC. People running Xen3 with Dom0 and DomU on NetBSD will have to adjust their kernel configs after this commit, changing cpu* at mainbus? to vcpu* at hypervisor? See Manuel's mail for more information. [Tags: xen]
[20060703]	Xen3 domain0 support is there! I was mostly idle and offline the past few days, but not everyone's been slacking: Manuel Bouyer has continued his work on Xen3 domain0 support for NetBSD, and he has announced success now: ``I'm proud to announce that, with today's commit NetBSD has finally usable Xen3 domain0 support. I have a NetBSD-current domU, acting as a NFS server for a linux domU doing a build.sh release with sources on NFS and obj on local storage, all running on NetBSD-current domain0 on Xen3!'' See Manuel's mail for all the details. [Tags: xen]
[20060508]	Xen3/NetBSD dom0 progress Manuel Bouyer has continues his work on NetBSD/Xen, and after Xen3 domU support, Xen3 dom0 support is coming: Manuel was able to boot a NetBSD domU from a NetBSD dom0, and he will continue working on various outstanding parts like block (disk) and network backend drivers. See his mail to the port-xen list for more information! [Tags: xen]
[20060428]	Overhead of vnd(4) as disk device in Xen When you run a Xen domU, the question is what to use as a harddisk to run the operating system from. Options are a dedicated harddisk partition, or using a vnd(4) disk-image. While disk-images are a lot more flexible the manage than partitions, there's a drawback because access to anything on the image goes through the dom0's filesystem, and is thus slower. How much slower exactly using a disk image instead of a raw partition is was asked on tech-perform, and Greg Troxel gave some interesting numbers. Now what I wonder is what impact the filesystem type has... [Tags: benchmark, xen]
[20060427]	dmesgs: NetBSD/Xen in qemu Being a qemu-whore^W^W^Wlazy, I wanted to play with Xen, but never found the hardware to do so. Once again, qemu came to the rescue, and following the fine NetBSD/Xen howto, I managed to setup Xen in qemu. Setup and configuration was dead easy, and NetBSD comes with some excellent infrastructure to setup a machine that starts up multiple domUs automatically, by simply adding the needed config files into /usr/pkg/etc/xen. The qemu disk image is 1GB in size so I'll not make this available (but can upload it on request, if someone wants?), but for kicks here are dmesg outputs of the host running qemu, the Xen dom0 running inside qemu and a Xen domU domain. Harddisk usage of the 1GB disk is, in dom0: two 180MB disk images for the domU filesystems, mounted via vnd(4). About 100MB of additional packages are installed to manage Xen plus some other things pulled in to support that (Python, Perl and lots of modules), 100MB for X, some 200MB for a full installation of NetBSD 3.0/i386 (used on the Xen kernel) which includes development and text processing environment, documentation and manpages. The rest of the disk is dedicated to swap. The system is setup to use grub as bootloader, which offers booting either a 'regular' NetBSD/i386 kernel (i.e. no Xen), or the Xen hypervison, which then boots a NetBSD/Xen kernel, that uses the NetBSD/i386 userland to boot. After the system has booted to multiuser mode, started the two domUs, and after logging in as root, the domU consoles can be accessed by telnetting to localhost port 9601 and 9602, respectively. Networking for the domUs is setup in the domU config files: all domUs, the dom0 plus the physical ethernet interface are all plugged into a (virtual) switch (implemented via bridge(4)), which is then bridged to the "normal" ethernet - Voila, network for all domains! FWIW, here's what a Xen domU config file looks: $ cat /usr/pkg/etc/xen/hf1 kernel="/netbsd-XENU" memory=32 name="hf1" nics=1 vif = [ 'mac=52:54:00:12:34:57, bridge=bridge0' ] disk = [ 'file:/harddisk.xen-hf1,wd0d,w' ] root="/dev/wd0d" Installation of a Xen domU with NetBSD works by creating a harddisk image, and then using the INSTALL_XENU kernel, which boots right into an installer that can then be used to install NetBSD on the disk(image). Installation sets can be fetched using the local network e.g. via FTP from dom0. Of course after setting up one domU harddisk image, setting up the other one is a mere "cp img1 img2", with some small changes for hostname and SSH keys etc. In summary, I'm very impressed by the "roundness" of the Xen integration into NetBSD - no hacking, just add config files, disk images, and off you go. Mmm, NetBSD! [Tags: dmesg, qemu, xen]
[20060419]	Xen Virtual Private Server Hosting Here's yet another Xen hosting provider, Xtrahost.net from the UK, lists NetBSD on the "Distros Available Soon" list... (Distro? What distro?!?) [Tags: hosting, xen]
[20060411]	Xen-3 support pulled up to netbsd-3 Manuel Bouyer writes on the port-xen list that `` the xen-3 domU support has been pulled up to the netbsd-3 branch. This means that it's now possible to run the stable branch on a xen-3 system (you'll have to use current or wait for netbsd-4 for dom0 support though). As in current, 3 xen-3 kernels will be build as part of the release: INSTALL_XEN3_U (to install netbsd on a domU) and XEN3_U (to use with a populated image disk, or root on NFS).''. This should serve as hint as what to put into a kernel config file to get Xen3 domU support. [Tags: xen]
[20060405]	Xen presentation (german language) Ulrich Habel gave a presentation about Xen and NetBSD at CeBit 2006. He has made his presentation available now. Available are two PDFs, a screen version with navigation and a print version for handout. [Tags: presentations, xen]
[20060320]	NetBSD/xen: Xen3 domU functionnal (Updated) Manuel Bouyer has continued his work on getting NetBSD to work on Version 3 of the Xen virtual machine monitor over the past few weeks, and he writes that ``NetBSD should now be functionnal on Xen3 with block and network devices''. See Manuel's mail to port-xen for more details, and check out the NetBSD/xen Howto to get started with Xen on NetBSD. (While the HowTo is still for Xen2, updating to Xen3 merely means replacing the Xen and NetBSD kernels) Update: Please note that Xen 3 support is for DomU only right now, Dom0 support is being worked on. Stay tuned! [Tags: xen]
[20060312]	Xen Forum I've stumbled across Rainer Brinkmöller's Xen-Forum, which is available in both german and english. Mmm, community building! [Tags: xen]
[20060307]	Article: Virtualization in Xen 3.0 Not mentioning NetBSD in particular, but this article by Rami Rosen titled "Virtualization in Xen 3.0" gives some details on how Xen 3.0 works, including CPU privileges with "paravirtualization", virtual split drivers, Xen and the new intel VT-x processors, live migration and more. [Tags: Articles, xen]

Previous 36 entries

Tags: , 2bsd, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, backup, banners, basename, bash, bc, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, callweaver, camera, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, information, init, initrd, install, intel, interix, internet2, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, java, javascript, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kitt, kmod, kolab, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mobile-ip, modula3, modules, mouse, mp3, mpls, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nat, ncode, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opengrok, openmoko, openoffice, openpam, opensolaris, openssl, oracle, oreilly, oscon, osf1, osjb, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrcCon, pkgsrccon, plathome, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, Products, products, proplib, protectdrive, proxy, ps, ps3, psp, pthread, ptp, ptyfs, Publications, puffs, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs6k, rss, ruby, rump, rzip, sa, safenet, san, savin, sbsd, scampi, scheduling, sco, screen, script, sdf, sdtemp, secmodel, Security, security, sed, segvguard, seil, sendmail, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slit, smbus, smp, sockstat, soekris, softdep, software, solaris, sony, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, squid, ssh, sshfs, ssp, stereostream, stickers, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, tso, ttyrec, tulip, tun, tuning, uboot, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, x.org, x11, x2apic, xbox, xcast, xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 14348230
Copyright (c) Hubert Feyrer