hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20130813] onetbsd.org is fully back
www.onetbsd.org was started as an experiment to bring alternative, community-provided NetBSD content. Technology-wise it does this through RSS feed aggregation ran by Kimmo Suominen on his machines, the domain is registered and DNS provided by David Brownlee, and contents come from a variety of NetBSD-related blogs' RSS feeds, e.g. mine.

A couple of weeks ago, the DNS service was moved to a different DNS provider, which led to some problems: The IP adresses for the authoritative DNS servers of the onetbsd.org zone were only IPv6 addresses, no IPv4 addresses. Those (IPv6 enabled) DNS servers did provide proper IPv4 (A) and IPv6 (AAAA) address records, but only to the clients speaking v6 in the first place. This led to funny effects that www.onetbsd.org worked when on IPv6-enabled networks (which also had v4 enabled), but not on IPv4-only networks. This was further aggravated by the holiday season with not everyone reachable, a longer-than-neccessary communication chain and no direct access to all systems directly.

In the end things are changed back to working now for both IPv4-only and IPv6-enabled networks again, so be sure to keep watching www.onetbsd.org.

(And for those wondering where the domain name comes from: it's from the kernel that doesn't lie :)

[Tags: ]



[20130723] NetBSD on BeagleBone Black HOW-TO
NetBSD runs on a number of ARM platforms, and the BeagleBone Black is one of many such platforms. It comes with Linux by default, and as such there are a few adventures to make if you want NetBSD on it.

John Klos was brave, and has collected his experiences in a "NetBSD on BeagleBone Black HOW-TO". Please note that there's an important update that's needed to not nuke your MBR.

So, anyone got some cool toys they make with a BeagleBone Black and NetBSD? Let me know!

[Tags: , , ]



[20130712] Spanish translation of my "Managing NetBSD with Ansible" article
Maria Ramos from Webhostinghub.com offered to translate my "Managing NetBSD with Ansible" blog post for the spanish-speaking WebHostingHub community.

Due to this, a spanish translation of my article is available now. Thanks Maria!

[Tags: , ]



[20130528] NetBSD's projects for Google's Summer of Code 2013 have been chosen
In this year's round of Google Summer of Code, we have again received a number of project proposals by interested students. After going into details and finding out what projects have a chance to give both the NetBSD project and the student the most benefit (besides the money), this is settled now, and I'm pleased to announce that the following students and their projects are our prospects for 2013:
  • Julian Fagir: System upgrade (system_upgrade)
  • Haomai Wang: Make NetBSD a supported guest OS under VirtualBox (Virtualbox)
  • Manuel Wiesinger: Defragmentation for FFS in NetBSD (defrag_ffs)
  • Myron Aub: Port Linux's drm/kms/gem/i915 (DRM2)
  • Przemyslaw Sierocinski: Implement file system flags to scrub data blocks before deletion (fs_scrub_flags)
Of course all other students who have submitted proposals can feel free to participate in NetBSD outside of the Summer of Code. For those participating, this is the time for the timeline.

[Tags: ]


[20130520] NetBSD 6.1 and 6.0.2 released
Following NetBSD's release scheme, two new releases are available now. NetBSD 6.1 is the next release from the netbsd-6 release, and it contains security fixes, bug fixes and some new feature. NetBSD 6.0.2 is the second stability update for NetBSD 6.0, and it also contains bugfixes and security fixes, but no new features. See the release map graph on the NetBSD website for a visual representation of the relationship between releases. Major news in 6.1 include:
  • Security: prevent kernel panics via userland requests from kqueue, a random number generator update to prevent weak cryptographic keys and a vulnerability in grep.
  • Networking: many updates to NetBSD's new packet filter npf, and improved SMP operations.
  • Embedded: Raspberry Pi now has working USB and ethernet, support for the watchdog timer in some Marvell SoCs, fixes to the Kirkwood IRQ code
  • Platforms: device driver for Hydra and ASDG Zorro2 bus network cards on Amiga, x68k's bootloader can now boot from CD and network, and dtrace support on amd64.
  • Drivers: add LSI Thunderbolt (SAS2208) controllers, Apple's Thunderbolt to Gigabit Ethernet adapter, and improve stability with multiple concurrent file system snapshots.
... plus numerous bugfixes. For more details see the release notes of NetBSD 6.1 and NetBSD 6.0.2. NetBSD is a volunteer project ran by a non-profit organization and with no commercial backing. As such, your donations are very important to the project, and can fund developing in various areas, including:
  • Improving network stack concurrency and performance.
  • Development of modern file systems and improvement of existing ones.
  • Features which are useful in embedded environments, such as high resolution timers and execute in place (XIP) support.
  • Automatic testing and quality assurance.
For more information about donating, visit http://www.NetBSD.org/donations/ The NetBSD Foundation is a 501(c)(3) organization in the US, and donations may be tax deductible.

[Tags: ]


[20130409] NetBSD is part of Google's Summer of Code 2013
News is out that NetBSD is part of Google's Summer of Code 2013 (GSoC) again. GSoC is about students doing work for Open Source projects over the summer, and getting paid while doing so. By Google. For projects proposed by both students and the Open Source projects.

Click on the above link for more information on GSoC in general, there is also a list of proposed projects for this year in NetBSD.

Next steps are:

  • April 9 - 21: Would-be student participants discuss application ideas with mentoring organizations.
  • April 22, 19:00 UTC: Student application period opens.
  • May 3, 19:00 UTC: Student application deadline.
  • Interim Period: Mentoring organizations review and rank student proposals; where necessary, mentoring organizations may request further proposal detail from the student applicant.
  • May 6: Mentoring organizations should have requested slots via their profile in Melange by this point.
  • May 8: Slot allocations published to mentoring organizations From there, students work on their projects with the help of their mentors. There's a "midterm" report due with a first part of the money paid, the rest is paid if the project is finished successfully.

    During the project, students are encouraged to publish news about their process to the world in blogs and other ways found appropriate by their mentoring organizations.

    Past NetBSD projects can be found on SourceForce.

    Interested? Act now!

    [Tags: ]



  • [20130324] Ansible & EC2 - Playbooks for orchestrating NetBSD into the cloud
    As follower of my blog you have seen the steps towards getting NetBSD instances started in Amazon's EC2 cloud with a simple web application deployed on one EC2 instance and the database on another one.

    These blog articles were very detailed on purpose, to have full logfiles available just in case needed. I have used these logs to prepare my pkgsrcCon 2013 talk about Ansible and Amazon's EC2, so things can be looked at without actually running anything. As it turns out this was good, because the 32bit NetBSD instances that I've used during my pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation was a bit more on the theoretical side than I originally planned.

    Now after pkgsrcCon is over, I would like to publish the presentation slides with all the details, and especially the playbooks and all other files to look at - enjoy!

    [Tags: , , , , , ]



    [20130321] Ansible, EC2 and NetBSD final milestone 4 reached: Web and DB on separate VMs in the cloud
    In the fourth and last step on my journey to use Ansible to bring a non-trivial system of a Web server and a DB server into Amazon's EC2 cloud, this is the final step. After starting out with a local VMware VM and making first steps with Ansible and EC2, the previous step was to push a single system into the cloud. Now, the final step is to setup two distinct VMs, one for the database and one for the webserver, and then make them known to each other.

    The single steps are:

    1. Prepare the two VMs
    2. Basic setup for all systems
    3. Install the database server
    4. Install the webserver
    5. Connect database and webserver
    Again, here are all the steps in detail:
    1. As before, ensure local time is correct when talking to Amazon, and also make sure the SSH agent has the proper key loaded.
      % date
      Thu Mar 21 00:45:37 CET 2013
      % ssh-add -l
      2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
      
    2. Make sure security groups are setup properly. We use one group for the database server, and one for the webserver. This defines the access permissions from the internet, and also allows to identify systems for their individual configuration and also for connecting them in the final step:
      % euca-describe-groups
      ...
      GROUP   sg-ae54b3c5     749335780469    ec2-dbservers   Database servers
      PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-dbservers   ALLOWS  tcp     3306    3306    FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-dbservers   ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
      GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
      PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
      
    3. Now, run our playbook to setup the two VMs. This uses the single playbook from the previous milestone, and just runs it twice with different security groups:
      % ansible-playbook -i hosts-HF config-ec2-prepare-db+web-vm.yml
      
      PLAY [localhost] ********************* 
      
      TASK: [ec2-webservers | Launch new EC2 instance] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Give the system 30 seconds to boot up] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
      changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
      
      TASK: [ec2-webservers | Install pkgin via /usr/bootstrap.sh] ********************* 
      changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
      
      TASK: [ec2-webservers | Copy over Ansible binary package] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Install Ansible dependencies] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Install Ansible package (manually)] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-webservers | Setup lame /usr/bin/python symlink] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Launch new EC2 instance] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Give the system 30 seconds to boot up] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
      changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
      
      TASK: [ec2-dbservers | Install pkgin via /usr/bootstrap.sh] ********************* 
      changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
      
      TASK: [ec2-dbservers | Copy over Ansible binary package] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Install Ansible dependencies] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Install Ansible package (manually)] ********************* 
      changed: [127.0.0.1]
      
      TASK: [ec2-dbservers | Setup lame /usr/bin/python symlink] ********************* 
      changed: [127.0.0.1]
      
      PLAY RECAP ********************* 
      127.0.0.1                      : ok=18   changed=18   unreachable=0    failed=0    
      
    4. Just to make sure, check that the two instances run properly, and are in the right security groups, ec2-webservers and ec2-dbservers:
      % euca-describe-instances
      RESERVATION     r-a419f9d9      749335780469    ec2-webservers
      INSTANCE        i-21b7c441      ami-5d0f8034    ...
      RESERVATION     r-641efe19      749335780469    ec2-dbservers
      INSTANCE        i-54a2ab3e      ami-5d0f8034    ...
      
    5. Next, bring the two freshly setup systems (which are already capable of acting as ansible targets) up to our basic system setup:
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml
      
      PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 
      
      TASK: [ping] ********************* 
      ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
      ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Install tcsh] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Add user feyrer] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Create ~feyrer/.ssh directory] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable ssh login with ssh-key] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Install sudo] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Disable ssh logins as root] ********************* 
      ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
      ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-54-234-139-151.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
      ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
      
    6. Check:
      % ssh ec2-54-234-139-151.compute-1.amazonaws.com id
      uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
      % 
      % ssh ec2-54-235-44-118.compute-1.amazonaws.com id
      uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
      
    7. Now that the two machines run with our basline configuration, install their individual software and settings. First the database server:
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml
      
      PLAY [security_group_ec2-dbservers] ********************* 
      
      TASK: [Install mysql] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Install MySQL rc.d script] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Start MySQL service] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Setup DB] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Add db-user] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Copy over DB template] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Import DB data] ********************* 
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-54-235-44-118.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0    
      
      
      
    8. Check and see if the database works as expected:
      % ssh -t ec2-54-235-44-118.compute-1.amazonaws.com mysql -u webapp -p webapp
      Enter password: ****
      ...
      mysql> show tables;
      +------------------+
      | Tables_in_webapp |
      +------------------+
      | names            |
      +------------------+
      1 row in set (0.01 sec)
      
      mysql> select * from names;
      +----+--------+------+
      | id | first  | last |
      +----+--------+------+
      |  1 | Donald | Duck |
      |  2 | Daisy  | Duck |
      +----+--------+------+
      2 rows in set (0.00 sec)
      
      mysql> bye
      
    9. Excellent. Now setup the webserver, too:
        
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml
      
      PLAY [security_group_ec2-webservers] ********************* 
      
      TASK: [Installing ap24-php53 package and dependencies] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Install Apache rc.d script] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable and start Apache service] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable PHP in Apache config file] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})
      
      TASK: [Make Apache read index.php] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Install phpmyadmin] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable phpmyadmin in Apache config] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Fix Apache access control for phpmyadmin] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Enable PHP modules in PHP config file] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})
      
      TASK: [Create directory for webapp] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Deploy example webapp] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      TASK: [Create webapp symlink for easy access] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      NOTIFIED: [restart apache] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-54-234-139-151.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0    
      
    10. Again, test:
      % links -dump ec2-54-234-139-151.compute-1.amazonaws.com/
                                         It works!
      %
      % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/phptest.php | head
         PHP Logo                                                                   
                                                                                    
                                     PHP Version 5.3.17                             
      
         System          NetBSD ip-10-80-61-33.ec2.internal 6.0.1 NetBSD 6.0.1      
                         (XEN3PAE_DOMU) i386                                        
         Build Date      Dec 14 2012 10:31:13                                       
                         './configure' '--with-config-file-path=/usr/pkg/etc'       
                         '--with-config-file-scan-dir=/usr/pkg/etc/php.d'           
                         '--sysconfdir=/usr/pkg/etc' '--localstatedir=/var'         
      % 
      % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
         Showing table hf.names:
      
         Cannot connect to database: Can't connect to local MySQL server through
         socket '/tmp/mysql.sock' (2)(2002)
      
    11. Close to optimum, but the last error is actually expectet: In order for proper operation, the Database needs to grant the webserver access, and the web server needs to know where the database server is. So let's connect them!

      This step is done by preparing a shell script on both systems, which will then be ran to - depending on the system's security group - perform the proper steps:

      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-connections.yml
      
      PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* 
      
      TASK: [Collect EC2 host information] ********************* 
      ok: [ec2-54-234-139-151.compute-1.amazonaws.com]
      ok: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Prepare connection-script in /tmp/do-connect-vms.sh] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      TASK: [Run connection-script] ********************* 
      changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
      changed: [ec2-54-235-44-118.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-54-234-139-151.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    
      ec2-54-235-44-118.compute-1.amazonaws.com : ok=3    changed=2    unreachable=0    failed=0    
      
    12. With that final step, our test web application works, and the webserver can access the database properly:
      % links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
         Showing table hf.names:
      
         +--------------------+
         | id | first  | last |
         |----+--------+------|
         | 1  | Donald | Duck |
         |----+--------+------|
         | 2  | Daisy  | Duck |
         +--------------------+
      
           ----------------------------------------------------------------------
      
         Enter new values:
      
         first:     _____________________ 
         last:      _____________________ 
         [ Submit ] 
      
    So much for this exercise. I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like, or stay tuned to find my presentation and all the data after pkgsrcCon 2013.

    [Tags: , , , ]


    [20130321] Happy 20th Birthday, NetBSD! (Update)

    20 years back from today, NetBSD was initially checked into CVS. Revision 1.1 of src/Makefile was committed on March 21st 1993 on 09:45:37 by Chris Demetriou (cgd@):

    % cvs log -Nr1.1 Makefile
    ...
    revision 1.1
    date: 1993/03/21 09:45:37;  author: cgd;  state: Exp;
    branches:  1.1.1;
    Initial revision 
    NetBSD was started as successor to the Berkeley System Distribution (BSD) Unix with a focus on multiplatform support.

    Personally, I've followed NetBSD since the day in 1993 when the Amiga port popped up, which was the first platform that the newly forked operating system was ported to after its separation from BSD.

    Many things have happened in the past 20 years, and a lot could be shown and told for the history books at this point. But I guess that can be done later - I'd be happy to help out with such a project if someone wants to start it, though :)

    For today I'm very happy that NetBSD is available on a wide range on platforms, runs the software that I want and gives me the assurrance it will be around tomorrow and hopefully for the next 20 years, too.

    Cheers, NetBSD!

    Update: Jeremy Reed pointed me at his BSDnewletter posting, which gives a number of details of NetBSD's history. Recommended reading!

    [Tags: , ]



    [20130321] Ansible, EC2 and NetBSD milestone 3 reached: Web and database in the cloud
    With the previous work on setting up a local VM as database and web server and setting up a Xen VM in Amazon's EC2 cloud combined, it is pretty straight forward to setup a EC2 instance that has all the software to serve a simple web application from the cloud.

    The single steps are:

    1. Prepare the environment with proper time, SSH agent and EC2 firewall groups
    2. Setup EC2 instance with pkgin and ansible
    3. Do basic preparations to meet our standards for logins, shells and general usability and security
    4. Setup database server with DB software, user and import of data
    5. Setup web server with all the software and some demo application
    The following details show all the commands can and their output in more detail:

    1. Make sure time is set properly - needed when talking to Amazon EC2:
      % sudo sh /etc/rc.d/ntpd stop
      ntpd not running? (check /var/run/ntpd.pid).
      % sudo sh /etc/rc.d/ntpdate restart
      Setting date via ntp.
      % sudo sh /etc/rc.d/ntpd start
      Starting ntpd.
      % date
      Sat Mar 16 16:46:19 CET 2013
      
    2. Teach our EC2 SSH key to SSH agent, so we don't have to type a password (which we don't know anyways - EC2 only works with SSH keys):
      % ssh-add -l
      Could not open a connection to your authentication agent.
      % 
      % eval `ssh-agent`
      Agent pid 10467
      % ssh-add -l
      The agent has no identities.
      % ssh-add ../../euca2ools/key-eucaHF.pem
      Identity added: ../../euca2ools/key-eucaHF.pem (../../euca2ools/key-eucaHF.pem)
      % ssh-add -l
      2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
      
    3. Check security (firewall) groups - those are stored in EC2, and we have previously set them up:
      % euca-describe-groups
      ...
      GROUP   sg-a854b3c3     749335780469    ec2-webservers  Web servers
      PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-webservers  ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
      PERMISSION      749335780469    ec2-webservers  ALLOWS  icmp    -1      -1      FROM    CIDR    0.0.0.0/0
      
    4. See if there are any EC2 instances running:
      % euca-describe-instances
      %
      
      No - that's fine, we are about to change that!

    5. Run first playbook to launch EC2 instance and prepare it for using with ansible:
      % ansible-playbook -i hosts-HF config-ec2-prepare1vm.yml
      
      PLAY [localhost] ********************* 
      
      TASK: [Launch new EC2 instance] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Give the system 30 seconds to boot up] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Fix /usr/bootstrap.sh to run pkgin with -y] ********************* 
      changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
      changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
      
      TASK: [Install pkgin via /usr/bootstrap.sh] ********************* 
      changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
      
      TASK: [Copy over Ansible binary package] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Install Ansible dependencies] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Install Ansible package (manually)] ********************* 
      changed: [127.0.0.1]
      
      TASK: [Setup lame /usr/bin/python symlink] ********************* 
      changed: [127.0.0.1]
      
      PLAY RECAP ********************* 
      127.0.0.1                      : ok=9    changed=9    unreachable=0    failed=0    
      
      We now have a EC2 instance running that has Ansible installed:
      % euca-describe-instances
      RESERVATION     r-d77272ad      749335780469    ec2-webservers
      INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com ...
      
    6. With this EC2 instance, we can do some basic preparations for our standards, e.g. a login without requiring root (and while there, actually disable allowing as root), setup sudo and a proper shell:
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml
      
      PLAY [security_group_ec2-webservers] ********************* 
      
      TASK: [ping] ********************* 
      ok: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install tcsh] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Add user feyrer] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Create ~feyrer/.ssh directory] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable ssh login with ssh-key] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install sudo] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Disable ssh logins as root] ********************* 
      ok: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=6    unreachable=0    failed=0    
      
      Let's have a look if things actually work:
      % ssh 107.22.69.112 id
      uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
      % ssh ec2-107-22-69-112.compute-1.amazonaws.com id
      uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
      % ssh ec2-107-22-69-112.compute-1.amazonaws.com sudo id
      uid=0(root) gid=0(wheel) groups=0(wheel),2(kmem),3(sys),4(tty),5(operator),20(staff),31(guest)
      
    7. Next, install database software and import our demo database, just as we did in out local VM:
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml
      
      PLAY [security_group_ec2-webservers] ********************* 
      
      TASK: [Install mysql] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install MySQL rc.d script] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Start MySQL service] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install python-mysqldb (for mysql_user module)] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Setup DB] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Add db-user] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Copy over DB template] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Import DB data] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-107-22-69-112.compute-1.amazonaws.com : ok=8    changed=8    unreachable=0    failed=0    
      
      Again, let's see if everything works as expected:
      % ssh ec2-107-22-69-112.compute-1.amazonaws.com
      ...
      ip-10-202-65-196: {1} mysql -u webapp -p webapp
      Enter password: ******
      ...
      mysql> show tables;
      +------------------+
      | Tables_in_webapp |
      +------------------+
      | names            |
      +------------------+
      1 row in set (0.00 sec)
      
      mysql> select * from names;
      +----+--------+------+
      | id | first  | last |
      +----+--------+------+
      |  1 | Donald | Duck |
      |  2 | Daisy  | Duck |
      +----+--------+------+
      2 rows in set (0.00 sec)
      
      mysql> exit
      Bye
      ip-10-202-65-196: {2} exit
      logout
      Connection to ec2-107-22-69-112.compute-1.amazonaws.com closed.
      
    8. Last, add Apache+PHP and our small demo web-application:
        
      % env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml
      
      PLAY [security_group_ec2-webservers] ********************* 
      
      TASK: [Installing ap24-php53 package and dependencies] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install Apache rc.d script] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable and start Apache service] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable PHP in Apache config file] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})
      
      TASK: [Make Apache read index.php] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Install phpmyadmin] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable phpmyadmin in Apache config] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Fix Apache access control for phpmyadmin] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Enable PHP modules in PHP config file] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})
      
      TASK: [Create directory for webapp] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Deploy example webapp] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      TASK: [Create webapp symlink for easy access] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      NOTIFIED: [restart apache] ********************* 
      changed: [ec2-107-22-69-112.compute-1.amazonaws.com]
      
      PLAY RECAP ********************* 
      ec2-107-22-69-112.compute-1.amazonaws.com : ok=14   changed=14   unreachable=0    failed=0    
      
    9. Test!
      % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/
                                         It works!
      
      % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/phptest.php
         PHP Logo                                                                   
                                                                                    
                                     PHP Version 5.3.17                             
      
         System          NetBSD ip-10-202-65-196.ec2.internal 6.0.1 NetBSD 6.0.1    
                         (XEN3PAE_DOMU) i386                                        
         Build Date      Dec 14 2012 10:31:13                                       
      ...
      
      % links -dump http://ec2-107-22-69-112.compute-1.amazonaws.com/webapp/
         Showing table hf.names:
      
         +--------------------+
         | id | first  | last |
         |----+--------+------|
         | 1  | Donald | Duck |
         |----+--------+------|
         | 2  | Daisy  | Duck |
         +--------------------+
      
           ----------------------------------------------------------------------
      
         Enter new values:
      
         first:     _____________________ 
         last:      _____________________ 
         [ Submit ] 
      
    10. At this point, everything is setup and can be enjoyed. If the instance is needed no longer, it can be terminated:
      % euca-describe-instances
      RESERVATION     r-d77272ad      749335780469    ec2-webservers
      INSTANCE        i-9fafc2f2      ami-5d0f8034    ec2-107-22-69-112.compute-1.amazonaws.com       ...
      % euca-terminate-instances i-9fafc2f2
      INSTANCE        i-9fafc2f2
      % euca-describe-instances
      RESERVATION     r-d77272ad      749335780469    ec2-webservers
      INSTANCE        i-9fafc2f2      ami-5d0f8034                    terminated      eucaHF  ...
      
    What's next on my journey is to put database and webserver into separate VMs. First one of each, and then see if I find the nerve to look into a setup with more redundancy.

    Shameless plug: I'll talk about the ansible and euca2ools packages at pkgsrcCon 2013 in Berlin. Join in if you're curious about what the actual playbooks used in the above examples look like!

    [Tags: , , , , , , ]



    More recent 10 entriesPrevious 10 entries
    Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

    Access count: 17754269
    Copyright (c) Hubert Feyrer