hubertf's NetBSD blog

The adventure of rebuilding g4u from source

Sat, 17 Mar 2018 20:18:00 GMT

I was asked by a long-time g4u user on help with rebuilding g4u from sources. After pointing at the instructions on the homepage, we figured out that a few lose odds and ends didin't match. After bouncing some advices back and forth, I ventured into the frabjous joy of starting a rebuild from scratch, and quick enough ran into some problems, too.

Usually I cross-compile g4u from Mac OS X, but for the fun of it I did it on NetBSD (7.0-stable branch, amd64 architecture in VMware Fusion) this time. After waiting forever on the CVS checkout, I found that empty directories were not removed - that's what you get if you have -P in your ~/.cvsrc file.

I already had the hint that the "g4u-build" script needed a change to have "G4U_BUILD_KERNEL=true".

From there, things went almost smooth: building indicated a few files that popped up "variable may be used uninitialized" errors, and which -- thanks to -Werror -- bombed out the build. Fixing was easy, and I have no idea why that built for me on the release. I have sent a patch with the required changes to the g4u-help mailing list. (After fixing that I apparently got unsubscribed from my own support mailing list - thank you very much, Sourceforge ;)).

After those little hassles, the build worked fine, and gave me the floppy disk and ISO images that I expected:

>       ls -l `pwd`/g4u*fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u1.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u2.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u3.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u4.fs
>       ls -l `pwd`/g4u.iso
>       -rw-r--r--  2 feyrer  staff  6567936 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u.iso
>       ls -l `pwd`/g4u-kernel.gz
>       -rw-r?r--  1 feyrer  staff  6035680 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u-kernel.gz

Next steps are to confirm the above changes as working from my faithful tester, and then look into how to merge this into the build instructions .

g4u 2.6 released

Thu, 08 Jun 2017 16:19:00 GMT

After a five-year period for beta-testing and updating, I have finally released g4u 2.6. With its origins in 1999, I'd like to say: Happy 18th Birthday, g4u!

About g4u: g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

The past: When I started g4u, I had the task to install a number of lab machines with a dual-boot of Windows NT and NetBSD. The hype was about Microsoft's "Zero Administration Kit" (ZAK) then, but that did barely work for the Windows part - file transfers were slow, depended on the clients' hardware a lot (requiring fiddling with MS DOS network driver disks), and on the ZAK server the files for installing happened do disappear for no good reason every now and then. Not working well, and leaving out NetBSD (and everything elase), I created g4u. This gave me the (relative) pain of getting things working once, but with the option to easily add network drivers as they appeared in NetBSD (and oh they did!), plus allowed me to install any operating system.

The present: We've used g4u successfully in our labs then, booting from CDROM. I also got many donations from public and private instituations plus comanies from many sectors, indicating that g4u does make a difference.

In the mean time, the world has changed, and CDROMs aren't used that much any more. Network boot and USB sticks are today's devices of choice, cloning of a full disk without knowing its structure has both advantages but also disadvantages, and g4u's user interface is still command-line based with not much space for automation. For storage, FTP servers are nice and fast, but alternatives like SSH/SFTP, NFS, iSCSI and SMB for remote storage plus local storage (back to fun with filesystems, anyone? avoiding this was why g4u was created in the first place!) should be considered these days. Further aspects include integrity (checksums), confidentiality (encryption). This leaves a number of open points to address either by future releases, or by other products.

The future: At this point, my time budget for g4u is very limited. I welcome people to contribute to g4u - g4u is Open Source for a reason. Feel free to get back to me for any changes that you want to contribute!

The changes: Major changes in g4u 2.6 include:

Make this build with NetBSD-current sources as of 2017-04-17 (shortly before netbsd-8 release branch), binaries were cross-compiled from Mac OS X 10.10
Many new drivers, bugfixes and improvements from NetBSD-current (see beta1 and beta2 announcements)
Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!)
Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz
In addition to MD5, add SHA512 checksums

The software: Please see the g4u homepage's download section on how to get and use g4u.

Enjoy!

g4u 2.6beta2 has been released - Happy 18th Birthday, g4u!

Fri, 21 Apr 2017 01:51:00 GMT

Just right in time for its 18th birthday, I have released the 2nd beta version for g4u 2.6 (2.6beta2). It took some time to get to this point, and I want to move to 2.6 soon - please take your time to test and get back to me soon, as I want to push out g4u version 2.6 in june 2017.

g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

For more information, see http://www.feyrer.de/g4u/.

Changes in 2.6beta2 include:

Make this build with NetBSD-current sources as of 2017-04-17, binaries were cross-compiled from Mac OS X 10.10
Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!)
Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz
Put all object files into one object directory. This may need more cleanup in the future. Feedback from people building g4u welcome!
Disable verbose device messages on Microchannel (MCA) machines
New drivers:
- Toshiba Dynabook hotkeys
- Intel S1200,C2000 (non-pch) SMBus storage controller
- Non-Volatile Memory Express (NVMe) storage controllers and devices
- Intel Wireless WiFi Link 7xxx PCI network
- Intel 8259x 10 gigabit PCI network
- Realtek 8188CE/8192CE 802.11b/g/n PCI network
- VMware VMXNET3
- Marvell Libertas PCMCIA network
- ASIX AX88178a/AX88179 based USB network adapters
- Realtek RTS5209/RTS5229 Card Reader
Happy 18th Birthday, g4u!

Download links:

The g4u 2.6beta2 floppy images (zipped/ uncompressed floppy one, floppy two, floppy three, and floppy four)
The g4u 2.6beta2 ISO CD image (zipped/uncompressed)
The g4u/NetBSD kernel for PXE boot (gzipped)
The g4u 2.6beta2 source code

Some md5 checksums:

MD5 (g4u-2.6beta2-1.fs) = 8214c1ded55e78aed2f34d3e318dc75d
MD5 (g4u-2.6beta2-2.fs) = 8b55f3fe6a4806c9a65a3d176463e1bc
MD5 (g4u-2.6beta2-3.fs) = 0f9629353ddd470d17c5b4fd4ec7fd8e
MD5 (g4u-2.6beta2-4.fs) = c25a955e5d876d64943fd03f957ed4c9
MD5 (g4u-2.6beta2.iso) = acabb0fb73d4a2f9062fb070b4ddd825
MD5 (g4u-2.6beta2.tgz) = f8387d5bf146f98dda040dfcc1c53e44
MD5 (g4u-2.6beta2.fs.zip) = 8ef324c71ae7b000708e69019e55ef4e
MD5 (g4u-2.6beta2.iso.zip) = 1f4bd37784405350ee3e8e6c75a6b6b6

g4u 2.6beta1 released

Sun, 10 Nov 2013 16:28:00 GMT

I have release g4u version 2.6beta1. Important changes are an update to recent NetBSD codebase, and moving the ramdisk from a separate file back into the kernel. This allows easy netbooting - at least I hope so, feedback is welcome here.

I'd like to push out version 2.6 within the next few weeks. Please test and let me know if there are any showstoppers!

Full list of news in g4u 2.6beta1:

Make this build with NetBSD-current sources as of 2013-10-20
Move back from a ramdisk that's loaded from a separate file back to a ramdisk that's part of the kernel image. This allows easier netbooting for those people who want it - added back by popular demand :-)
Added more kernel buffer space, to hold all kernel messages for dmesg, even on machines with large ACPI tables (Hello VMware Fusion!)
New drivers:
- LSILogic 9x9 and 53c1030 (Fusion-MPT) PCI SCSI
- LSI Logic Fusion-MPT II PCI SCSI
- Atheros AR9k (802.11a/g/n) PCI Wireless
- Marvell PCI Libertas Wireless
- Atheros AR9k (802.11a/g/n) PC-Card Wireless
- Broadcom BCM43xx PC-Card Wireless
- Atheros AR9002U USB Wireless
- Ralink Technology RT2500USB 802.11a/b/g USB Wireless
- Ralink Technology RT(2[78]|30)00 802.11a/b/g/n USB Wireless
- Realtek RTL8187/RTL8187B 802.11b/g USB Wireless
- Realtek RTL8188CU/RTL8192CU 802.11b/g/n USB Wireless
- Intel Atom E6xx PCI-LPC

Download links:

The g4u 2.6beta1 ISO CD image (zipped/uncompressed)
The g4u 2.6beta1 source
The g4u 2.6beta1 floppy images (zipped/ uncompressed floppy one, floppy two and floppy three)

Some md5 checksums:

MD5 (g4u-2.6beta1-1.fs) = 264d5b27f208f1053777774d0c2b87a7
MD5 (g4u-2.6beta1-2.fs) = 533b37fbeeb2d537928de61b25eb2ade
MD5 (g4u-2.6beta1-3.fs) = 5abe1327326fcdac18bf4d68566f3e9b
MD5 (g4u-2.6beta1.iso) = 37596efb42c014eb049e0027db059bd9
MD5 (g4u-2.6beta1.tgz) = acd5907a7fb2cc8fbd4645a1b9db5272
MD5 (g4u-2.6beta1.fs.zip) = 9b8da62fac57dcd38a4e3376189a087d
MD5 (g4u-2.6beta1.iso.zip) = 44cb8d01407b17f9d9839740b5a0d2d9

G4U Opinion Time: kernel with embedded RAMdisk vs. miniroot?

Sun, 27 Oct 2013 02:08:00 GMT

Quoting my own mail to the g4u-help mailing list: ``I've found little time to hack on g4u in the recent past. Yet, I've managed to setup my development and test environment for g4u (crosscompiling NetBSD from Mac OS X, getting recent Qemu to compile), and also got g4u built from recent NetBSD-current sources. As such, take this as small sign of life.
Now, while I don't have any plans for large changes, I'd like to bring an update with latest drivers and bugfixes from NetBSD.
There's one change that I'm pondering, though: g4u originally came as one kernel-file that had an embedded RAM-disk. This was changed in the last release to reflect NetBSD's ability to load a RAM-disk from a separate file. This change broke the ability to netboot g4u from a single file, and required some more effort. There were no real wins for g4u as such.
So, opinion time: keep the RAM-disk as separate file, or move it back into the kernel?

Looking forward for your opinions!''

Announcing g4u v2.5

Mon, 16 Jul 2012 22:30:00 GMT

After an extended time for beta testing, I'm pushing out g4u V2.5 now, with no functional changes between 2.5beta1 and the final release. Of course full release testing was done on the final release. G4u 2.5 is mainly a maintenance release that brings in commands to upload and restore partition tables with the MBR, has driver updates from NetBSD, and some minor enhancements like (finally!) enabling command line history. See the g4u homepage for more details.

NetBSD vs. disk transfer speeds vs. BIOS settings

Sun, 29 Jan 2012 12:40:00 GMT

A few days ago, Brian Hoard made an interesting finding about performance of g4u, a NetBSD/i386-based disk cloning system. Citing from Brians mail:

``First, my problem was I had just replaced my motherboard on my custom build PC. Once I got Windows 7 64-bit loaded and everything working, I sat up to clone my system drive. The drive is a 500GB Seagate Barracude, SATA 2 drive. Cloning locally to an identical drive.

When booting into g4u, my transfer speeds were extremely slow. Normally, my 500 Gb clones take only about 90 minutes. But this was still working after over 6 hours. The g4u transfer speed was reporting only 1.5 Mb/sec.

I shut things down, and went into my system BIOS. I noticed that the SATA mode was set to "IDE Mode" for my drives. I changed this to "AHCI Mode" and continued to boot into g4u. This worked to fix the transfer speeds, and my clone finished normally. Getting 83 Mb/sec.

Once the drive was finished, I attempted to boot into Windows, but it would not boot. I had to change my BIOS back to "IDE Mode", then Windows behaved normally.

Upon researching this, I am now learning that you should enable AHCI Mode BEFORE installing Windows for it to work. Apparently, if Windows is not installed while using AHCI Mode, it disables the drivers for AHCI on the system drive. So if you later enable AHCI in your BIOS as I did, Windows will not have the driver loaded. I saw there is a fix on the Microsoft web site, but I haven't attempted to try it yet.

If someone else runs into a similar problem, hopefully this will help you.''

FWIW, g4u-2.5beta1 is based on NetBSD-current from January 2012, so checking your BIOS may help anyone seeing bad disk performance out there.
(Emphasizes in the text added by me)

g4u 2.5beta1 supports handling of partition tables and bad disk sectors

Sat, 14 Jan 2012 13:48:00 GMT

After some absence (job-related) and technical problems (building of NetBSD failing for me from Mac OS X), I'm very happy to release a beta version of g4u with some long-overdue changes. Those include being able to backup/restore the MBR, which includes the partition table - needed when recovering single partitions to a new disk. Also, the various commands reading disks are now adjusted to not abort when a disk sector cannot be used. Instead, the bad bytes are skipped and the rest of the disk is recovered. Please give me feedback on this feature as I didn't have a bad disk to test this! Other news include a command to wipe a disk by completely overwriting it with 0-bytes (once). Last, command line editing was enabled - finally!

Remember that this is a test release, so your feedback is wanted - either to me in person, or to the g4u-help mailing list. Thanks!

Here's a full list of changes:

New commands "uploadmbr" and "slurpmbr" to backup and restore the master boot record, which includes the partition table. Required to restore a partition to an empty disk.
New command "copymbr" to copy the MBR from one disk to another, similar to "copydisk"
New command "wipedisk" to write the disk full with 0-bytes once from start (sector 0) to end (last sector)
Enable command line history/editing by forcing /bin/sh to be built without -DSMALL (ugly hack... there be lots of dragons!)
When setting up a fresh compile tree, g4u patches are now applied automagically without aborting the build
Error detection was now enabled in the dd(1) command, which is the core of g4u (surprise!). With that, disks with broken/unreadable sectors should now be copied, skipping the unreadable sectors and copying the rest. This affects a number of programs: copydisk, copypart, uploaddisk, uploadpart. BEWARE: I wasn't able to actually test this as I do not have a disk with bad sectors here. Please report back your experiences!!!
Make this build with NetBSD-current sources as of 2012-01-12
New drivers added to the kernel:
- RDC PMX-1000 IDE controllers
- Intel SCH IDE controllers
- TOSHIBA PICCOLO controllers
- Attansic/Atheros L1C/L2C Ethernet
- Broadcom BCM43xx wireless
- Agere/LSI ET1310/ET1301 Gigabit Ethernet
- RDC R6040 10/100 Ethernet
- USB LCDs and USB-VGA adaptors, e.g.:
- DisplayLink DL-1x0/1x5
- Option N.V. Wireless USB WAN modems
- Microsoft RNDIS specifications USB ethernet
- Atheros AR9001U USB Wifi
- Intersil PrismGT USB Wifi
- Virtio PCI, memory balloon, disk & network devices
- ... and many more that slipped past QA
... and any driver updates, optimizations and bug fixes and other enhancements from NetBSD-current

Get g4u 2.5beta1:

Netbooting g4u via PXE

Thu, 24 Nov 2011 01:17:00 GMT

Doing a network based boot with PXE is not exactly hard, but you need some debugging and the right tools in place. If you want to netboot g4u, the NetBSD-based tool for harddisk image cloning via FTP, via PXE, there's a description on how to do Netbooting of g4u via PXE by Mariusz Zynel.

Details include setting up a TFTP server for loading the bootloader and getting DHCP sending out the right files.

Enlarging a (virtual) disk

Sat, 22 Oct 2011 21:04:00 GMT

I've tried to build NetBSD-current at various points in the past few months, and always hit one of two bugs: -current blows up with a gcc Internal Compiler Error when crossbuilding on Mac OS X, and kernel panics with native NetBSD builds with sources on NFS. This stinks, and I've successfully managed to do a successful -current build with sources on (local) disk. With NetBSD running within VMware Fusion on Mac OS X.

To go on from there, I found that my NetBSD VM's only disk was too small to do anything useful. Options for enlarging that came to mind:

NFS - see 'panic' above, no go.
Adding another (virtual) disk - easily doable, but I felt like not adding one
Extending the existing disk - adventure time!

Option #3 was it, and after removing all VMware snapshots, enlarging the disk was easy with VMware Fusion, going from 10GB to 20GB. After growing the disk itself, the next question was how to use the newly gained disk. Of course some file system needs to use it, and in theory there are the following options:

Enlarge the last file system on disk
Fix the partition table to add another partition for the new space

The disk was resized from 10GB to 20GB. The partition table (disklabel) was created by a standard NetBSD install, and first had the root file system, followed by the swap partition. From that, adding 10GB more swap was not useful, so I've decided to change the disklabel to add the new disk space as a new partition behind the existing partitions. This is also an excuse to not frob with growfs and resize_ffs. (And of course I'm ignoring the option of backing up the full file system, doing a full rebuild of the filesystem and then doing a restore :-)

For those in a similar situation, here are the steps to use the newly gained space on an enlarged (virtual) disk:

Prepare: save the old output of "dmesg" (/var/run/dmesg.boot is OK)
Enlarge - VMware Fusion wants a shutdown for that, you cannot suspend the machine
After booting, run a diff on the saved "dmesg" output, to learn what the old and new size of the disk is, in sectors. My diff looks like this, note the size change in sectors:
```
-wd0: 10240 MB, 22192 cyl, 15 head, 63 sec, 512 bytes/sect x 20971520 sectors
+wd0: 20480 MB, 44384 cyl, 15 head, 63 sec, 512 bytes/sect x 41943040 sectors 
```
Backup the existing/old disklabel, just in case: disklabel wd0 >disklabel.BAK
Edit the disklabel: disklabel -e wd0
In the editor, adjust the disk size in sectors from 20971520 to 41943040:
```
total sectors: 41943040 
```

Partition 'd' is the full disk on i386/amd64, it starts at sector 0 and is 41943040 sectors big

#        size    offset     fstype [fsize bsize cpg/sgs]
 d:  41943040         0     unused      0     0        # (Cyl.      0 -  44384*)

Partition 'c' is the NetBSD part of the disk. As this VM only has NetBSD, all the usable space is used. Note that "usable" space excludes the first 63 sectors of the disk (mbr etc.), i.e. it is 41943040 - 63 = 41942977 sectors:
```
#        size    offset     fstype [fsize bsize cpg/sgs]
 c:  41942977        63     unused      0     0        # (Cyl.      0*-  44384*)
```
After this everything is in sync with the new disk again, and the remaining/new space can be used for new partition 'e'. As the new space starts where the disk used to end, its offset is the old size, 20971520 sectors.
The size of the new partition expands from the offset sector 20971520 to the end of the disk at sector 41943040, i.e. the partition size is:
```
% expr 41943040 - 20971520
20971520
```
In total, this gives for the new partition:
```
#        size    offset     fstype [fsize bsize cpg/sgs]
 e:  20971520  20971520     4.2BSD   2048 16384     0  # (Cyl.  22192*-  44384*)
```

Last, create file system, mount and populate:

# newfs /dev/rwd0e
# mkdir /disk2
# echo '/dev/wd0e /disk2 ffs rw,log 2 2' >>/etc/fstab
# mount /disk2
# cd /usr ; pax -rw -pe -v stuff /disk2
# rm -fr stuff ; ln -s /disk2/stuff .

Now let's see if I get things far enough to get a build of g4u going... wish me luck!

P.S.: I'm offering choccolate to anyone fixing crossbuilding of NetBSD-current from Mac OS X. Any takers?