hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20100212] Musing about git's object store efficiency
I'm currently looking at git to see what it can and cannot do, and one thing I've looked today is how effective the backing store mechanism is. To recall: CVS stores a list of patches between versions in a single file, and git stores each new revision in full in a separate file in the so-called object store. Is that an issue for NetBSD? Let's see;

One of the more frequently updated files is the i386 port's GENERIC kernel config file, which is at revision 1.963 right now. This means that since it's import into CVS, 963 different revisions have been made. In CVS, all those files are kept in a single GENERIC,v file. In git, this puts 963 files on the file system. A bit of a difference.

Looking at the space requirements for storing the repository data itself, the GENERIC,v file is 883,233 bytes[1]. Extracting all 963 versions from revision 1.1 to revision 963 results in disk space usage of 32,805,828 bytes[2,3]. And that's not counting the overhead of 962 inodes and the related directory bookkeeping.

In other words, the git model requires about 37 times the space that CVS does.

Sure the example file is not exactly one with an average number of revisions, and I know that git offers some more efficient storage methods via "pack" files, but investigating those is left as an exercise to the reader. :-)


[1] Obtained via rsync from cvs.netbsd.org:
% ls -la GENERIC,v 
-r--r--r--  1 feyrer  wheel  883233 Feb 12 16:57 GENERIC,v 

[2]

% mkdir extracted
% chdir extracted
% sh -c 'for i in `jot 964`; do echo $i ; co -p -r1.$i ../GENERIC >GENERIC-`printf %04d $i` ; done'

[3]

% cat extracted/* | wc -c
 32805828 


[Tags: , ]


[20091014] Catching up: Webfwlog, git, vnd and sparse disk images, acpismbus
Here are two news items from the past few days:
  • Web-based firewall log reporting and analysis tool Webfwlog 0.94 released: ``Webfwlog is a flexible web-based firewall log analyzer and reporting tool. It supports standard system logs for linux, FreeBSD, OpenBSD, NetBSD, Solaris, Irix, OS X, etc. as well as Windows XP. Supported log file formats are netfilter, ipfilter, ipfw, ipchains and Windows XP. [...]

    With Webfwlog you can design reports to use on your logged data in whatever configuration you desire. Included are example reports as a starting point. You can sort a report with a single click, "drill-down" on the reports all the way to the packet level, and save your reports for later use. You can also create a link directly to any saved report.''

    See the webfwlog homepage for more information.

  • Git copies of cvs modules available - spz@ writes that ``htdocs, othersrc, pkgsrc, src and xsrc are now available as (bare) git repository copies at http://ftp.netbsd.org/pub/NetBSD/misc/repositories/git. These repositories are currently updated every 30 minutes.

    They are not authoritative and are not meant to replace cvs, but if you want to track your development that is not yet fit for committing into cvs head or sending in as a PR, and want to use git for that, they may serve as a convenient starting point or reference.''

  • Support for writing file systems on sparse disk images - File system hacker Antti Kantee writes: ``Yesterday I wanted to write to a file system which was on a sparse disk image. Normally I would have just used a rump mount, but since the file system was not at offset = 0 in the file, this failed. vnd wasn't helpful either: it allowed me to mount the image and then I got weird errors when writing.

    I just finished adding disklabel support to the various rump_fs utilities. Due to them using the option parsing code of the real mount_fs utilities, I decided to signal the label number with a magic parameter at the end of the device (yes, this really simplifies things currently).

    Let's assume you're looking at wd0.img. Here's how it works:

    === SNIP ===
    golem> disklabel wd0.img
    [...]
    16 partitions:
    #        size    offset     fstype [fsize bsize cpg/sgs]
     a:    719712        63     4.2BSD   1024  8192     0  # (Cyl.      0*-    714*)
     b:     66657    719775       swap                     # (Cyl.    714*-    780*)
     c:    786369        63     unused      0     0        # (Cyl.      0*-    780*)
     d:    786432         0     unused      0     0        # (Cyl.      0 -    780*)
    
    golem> rump_ffs wd0.img%PART:a% /puffs
    rump_ffs: "wd0.img" is a non-resolved or relative path.
    rump_ffs: using "/usr/home/pooka/wd0.img" instead.
    golem> df /puffs
    Filesystem                      1K-blocks       Used      Avail %Cap Mounted on
    /usr/home/pooka/wd0.img%PART:a%     338471     256211      65337  79% /puffs
    === SNIP ===
    So you specify the partition number as %PART:n%, where is n is obviously the partition number. Other than that, things work as usual. No vnconfig etc. necessary.
    ''

  • New ACPI driver: acpismbus(4) - call for testers - Paul Goyette writes that ``At the request of some folks off-list, I have created a driver for ACPI's SMBus Control Method Interface. Basically, this is an acpi-based wrapper to provide device-independant access to i2c/SMBus controllers. (See spec at http://smbus.org/specs/smbus_cmi10.pdf for further info.)

    To see if your system can use this driver, check the output of acpidump for the existence of a Device(SMB0). If you have an SMB device, I'd appreciate it if you could help test this driver!''

    Continue with Paul's mail to learn on how to test the driver, and what limitations there are currently.



[Tags: , , , , , ]


[20090919] pkgsrc via git
Did you always feel like exposing your soul to the dark side trying out git, but couldn't find a good reason for? Maybe here's one for you, found via The DragonFly BSD Digest: Matthew Dillon hass etup a GIT pkgsrc repository to give DragonFly users and developers a more reliable(?) way to track the pkgsrc tree. Right now this is just a copy from cvs every 15 minutes, so it won't allow changes back to pkgsrc, but apparently it's much faster to download via git than it is via cvs.

For a quick start, run "git clone git://avalon.dragonflybsd.org/pkgsrc.git". See Matt Dillon's mail for a bit more information.

[Tags: , ]


Tags: , 2bsd, 34c3, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, adsense, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, embedded, Embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, hardware, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ncq, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrccon, pkgsrcCon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, Products, products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, releases, Releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, sata, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, security, Security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, storage, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 34935190
Copyright (c) Hubert Feyrer