hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20080904] Speeding up operations with directory hashing
While working on UDF, Reinoud Zandijk has added routines to speed up directory operations. Remember that in FFS, directories are list of struct dirent entries that are stored one after another. This makes lookup sequential, deleting leaves holes, and inserting needs to check for possible holes. In other words: it's suboptimal. The proper solution would be to move some tree-based directory format, but that's not around the corner - file systems like SGI's XFS have that, and anyone's welcome to do the work on FFS.

Another option to mitigate the suboptimal on-disk format is to use in-memory caching and fast lookup, which is just what Reinoud did. His runs of the "postmark" benchmark show some impressive speedups:

What Old New
File create: 2/s 208/s
File create alone: 4/s 230/s
File create + transaction: 1/s 318/s
File read: 1/s 330/s
File append: 1/s 336/s
File delete: 2/s 208/s
File delete alone: 9/s 970/s
File delete + transaction: 1/s 348/s
Data read: 3 kb/s 283kb/s
Data written: 14kb/s 1150kb/s

See the commit log for more information.

P.S.: Someone please add wikipedia pages for FFS/UFS!

P.P.S.: Yes, this has been in UFS/FFS as "dirhash" for some time

[Tags: ]


[20080610] UDF write support
Reinoud Zandijk has been working on support for the Universal Disk Format in NetBSD for quite some time, and in mid-may he has reached another major mile stone by adding write support to NetBSD's UDF file system: ``It can now read and *write* files and directories on CD-R, CD-RW, CD-MRW, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD+MRW, (USB) flash media and harddisc partitions. Media like Iomega Rev should also work fine since they attach to the cd(4) device.

Rewritable media needs to be optionally blanked and then (packet) formatted first using the new mmcformat(8) tool. This will format the disc to be usable for UDF. For the drives that support it, you can also format CD-MRW and DVD+MRW wich give background formatting and a precieved error free media.

For recordable media and non optical media a newfs_udf(8) is enough. See the manpage for the various options. Note that the UDF version specifying is still a bit odd, it needs to be done in hex.

As for recordable optical media, for formatting and newfsing you need to specify the specify the raw complete disc i.e. /dev/rcd0c or /dev/rcd0d (on i386, amd64).

For all other media (i.e. NOT optical media), one has the choice to completely use the disc by using the complete device extent or to use a partition scheme. newfs_udf(8) will use the complete partition specified.

Discs can be mounted with mount_udf(8). Supported options next to the UDF specific options are async and noatime.

Current limitations are that disc accounting is not done correctly yet so be carefull not to run out of space on disc and that writing to metadata partitions is not implemented yet. Also fsck_udf(8) is not yet finished, but will be added later.''

[Tags: , ]


[20080131] Article: Waving the flag: NetBSD developers speak about version 4.0
Federico Biancuzzi has collected interviews from more than twenty NetBSD developers in an multiple-page article which talks about what's new in the NetBSD 4.0 release: If you have any comments, there's also a page for comments and discussion available.

[Tags: , , , , , , , , , , ]


[20070106] Using your Xbox 360 HD DVD drive with NetBSD
Jared McNeill bounced me this gem: The Xbox 360 comes with an external HD DVD drive that's connected to the Xbox via USB. Using the drive on a NetBSD box via USB is easy, but accessing the data on the HD DVDs was not possible so far. After Reinoud Zandijk's last round of changes to the UDF filesystem driver to bring it upto UDF 2.60, the Xbox 360 HD DVDs' data can now be accessed from NetBSD: 

# dmesg
...
uhub5 at uhub3 port 1
uhub5: NEC product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub5: 4 ports with 2 removable, self powered
ugen0 at uhub5 port 3
ugen0: Microsoft Corporation Xbox 360 HD DVD Memory Unit, rev 2.00/1.08,
addr 3
umass0 at uhub5 port 4 configuration 1 interface 0
umass0: Microsoft Xbox 360 HD DVD Player, rev 2.00/1.00, addr 4
umass0: using ATAPI over Bulk-Only
atapibus1 at umass0: 2 targets
cd1 at atapibus1 drive 0:  cdrom removable
...
# mount -t udf /dev/cd1a /mnt
#
# ls -al /mnt
total 26
dr--r--r--   6 nobody  nobody   228 Jul 20 00:20 ./
drwxr-xr-x  21 root    wheel    512 Nov 29 01:36 ../
dr--r--r--   2 nobody  nobody   616 Jul 20 00:20 AACS/
dr--r--r--   2 nobody  nobody   556 Jul 20 00:20 AACS_BAK/
dr--r--r--   2 nobody  nobody   240 Jul 20 00:20 ADV_OBJ/
dr--r--r--   2 nobody  nobody  2124 Jul 20 00:20 HVDVD_TS/
#
# ls -al /mnt/AACS
total 6288
dr--r--r--  2 nobody  nobody      616 Jul 20 00:20 ./
dr--r--r--  6 nobody  nobody      228 Jul 20 00:20 ../
-r--r--r--  1 nobody  nobody      120 Jul 20 00:20 CONTENT_CERT.AACS
-r--r--r--  1 nobody  nobody   134496 Jul 20 00:20 CONTENT_HASH_TABLE1.AACS
-r--r--r--  1 nobody  nobody    61908 Jul 20 00:20 CONTENT_HASH_TABLE2.AACS
-r--r--r--  1 nobody  nobody  1000000 Jul 20 00:20 CONTENT_REVOCATION_LIST.AACS
-r--r--r--  1 nobody  nobody       64 Jul 20 00:20 DKF.AACS
-r--r--r--  1 nobody  nobody  1000000 Jul 20 00:20 MKBRECORDABLE.AACS
-r--r--r--  1 nobody  nobody  1000000 Jul 20 00:20 MKBROM.AACS
-r--r--r--  1 nobody  nobody      301 Jul 20 00:20 MNGCPY_MANIFEST.XML
-r--r--r--  1 nobody  nobody     2480 Jul 20 00:20 VTKF000.AACS
-r--r--r--  1 nobody  nobody      144 Jul 20 00:20 VTUF000.AACS
#
# ls -al /mnt/HVDVD_TS
total 32933404
dr--r--r--  2 nobody  nobody        2124 Jul 20 00:20 ./
dr--r--r--  6 nobody  nobody         228 Jul 20 00:20 ../
...
-r--r--r--  1 nobody  nobody   286636032 Jul 20 00:20 HDintro.EVO
-r--r--r--  1 nobody  nobody        2048 Jul 20 00:20 HDintro.MAP
...
-r--r--r--  1 nobody  nobody    63297536 Jul 20 00:20 TERM3NPCTLR1_HD.EVO
-r--r--r--  1 nobody  nobody        2048 Jul 20 00:20 TERM3NPCTLR1_HD.MAP
-r--r--r--  1 nobody  nobody    73046016 Jul 20 00:20 TERM3NSGTV1_HD.EVO
-r--r--r--  1 nobody  nobody        2048 Jul 20 00:20 TERM3NSGTV1_HD.MAP
-r--r--r--  1 nobody  nobody    17657856 Jul 20 00:20 WHVlogo240.EVO
...
-r--r--r--  1 nobody  nobody        2048 Jul 20 00:20 WHVlogo240.MAP
-r--r--r--  1 nobody  nobody     1124352 Jul 20 00:20 intro_black.EVO
-r--r--r--  1 nobody  nobody        2048 Jul 20 00:20 intro_black.MAP
#
Please note that while the HD DVDs' can now be accessed, it is still encrypted, but it's a start. Jared tells me: ``Basically, the video files on the disc are encrypted and wrapped in an MPEG program stream. You need to extract the title key (also encrypted) from the disc to be able to decrypt the program streams. The video may be either MPEG2, VC-1, or H.264. I can't remember off the top of my head what formats are allowed for audio''.

The changes to UDF should theoretically work for BluRay DVDs also, but this hasn't been confirmed due to lack of hardware - maybe someone can send some feedback on this? Reinoud has the following to say about this: ``BluRay discs have most likely UDF 2.60 filesystem on them since this is the first standard that mentions this disc type. For BD-ROM and BD-RE this should be equal to version 2.50. On BD-R either VAT or logical overwrite is used but for read-only access this shouldn't differ. Testing it of course would be good.''

Reinoud also points out that the next points on his roadmap include optical disc formatting, creating newfs_udf(8), fsck_udf(8) and write support.

[Tags: , , ]


[20060202] Universal Disk Format (UDF) filesystem added to NetBSD
Reinoud Zandijk has committed his UDF filesystem code to NetBSD-current: ``UDF is a file system defined by the OSTA standardisation group and is tailored for data interchange on optical discs (like CDs and DVDs) between different operating systems. Its also more and more common on other media like Compact Flash (CF) cards.''

See his mail to tech-kern for more information!

[Tags: , , ]


[20050418] UDF project status update (Update #1)
Reinoud Zandijk has posted a status report of his work to create a UDF driver for NetBSD. The idea is to have a userspace-implementation first, which is then made into a kernel driver. Notable differences to other implementations include better support of the UDF standard as well as read/write support.

Update #1: Reinoud wrote more about differences between his implementation and others, esp. from the BSD camp.

[Tags: , , ]


Tags: , 2bsd, 34c3, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, adsense, advocacy, Advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, Hardware, hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ncq, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrccon, pkgsrcCon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, products, Products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, Releases, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, sata, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, Security, security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, storage, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.
Access count: 35118814
Copyright (c) Hubert Feyrer