hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20080610] UDF write support
Reinoud Zandijk has been working on support for the Universal Disk Format in NetBSD for quite some time, and in mid-may he has reached another major mile stone by adding write support to NetBSD's UDF file system: ``It can now read and *write* files and directories on CD-R, CD-RW, CD-MRW, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD+MRW, (USB) flash media and harddisc partitions. Media like Iomega Rev should also work fine since they attach to the cd(4) device.

Rewritable media needs to be optionally blanked and then (packet) formatted first using the new mmcformat(8) tool. This will format the disc to be usable for UDF. For the drives that support it, you can also format CD-MRW and DVD+MRW wich give background formatting and a precieved error free media.

For recordable media and non optical media a newfs_udf(8) is enough. See the manpage for the various options. Note that the UDF version specifying is still a bit odd, it needs to be done in hex.

As for recordable optical media, for formatting and newfsing you need to specify the specify the raw complete disc i.e. /dev/rcd0c or /dev/rcd0d (on i386, amd64).

For all other media (i.e. NOT optical media), one has the choice to completely use the disc by using the complete device extent or to use a partition scheme. newfs_udf(8) will use the complete partition specified.

Discs can be mounted with mount_udf(8). Supported options next to the UDF specific options are async and noatime.

Current limitations are that disc accounting is not done correctly yet so be carefull not to run out of space on disc and that writing to metadata partitions is not implemented yet. Also fsck_udf(8) is not yet finished, but will be added later.''

[Tags: , ]



[20080428] Recent development related to puffs, ReFUSE, rump, and more (Updated)
NetBSD's kernel is under very active development these days, and while many changes are related to improve SMP, it's not the only area. An area where very interesting and unique work is being done is the filesystem interfaces that Antti Kantee is working. Things started out as a past year's Google "userfs" SoC project to implement an interface for running filesystem code in userland. The project was imported into NetBSD some time ago. On top of that, a library that mimics the Linux interface for filesystems in userland. Following the Linux name FUSE, the re-implementation is called ReFUSE (pun intended :). See the webpage about puffs, refuse, FUSE on the NetBSD website for more information.

Another project that was started by Antti after his work to run filesystem code in userland is "rump". The project allows to use "ordinary" filesystems that usually run inside the kernel, and mimic an environment similar to what's available inside the kernel, and move the whole filesystem into userland - verbatime, with no code changes! This allows to develop filesystem code in userland, and later on move it inside the kernel with no further changes - a bit step forward for filesystem development!

This all sounds rather easy, but as filesystems need to move data between storage and memory, a big issue in filesystems is interfacing with the virtual memory subsystem, and adding interfaces like puffs and ReFUSE also needs to consider VM for efficient transfers and caching.

Work in this area is still ongoing, and I've asked Antti about his recent achievements in this area[1]. While the only user-visible change is caching and performance improvements in the Secure Shell filesystem's handler "mount_psshfs", most of the changes are on the inside. Antti wrote me: ``The interesting ones from a programmer's perspective are probably:

  • Splitting userspace transport out of puffs in the kernel (putter)
  • Using putter to implement support for userspace block/char device drivers (pud). pud does still not have a userspace library similar to libpuffs. libpuffs needs to become libputter and lib{puffs,pud}.

  • Removing special case handling for the puffs user/kernel protocol transport. This means that file system requests can now be read/written like any other protocol. This is covered in the AsiaBSDCon 2008 paper "Send and Receive of File System Protocols: Userspace Approach With puffs"

    With some minor work in libpuffs, it possible to e.g. do an ffs mount from a remote site with the help of rump.

Finally, while not really useful for anything except puffs development, I think the following is cool from the perspective of completeness:

  • Add support to rump to be able to run the puffs kernel module in userspace. This means that that *any* puffs file system (incl. rump ones) can be mounted so that requests pass once through the puffs kernel module running in the kernel and once through the puffs kernel module running in userspace before being delivered to the file system driver. Example:
      sys/rump/fs/bin/syspuffs> ./syspuffs mount_psshfs server.address /path 
''

With puffs and rump, there are two very interesting and active projects doing research in filesystems on NetBSD, which may lead to changes in the way filesystems are understood in the Unix world. While there, a third project that may be worth watching in this regards is this year's Google hurdt Summer of Code project by Marek Dopiera, which aims at implementing Hurd translators for NetBSD,

Update: Antti dropped me a note that another project related to filesystems is this year's "fs-utils" SoC project. The goal is to create a userland tool to manipulate filesystem images, and the idea is to reuse kernel code with the ukfs library. That way, no redundancy between kernel sources and userland sources are created, and both areas benefit from mutual testing and code maturity.

[Tags: , , , , , ]



[20070221] ReFUSE - Linux-compatible filesystem in userspace interface
After Antti Kantee has picked up his previous userfs: Userspace file system hooks Summer-of-Code project, it has since been integrated into NetBSD, together with a small number of example filesystems -- see my previous report on ssshfs.

Now while having a new interface for filesystems in userspace is all nice and dandy, there's already such a thing which (of course) differs in details and is not compatible with the puffs interface: Linux' Filesystem in Userspace (FUSE). Alistair Crooks has sat down and implemented "refuse", a FUSE-compat library for puffs that allows using Linux FUSE filesystems with NetBSD's puffs.

After compatibility with Linux on the source level was realized, FUSE-based filesystems can be ported to NetBSD easily, and a separate category was created for them in NetBSD's Packages System: pkgsrc/filesystems (link will work shortly ;). Packages that are available right now:

  • fuse: Filesystem in Userspace (compat headers, pkg-config files, etc.), needed for pkgsrc on Linux
  • fuse-archivemount: FUSE gateway to libarchive
  • fuse-cddfs: FUSE filesystem that uses libparanoia for audio CDs
  • fuse-cryptofs: FUSE encrypted filesystem
  • fuse-curlftpfs: FUSE filesystem for acessing FTP hosts
  • fuse-httpfs: FUSE HTTP filesystem
  • fuse-lzofs: Filesystem which allows you to transparently use compressed files
Much of the work on pkgsrc was done by Juan 'xtraeme' Romero Pardines, who also told me about his (spanish-language) blog.

Please note that this is an area where heavy development is still going on, and that this work is only available on NetBSD-current. Upcoming work that we can look forward for are improvements on FUSE compatibility and a port of the ntfs-3g filesystem, an open source, freely available NTFS driver for Linux with read and write support. Many more FUSE modules/filesystems are available, so it can be expected that there's more to come.

[Tags: , , ]



[20061109] tmpfs improvements
Last year's tmpfs Summer-of-Code project has been integrated into NetBSD for some time now under an 'experimental' status. After discussion a few weeks ago, author Julio Merino Vidal fixed another round of nits that made tmpfs all good & shiny, Julio's posting to tech-kern. With those changes, I think we can hope to see tmpfs as non-experimental in NetBSD 4.0.

[Tags: , , , ]


[20060228] Experimental benchmark result of recent PC UNIX file systems
I haven't seen this one before, and it's a bite dated (from 2003, speaking about NetBSD 1.6ZC, FreeBSD 5.1 and RedHat 9), but I think the results are not uninteresting, citing from the conclusions: ``
  • Write:
    • Linux software RAID5 25MB/s
    • NetBSD software RAID5 10MB/s
    • FreeBSD software RAID5 5MB/s
    • Linux hardware RAID5 10MB/s
    • NetBSD hardware RAID5 25MB/s
    • FreeBSD hardware RAID5 20MB/s
  • Read:
    • Linux software RAID5 45MB/s
    • NetBSD software RAID5 33MB/s
    • FreeBSD software RAID5 25MB/s
    • Linux hardware RAID5 25MB/s
    • NetBSD hardware RAID5 30MB/s
    • FreeBSD hardware RAID5 24MB/s
'' Seems there's more than 1-2 operating systems that are "good" on PCs. (Oh wait, NetBSD isn't strictly for PCs... but I guess adding all those abstraction layers that make NetBSD portable did not hurt performance too hard; I'd be interested to see a 2006 edition of that benchmark)

[Tags: , , ]


[20060202] Universal Disk Format (UDF) filesystem added to NetBSD
Reinoud Zandijk has committed his UDF filesystem code to NetBSD-current: ``UDF is a file system defined by the OSTA standardisation group and is tailored for data interchange on optical discs (like CDs and DVDs) between different operating systems. Its also more and more common on other media like Compact Flash (CF) cards.''

See his mail to tech-kern for more information!

[Tags: , , ]



[20060127] File systems documentation
Julio M. Merino Vidal reminds us that ``one of the goals of the tmpfs Summer of Code project was to wr ite documentation about NetBSD's VFS subsystem''. A first draft of his work is now available, feel free to give him any feedback!

[Tags: , , ]


[20060115] Article: DDJ on Google's Summer of Code: Part III
In their series of articles introducing a number of SoC projects, Doctor Dobbs Journal introduces the Userspace Filesystems Framework Antti Kantee wrote for NetBSD. Other projects introduces are FreeBSD/nsswitch and Caching, gloox: A High-Level Jabber/XMPP Library for C++, SPARQL for Sesame and TSC-I2: A Lightweight Implementation for Precision-Augmented Timekeeping. See the webpage for more!

[Tags: , , ]


[20050909] Filesystem layering: FFS, LFS, MFS
jmmv has a nice blog entry describing the various layers in FFS, including brief description of the layers and code references.

[Tags: , , , ]


[20050728] Knowledge FileSystem for NetBSD
``Knowledge FileSystem for NetBSD is an intelligent and auto-sorting filesystem developed as a final year project by me, Kailash Sethuraman, for Nanyang Technological University. It is more of a framework than a conventional filesystem. The purpose of the filesystem is to allow the actions of every VFS operation to be controlled by plugins in userspace, to provide an intelligent filesystem which can automatically sort and in the future, search your data. [...]

[After mounting knowfs], all VOP_RENAMEs to the smart folder will be intercepted by KFS. The use of the mv program to drop files into the smart folder will result in the data getting sorted automatically. For example one can just state that all pdfs go into a pdf/ folder, and all mp3s are to be parsed by a mp3.s script. A drag and drop into this folder, will cause the action to be taken on the file. '' See the homepage for a lot more data and nice graphs. Reminds me a bit of the NetBSD/Summer-of-Code "userfs" project, but I may be wrong. Good to see the tradition of research being carried out on NetBSD continuing!

[Tags: , ]



Previous 1 entries

Tags: , 2bsd, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, hardware, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrcCon, pkgsrccon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, Products, products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, Releases, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, Security, security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.

Access count: 23621981
Copyright (c) Hubert Feyrer