Tuesday, November 23, 2010

NFS and IO testing : Bonnie++

Been spending time lately testing out and trying to tune NFS mounts.
For testing IO, I've been using
  • dd
  • iozone
  • bonnie++
  • orion
This first option, dd, is fine for file creation tests and/or sequential reads but less flexible for random reads or random writes
To test random reads I thought I'd use iozone but don't see a solely random read test.
Then I thought I'd use the Oracle supplied tool, orion, to do random read test on an NFS mounted file system, but this doesn't work, at least on AIX 6.1, with my mount settings.
First orion test gave this error
$ orion -run simple -testname orion
ORION: ORacle IO Numbers -- Version
rwbase_read_luncfg: SlfFopen error on orion.lun
orion_parse_args: rwbase_read_luncfg failed
OK, have to create "orion.lun" with either my lun locations or my file locations. I put in file locations:
$ cat orion.lun
Now get another error
$ orion -run simple -testname orion
ORION: ORacle IO Numbers -- Version
Test will take approximately 9 minutes
Larger caches may take longer
orion_spawn: skgpspawn failed: Error category: 27155, Detail: 2
orion_main: orion_spawn failed
Non test error occurred
Orion exiting
Illegal instruction(coredump)
Looks like the "orion" executable wasn't being found, at least not by execve
$ truss -f orion -run simple -testname orion
700502: execve("orion", 0x0FFFFFFFFFFBF2D0, 0x0FFFFFFFFFFFFB30) Err#2 ENOENT
so I ran it from my bin directory where the orion executable could be found. Now get another error
$ orion -run simple -testname orion
ORION: ORacle IO Numbers -- Version
Test will take approximately 9 minutes
Larger caches may take longer
storax_skgfr_openfiles: File identification failed on /kyle/system01.dbf
OER 27054: please look up error in Oracle documentation
Additional information: 6
rwbase_lio_init_luns: lun_openvols failed
rwbase_rwluns: rwbase_lio_init_luns failed
orion_thread_main: rw_luns failed
Non test error occurred
Orion exiting
If the datafile was on "/tmp" it word fine but if it was on my NFS mount it failed with the above error. Hmm - doesn't work over NFS?
Finally I decided to try bonnie++. I download, unziped, tar extracted bonnie++ , ran "./configure" and then make, but got error
$ make
make: 1254-002 Cannot find a rule to create target bon_io.o from dependencies.
For some reason the rule for the ".o" doesn't seem to work:
%.o: %.cpp %.h bonnie.h port.h
$(CXX) -c $<
Not feeling like trying to figure make out I just did it by hand
$ for i in *cpp; do
name=`echo $i | sed -e 's/.cpp//'`
echo "g++ -c $name.cpp $name.h bonnie.h port.h"
done > makeo
$ sh makeo
$ make
That worked. Test results coming


  1. ORION is somewhat picky on NFS. I did manage to run it on NFS but I remember I had to play with strace for a while to get to the bottom of it. I think it was issue with ASYNC IO settings.

    Btw, it still didn't work on NFS with ASYNC IO on RHEL 4. Even though IOs were done via async syscall, based on strace I concluded that there was serialization somewhere and ORION couldn't schedule multiple concurrent IOs.

  2. Did get ORION working. Looks like Orion, as well as Oracle, require an entry in /etc/filesystems. For example I added

    dev = /kyle
    vfs = nfs
    mount = true
    options = rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600
    account = false

    and now Orion works.
    Will do some testing ...

  3. good morning Kyle and all readers,
    I have been using the fio (http://freshmeat.net/projects/fio/) tool with pleasure. It enables to have direct IO, async IO, traces of IO time... quite flexible.
    I have to say that I have used it exclusively on Linux (but the README file mentions AIX so maybe it works as well, I do not have an AIX box to test).

  4. Eric, thanks for the pointer. Look forward to trying fio out.