Emulating Netbench ================== Netbench is a terrible benchmark, but it's an "industry standard" and it's what is used in the press to rate windows fileservers like Samba and WindowsNT. The big problem with netbench for the open source community is that very few people who work on open source software have the facilities to run it properly. You need a lab with between 60 and 150 windows PC, all on switched fast ethernet and a really grunty server (say a quad xeon with 1GB ram and hardware raid). Then you need some way to nurse all those machines along so they will run a very fussy benchmark suite without crashing. Not easy and very expensive. Only one person in the open source community that I know of has access to such a lab (Jeremy Allison at SGI). In order for the development methodologies of the open source community to work we need to be able to run this benchmark in an environment that a bunch of us have access to. We need the source to the benchmark so we can see what it does. We need to be able to split it into pieces to look for individual bottlenecks. In short, we need to open up netbench to the masses. To do this I have written three tools, dbench, tbench and smbtorture. All three read a load description file called client.txt that was derived from a network sniffer dump of a real netbench run. client.txt is about 4MB and describes the 90 thousand operations that a netbench client does in a typical netbench run. They parse client.txt and use it to produce the same load without having to buy a huge lab. They can simulate any number of simultaneous clients. client.txt must either be in the working directory, or specified on the command line with the -c option. dbench ------ dbench produces only the filesystem load. It does all the same IO calls that the smbd server in Samba would produce when confronted with a netbench run. It does no networking calls. You can get dbench from ftp://samba.org/pub/tridge/dbench/ You run it as "dbench N" where N is the number of clients to simulate. It gives out three numbers like this (this is from a 144 client run on a quad xeon box): Throughput 40.6701 MB/sec (NB=50.8376 MB/sec 406.701 MBit/sec) the first is the true throughput as seen by dbench. The second and third numbers are "netbench scaled" numbers that give the throughput that would be seen by Win9X boxes after taking into account the client file cacheing performed by oplocks. They are given in both MB/sec and MBit/sec as different netbench reports use different scales. tbench ------ tbench produces only the TCP and process load. It does the same socket calls that smbd would do under a netbench load. It does no filesystem calls. The idea behind tbench is to eliminate smbd from the netbench test, as though the smbd code could be made infinately fast. The throughput results of tbench tell us how fast a netbench run could go if we eliminated all filesystem IO and SMB packet processing. tbench is built as part of the dbench package. To run tbench first run tbench_srv on the server. Then run "tbench N SERVER" on the client. N is the number of clients to simulate and SERVER is the hostname of the server. The results are given in the same format as dbench. smbtorture ---------- smbtorture is the stress tester from the Samba suite. I've recently added a stress test that allows you to simulate the full netbench benchmark, including all network traffic and filesystem IO. To run smbtorture you first need to install Samba version 2.0.X. The binary distrubtions at ftp://ftp.samba.org/pub/samba/bin-pkgs/redhat/ would do fine if you don't want to compile it yourself. Then setup a netbench share on the fastest disk you have, making sure you have at least 25MB free per simulated client. The simplest smb.conf would look something like this: [global] security = share [netbench] path = /data/netbench read only = no guest ok = yes Then you need smbtorture. You can either grab a precompiled i386 Redhat5.2 binary from ftp://samba.org/pub/tridge/dbench/smbtorture.gz or you can follow the instructions at http://samba.org/cvs.html to download the SAMBA_2_0 branch of the Samba cvs tree and use "make smbtorture" to build it. Finally, you'll need client.txt from ftp://samba.org/pub/tridge/dbench/dbench.tgz in the same directory that you run smbtorture from. To run it you do this: smbtorture //localhost/netbench -U% -N 32 NBW95 that will run a 32 client load. You can, of course, also run smbtorture against a dfferent SMB server (such as a NT server) to give comparitive results for any client load that the server can handle. Even better is to run smbtorture on one machine and smbd on another, connected by a very fast network (such as gigabit ethernet). That will stop the smbtorture code itself from loading the server and will also test the network driver instead of the loopback driver. results ------- To give you an idea of what to expect here are some results on a quad xeon machine running Linux 2.2.9 with 1GB of memory (it has 4GB but Linux only uses 1GB by default). The machine also has a 45GB hardware raid system and a Alteon AceNIC gigabit network card. The results below are in netbench MB/sec (the NB= number in the result). Multiply by 8 to get the MBit/sec numbers that Mindcraft used in their graphs. The first column is the number of simulated clients, the second is the result. With dbench I get: 16 118 32 103 48 81 64 74 80 67 96 57 112 57 128 54 With tbench on loopback I get: 16 31 32 30 48 29 64 29 80 28 With tbench running across the gigabit network (using a dual processor Origin200 as the client) I get: 16 29 32 29 48 29 64 29 With smbtorture running over loopback I get: 16 18 32 18 48 18 64 17 80 16 With smbtorture running across the gigabit network (using a dual processor Origin200 as the client) I get: 16 16 32 20 48 21 64 20 80 19 With smbtorture running across the gigabit network but with smbd modified so that write_file() and read_file() are null operations (which eliminates the file IO) I get: 16 17 32 22 48 24 64 24 80 24 The above results show that, at least for this hardware configuration, the problem isn't the filesystem code or the raid drivers. More tests will be needed to find out exactly what the problem is but it looks like a TCP scaling problem. comparison with NT ------------------ Hopefully Jeremy will be able to run smbtorture against NT on the same hardwre sometime in the next week so we have direct numbers for comparison, but from looking at the mindcraft numbers under netbench we would expect NT to get about the following: 16 8 32 14 48 21 64 27 80 31 so we do well by comparison with small client loads but fall behind quite a lot with large loads. Note that the numbers in the mindcraft report for Linux/Samba are quite a long way behind what I give above because mindcraft did a hopeless job of tuning Linux/Samba. comparison with netbench ------------------------ An initial comparison with real netbench results shows that smbtorture does produce very similar throughput numbers. They aren't exactly the same but they are similar enough for us to target our tuning efforts and expect to see improvements reflected in real netbench runs. When we find something that looks promising we can get Jeremy to run a real netbench test. a quick hack ------------ the tbench results really pointed at the problem being the Linux TCP stack. I made a quick (and very unsafe!) hack to Samba and the Linux kernel to see if I could remove the lock_kernel() in sys_sendto() and sys_recvfrom() for smbd processes by passing a MSG_NOLOCK flag in send() and recv(). That gave an enormous improvement in the loopback tbench results: 16 31->51 32 30->49 48 29->47 64 29->46 80 28->44 and in the loopback smbtorture results I also saw a big improvement: 16 18->26 32 18->27 48 18->26 64 17->25 80 16->24 that's a 50% improvement. I suspect the numbers will be higher with a real netbench run as it won't have the overhead of running 80 smbtorture clients on the same box as the server. realistic load? --------------- One question some people may ask is whether the above represents a realistic load on a fileserver. It doesn't. Nearly 90% of the read/write IO operations in netbench are writes whereas in a "normal" office load reads dominate. Also, the load is *much* higher than a normal office PC would put on a server. There aren't many office PCs that write over a hundred megabytes of data to a server in a few minutes, unless maybe they are copying a CD. That doesn't mean the benchmark is useless, it just means you shouldn't use this for purchasing decisions unless you really understand the results and how they relate to your environment. license ------- smbtorture and dbench are released under the terms of the GNU Public License version 3 or later. Andrew Tridgell tridge@samba.org
