3

We discovered an I/O related performance problem when using the default SLES11 SP2 kernel. Our same application on the same hardware had no issue with (the admittedly ancient SLES9 SP3).

We had a feeling it has something to do with the kernel and have done the following experiment so far:

  1. Downloaded and unpacked the kernel.org 3.0.13 source tree.
  2. Installed Novell's kernel-source-3.0.13-0.27.1 package. This is the 3.0.13 source with all of Novell's patches/tweaks already applied.
  3. In the 3.0.13 tree did a make x86_64_defconfig to generate a default config file, then used make menuconfig to turn on the drive controller and network card drivers our hardware needed and built the kernel. This kernel had no performance problem.
  4. In the 3.0.13-0.27.1 tree we got the config file that Novell ships with their compiled kernel, built with it, and had the same performance problem we had with the Novell-compiled kernel.
  5. In the 3.0.13-0.27.1 tree, took the config file used in (3), built with it (accepting the additional config defaults that Novell's version of the configurator wanted to add). There was no performance problem.
  6. In the 3.0.13 tree, took the config file used in (4), built with it (losing the Novell-only config options in the file). There was no performance problem.
  7. In the 3.0.13-0.27.1 tree, took the config file used in (4), and made the device drivers (but nothing else) match the config file in (3) (i.e. turned off lots and lots of device drivers that weren't being used anyways). There was no performance problem.

So the kernel.org 3.0.13 tree had no performance problem either way. Novell's 3.0.13-0.27.1 tree only had the problem when we built with Novell's config file or a file that was identical to Novell's but with most of the device drivers de-configured.

Given that, it would seem to be that there's either some Novell-only config option (and implicitly the Novell code it activates) causing the problem, or some bad interaction between a Novell patch and some standard option (given that the as-close-as-possible-to-Novell's-config did not have the problem when built against the virgin kernel.org source).

While we continue to investigate, since I am no kernel hacker (I know how to build the kernel but that's about it) I was wondering what "families" of config options would be the best ones to focus attention on. The idea would be to change some options from the "Novell" setting to the "kernel.org" setting and see if the problem goes away, then set them back and try another set, and so on.

But I'd like to narrow that down -- hence the question about which config options would be good bets to play with first.

Improve this question
3
  • 3
    Run diff on the generated .config files to see what the differences are. Commented Nov 13, 2012 at 1:50
  • 1
    SuSE is not part of Novell any longer. Apart from that - what IO (network, disk, ram, ...) are you talking about and how did you measure that problem? Commented Nov 13, 2012 at 21:48
  • It's historic now, but it would have been interesting what the exact problem was, and how the performance was measured. Commented May 3, 2022 at 8:38

1 Answer 1

Reset to default
1

We solved the problem! Doing a quasi binary search on the config options differences between "good" and "bad" kernels, we found out that the problem is due to SLES11 turning on ext3 write barriers by default. A kernel built with write barriers off by default did not have the problem. And when we used a kernel with the problem but put barrier=0 in the mount options in /etc/fstab, the problem went away.

Improve this answer
4
  • 1
    Just make sure you know what that change means. I seem to remember some scary silent data corruption issues around that timeframe. No newer kernels available either? Commented Jan 24, 2013 at 17:58
  • barriers off means that fsync (and others) are satisfied by flushing buffer cache to the volatile on-disk cache. Losing power means that you could lose data in that disk cache that the client process was told had been flushed to stable storage. Commented Feb 14, 2013 at 21:08
  • We understand the risks in having them off. However, for our use case the performance with them on is unacceptable. Commented Feb 16, 2013 at 1:12
  • Wouldn't it have been much easier to use different journaling options in /etc/fstab instead? Commented May 3, 2022 at 8:40

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.