(Solved) Serious GCC regression

[muungwana]

The test seem to produce different numbers on every run

i run the test using gcc and ruby both from the repository and the numbers produced are on table one.

i run the test again using ruby build from source using gcc version from the repository and the results are on table two

i run the test again on ruby build from source using gcc version 4.4.4 build from source and the results are on table three

ruby version used: 1.8.7
gcc version from the repository used: 4.4.1

conclusion reached.
building ruby from source seem to produce lower numbers but the version of gcc used doesnt seem to a factor

table1:
primes up to 10000001:19.250000   2.330000  21.580000 ( 26.314624)
primes up to 10000001:  29.390000   4.690000  34.080000 ( 41.521611)
primes up to 10000001:  10.280000   1.230000  11.510000 ( 13.976745)
primes up to 10000001:   0.000000  15.110000  15.110000 ( 18.336620)
primes up to 10000001:   0.000000  11.460000  11.460000 ( 13.877479)
primes up to 10000001:   0.000000   9.900000   9.900000 ( 11.971750)
primes up to 10000001:   0.000000  11.010000  11.010000 ( 13.361440)
primes up to 10000001:   0.000000  15.060000  15.060000 ( 18.302800)
primes up to 10000001:   0.000000  11.630000  11.630000 ( 14.094769)
primes up to 10000001:   0.000000   9.720000   9.720000 ( 11.773007)
primes up to 10000001:   0.000000  11.360000  11.360000 ( 13.719820)

table two
primes up to 10000001:  12.930000   0.140000  13.070000 ( 15.952930)
primes up to 10000001:  18.250000   0.190000  18.440000 ( 22.360552)
primes up to 10000001:   7.000000   0.080000   7.080000 (  8.604609)
primes up to 10000001:   9.840000   0.110000   9.950000 ( 12.018939)
primes up to 10000001:   7.530000   0.100000   7.630000 (  9.296197)
primes up to 10000001:   6.310000   0.070000   6.380000 (  7.760052)
primes up to 10000001:   0.000000   7.290000   7.290000 (  8.844797)
primes up to 10000001:   0.000000   9.480000   9.480000 ( 11.630735)
primes up to 10000001:   0.000000   7.280000   7.280000 (  9.128416)
primes up to 10000001:   0.000000   5.970000   5.970000 (  7.254754)
primes up to 10000001:   0.000000   7.110000   7.110000 (  8.611752)

table three
primes up to 10000001:  13.190000   0.150000  13.340000 ( 16.254554)
primes up to 10000001:  17.940000   0.170000  18.110000 ( 21.855186)
primes up to 10000001:   7.090000   0.110000   7.200000 (  8.720106)
primes up to 10000001:   9.370000   0.120000   9.490000 ( 11.517667)
primes up to 10000001:   7.050000   0.080000   7.130000 (  8.684816)
primes up to 10000001:   5.890000   0.070000   5.960000 (  7.197289)
primes up to 10000001:   0.000000   6.860000   6.860000 (  8.355789)
primes up to 10000001:   0.000000   9.300000   9.300000 ( 11.228579)
primes up to 10000001:   0.000000   7.140000   7.140000 (  8.629654)
primes up to 10000001:   0.000000   5.970000   5.970000 (  7.264170)
primes up to 10000001:   0.000000   7.760000   7.760000 (  9.412921)

[jzakiya]

Here are reference results using Ruby 1.9.1-378p which I had been primarily using.

Tuesday 2010/5/18
A - XP Ruby 1.9.1-378 build i386-mingw32/EratSievesY.rb
B - PLCOS 2010.1 Ruby 1.9.1-378 gcc 4.4.1/EratSievesY.rb

    A                   B
4.025789     17.860419
6.309072     25.876720
2.123052       9.828090
2.683860     12.188742
2.133067       9.628692
1.752520       7.572812
2.022909       9.614535
2.723916     11.951083
2.123053       9.076395
1.742506       7.677763
2.062966     10.361113

The results in column A under Win XP were similar to what I had been getting
with 2009 under Ruby 1.9.1-378 compiled from source.

The results I was getting under 2009 for a separate compiled version of 1.8.7
were between 2-3 times slower, which is the native performance difference
between 1.9.1 and 1.8.7 (meaning 1.9.1 is inherently 2-3 times faster than 1.8.7 on
these benchmarks).

The results muungwana first posted on his P4 are about 2-3 times slower than I
was getting for my 2009 build for 1.8.7, so I would say his present build has the
same problem I am experiencing with 2010.

His subsequent posts of table 1-3 also certainly indicates to me the great difference
between the repos build of 1.8.7 and the source compiled versions.  His table 2 is much
closer to what I would have gotten under my old 2009 system for 1.8.7.

Mellon's results also indicate to me that native 2010 configuration of gcc 4.4.1 is more optimized for the newer multi-core processors, as his results are closer to what I would expect, but still are slower than what I was getting on my P4 under 2009.

Given the empirical data shown here, I contend even stronger now that the difference(s) between the 2010/2009 builds has caused a regression on my P4 system, especially since the XP build using a i386 code build is demonstrably faster.

What would show this more clearly is to see the benchmarks on a 2009 P4 system for Ruby compiled from source with that gcc system (which includes how the config and make processes are handled).

I have been reading the gcc back forums and have seen posts identifying 4.4 series regressions with C code (Ruby source is C code) when using some loop structures. My benchmarks are highly loop intensive.  GCC 4.5 (which was recently released) is supposed to have fixed these problems, and some problems with compiling code that doesn't strictly adhere to the new C standard protocols (not being a C person I don't know exactly how that would effect the Ruby C source tree, but I assume the Ruby core people do).

I will compile my benchmarks using the gcc 3.3 compiler in the repos and see what the differences would be, and I have downloaded GCC 4.5 source too, and am debating to compile and use that.  Of course, even by installing the repos gcc 3.3, it is still being compiled (suboptimally for my P4) with the 4.4.1 compiler. Classic chicken/egg problem.

Also, I see no technical reason why you need to reformat /home.  The HD install is not supposed to touch your /home, and it completely reformatted the / partition.  I don't see why this is suggested as a solution to this problem when I never had to do this going from 2007 to 2009.  I would appreciate a technical explanation of why this is being cited as a solution to my whole system being much slower.

However, I am completely open to suggestions that directly address why my system is demonstrably slower than my 2009 system.  I'm all ears! :-)

But before I even consider taking the 'nuclear option' and wiping out everything and starting over again, I would want to be ABSOLUTELY SURE this would be a solution to the  problem, i.e., producing a system that would (minimally) be just as fast as my 2009 system.

[muungwana]

2010 ships with a kernel that uses BFS schedular, maybe the problem is with the scheduler. Try to change the kernel to  the one that uses linux native scheduler and see if it will make a difference.

[gseaman]

I believe your problem is real and hope that together it can be resolved. I still say that programs that have been compiled with 4.4.1 have not had an overall performance decline. If there is a regression with gcc, maybe it only affects a specific program or set of programs and most of us have not encountered it.

However, I still think it is likely that your system is not working optimably. The reason for reformatting / and /home is that leftover configuration files can cause a multitude of performance issues and would be nearly impossible to track down. With a clean system, problems can be isolated and fixed. If you want to try many things first, that is fine and you will learn a lot about how things interact, it's just not always efficient that way. Good luck!

Galen

[jzakiya]

I installed from the repos gcc3.3 (actually 3.3.6), but it will not run
and has an executable file on the order of 32,000 byte, whereas the
gcc 4.4.1 has an executable on the order of 150,000 bytes.

So, I cannot recompile the languages to see what performance differences I would get.

But in continuing to understand this performance regression issue, if the
compiler hasn't  changed (gcc 4.4.1) since 2009, certainly the support
files (libraries, headers, helpers, etc) for the compile process has, and/or
the system build process.

From looking at the output of  $ gcc -v (shown in my original post), there
are a whole lot of compiler flags apparently used, and I don't remember there
being that many used with gcc under 2009.

FYI here are links to whats up with gcc 4.5, and differences with 4.4, with
the aime to produce more optimized (performance) executable binaries:

http://lwn.net/Articles/387122/
http://gcc.gnu.org/gcc-4.5/changes.html

For those who run the benchmark file I provided, be sure to close any browser, IM,
or anything that uses memory, or is active when the benchmark is running.  I noticed
just having FireFox open when the benchmark is running will degrade times appreciably.

If I can't get this issue fixed I probably will revert back to 2009, or look for another until its
resolved, which is a shame because I'm just getting familiar with the new KDE4 goodies.
I'm writing this on my Win XP boot, because it takes just too much d**n time to do anything
with FireFox now (even Chromium sputters) that I'm forced to use XP if I just want to get work done.

Sign..... :'(

[muungwana]

My test hast shown me the compiler isnt the issue and this leads me suspect the problem could be with the kernel. i cant rule this one out if you dont test on a different kernel. I cant change mine cause i have my stuff that might break if i change it and i dont want to try it

The default kernel in 2010 uses BFS scheduler.
The default kernel in previous versions uses CFQ scheduler.

The problem could be with the scheduler in use. Try to change the kernel to the one that uses CFQ and run your test again.

[mellon]

These are the results on the same HP machine with dual core as before but with the default kernel with BSF scheduler.  1Gb less memory compared to PAE.   

Code: [Select]

ruby EratSievesY.rb
EratSievesY.rb:1877: warning: statement not reached
                    user     system      total        real
primes up to 10000001:  10.990000   2.450000  13.440000 ( 13.547189)
primes up to 10000001:  15.750000   5.010000  20.760000 ( 20.933011)
primes up to 10000001:   5.760000   1.320000   7.080000 (  7.137281)
primes up to 10000001:   7.770000   1.630000   9.400000 (  9.483368)
primes up to 10000001:   5.750000   1.300000   7.050000 (  7.101025)
primes up to 10000001:   4.850000   1.120000   5.970000 (  6.014364)
primes up to 10000001:   5.500000   1.100000   6.600000 (  6.653810)
primes up to 10000001:   7.800000   1.620000   9.420000 (  9.489824)
primes up to 10000001:   5.800000   1.270000   7.070000 (  7.141448)
primes up to 10000001:   4.830000   1.130000   5.960000 (  5.994875)
primes up to 10000001:   5.780000   1.130000   6.910000 (  6.964769)

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579
And this is a run on my 7 year old desktop Dell Dimension 4550 with 1Gb memory with same default BFS kernel.

Code: [Select]

ruby EratSievesY.rb
EratSievesY.rb:1877: warning: statement not reached
                    user     system      total        real
primes up to 10000001:  16.190000   3.380000  19.570000 ( 20.202296)
primes up to 10000001:  25.470000   7.090000  32.560000 ( 33.597993)
primes up to 10000001:   8.980000   1.770000  10.750000 ( 11.070494)
primes up to 10000001:  11.810000   2.210000  14.020000 ( 14.477141)
primes up to 10000001:   8.970000   1.740000  10.710000 ( 11.021997)
primes up to 10000001:   7.600000   1.500000   9.100000 (  9.390687)
primes up to 10000001:   8.430000   1.580000  10.010000 ( 10.308822)
primes up to 10000001:  11.410000   2.200000  13.610000 ( 14.692460)
primes up to 10000001:   9.070000   1.840000  10.910000 ( 11.260996)
primes up to 10000001:   7.550000   1.470000   9.020000 (  9.314777)
primes up to 10000001:   8.710000   1.570000  10.280000 ( 10.585288)

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

[9999889, 9999901, 9999907, 9999929, 9999931, 9999937, 9999943, 9999971, 9999973, 9999991]
664579

Mellon

[jzakiya]

OK, I installed the 2.6.33.4 kernel and.....didn't make a difference, still slow as molasis.
i
As a sanity check, I ran the LIVE 2010.1 CD.
Ruby 1.8.7 on it ran the Ruby benchmark much, much faster than the installed version.

Firefox is absolutely on steroids compared to its HD installed performance.
Same with everything else.

How can the LIVE CD performance be soooooo much faster than the HD installed system?

I'm next going to see if I can get gcc 4.5 to work (since installing gcc3.3 from repos didn't even work, and I got no feedback on how to make it work).

[kjpetrie]

You cannot simply install a new version of GCC. It is part of the GNU toolchain which is the basis of any GNU/Linux system. The whole toolchain needs to be built together in a recursive process (see Linux from Scratch for details).

The fact it runs fast on the livecd clearly demonstrates the problem is in your hardware/BIOS/HDD setup and the way the OS interacts with it and not in the OS itself. Normally livecds run more slowly, so something is making the installed version slow on your machine. You need to investigate that by comparing the livecd with the installed version (running processes, Load average, etc) to solve this.

[muungwana]

You cannot simply install a new version of GCC. It is part of the GNU toolchain which is the basis of any GNU/Linux system. The whole toolchain needs to be built together in a recursive process (see Linux from Scratch for details).

you can actually. You can install a new version, build program using a newer version and the program will run without complains. How else will third party programs like firefox or the generic virtualbox version from their website work?

I dont know what constitute the "toolchain" but tying them up together and fix them in place is not required.

@jzakiya
It will be wise if you do not replace the version from the repos with yours if you want to check out gcc 4.5. Install it somewhere else like /usr/local

[jzakiya]

OK, found the problem. 

I was in PCLOS and restarted (versus shutdown) the system, and booted into XP at the restart. I ran the Ruby benchmark and got the same slow times I was getting in Linux.  Immediately I knew what was happening.

I shutdown and booted into PCLOS this time, and went into the PCC and looked at hardware settings.  Sure enough, for the cpu settings, even though it has 2.8Ghz, the performance setting showed it was running at 699Mhz, i.e. 1/4 of its top speed, which corresponded to the slowdown seen in the Ruby/Python benchmarks I had been running, and explained completely the general slowdown of everything.

So then I went to System settings in PCC, and first stopped the cpufreq setting, but on reboot the clock was still slow. So then I turned off the acpid setting, rebooted, and the cpu was at full speed.  To check, I turned back on the cpufreq setting and the cpu remained at full speed.

So the problem wasn't the gcc compilation, or not reformatting /home, but the acpid settings which set my cpu speed at 1/4 full speed.

Actually, it was because I had faith that it must have been something really simple, that I didn't take radical measures to do things that would have been harmful/useless.

I think people should be instructed to look at their cpu performance setting via PCC and make sure its running at full speed, and if not, apply the fix that worked in this case to possibly correct it.

It's so nice to get back my real laptop. 

[gseaman]

I glad to see you got it fixed! Congrats.

galen

[menotu]

I think people should be instructed to look at their cpu performance setting via PCC and make sure its running at full speed, and if not, apply the fix that worked in this case to possibly correct it.

Very good suggestion.

Pleased your now running at full speed. (just remember, speed kills     )