Measuring Matrix Multiplication Performance on Different Operating Systems

A comparison of Windows XP vs Mandriva Linux

Chris Constantine
MSE655
April 28, 2008


Introduction

Does an operating system play a significant role in software performance?

The intent of this project is to determine the answer to this question objectively via statistical analysis. This
experiment will perform a matrix multiplication algorithm with matrices of size 100x100, 200x200, 400x400 and 700x700.


System Under Test

Compaq Presario V2000 Laptop
AMD Turion 64 ML-37 1MB L2 Cache
2 Ghz
1 GB PC3000
60 GB 5400 RPM
Dual Boot setup
Windows XP Pro
Mandriva Linux


Component Under Study

The Component Under Study in this experiment is the Operating System used to execute the application program


Experiment Design

This experiment is designed to be performed via command line application execution.
Using the Java program provided here (matrix.java), perform the test using the arguments of matrix size and number of repetitions.


Analysis

SUMMARY

100

200*

400

700

Windows

 

 

 

 

Average

12.11065321

97.83579832

819.1669547

4397.848333

Linux

 

 

 

 

Average

29.33169714

88.98707218

1536.734641

8340.853892


Mean values for each of the matrix size on each Operating System. These are the
results of 3120 measurements taken in two separate intervals, containing 5 sessions
of 321 results.
 

 

Windows

c1

c2

100

12.10103285

12.12027357

200

97.7881091

97.88348754

400

818.6751403

819.658769

700

4395.759882

4399.936785

 

Linux

100

28.75337469

29.91001959

200

88.12958294

89.84456142

400

1533.986619

1539.482664

700

8337.101127

8344.606657

These intervals show that the 99% confidence range for each level of factors do not overlap.
We can now say there is a significant different between the run times between operating systems.

ANOVA

 

 

 

 

 

 

Source of Variation

SS

df

MS

F

P-value

F crit

OS (A)

8.74687E+20

1

8.74687E+20

798193.1336

0

3.845094

Size (B)

1.75208E+22

3

5.84027E+21

5329518.077

0

2.608379

Interaction (AB)

1.70336E+21

3

5.67785E+20

518130.7974

0

2.608379

Error (E)

2.80533E+18

2560

1.09583E+15

 

 

 

 

 

 

 

 

 

 

Total

2.01016E+22

2567

 

 

 

 

As the ANOVA table shows, the sources of variation in the measurements are all significant, comparing
the F values with the critical F value given. Also, the ANOVA table gives us a way to determine the amount
of variation each item describes.

For example, the amount of variation explained by the change in matrix size is given by SSB/SST * 100%
Variation due to matrix size = 87.2%

The amount of variation explained soley by the change of operating system is givenby SSA/SST * 100%
This results in 4.35%. This means 4% of the variation in the resulting performance time is related strictly
to the change of operating systems.

Alternative Analysis Method (Regression Model)

Another way to analyze the data given is to generate a linear regression model to explain the variation and model the data to provide a method of obtaining results for different matrix sizes without running the application program. This allows analysis of different sizes of matrices when the SUT is unavailable or unable to produce the results desired.

  The graph displayed shows that the results do not depict a linear relationship.

This result shows that the data must be linearized before the regression model can be built.

 
This graph displays the linearized data. The data was transformed with a natural log function.d

Once the data has been linearized, we can compute the results based on a given input. Using a linear regression result,
the equations generated for Windows and Linux  and a value of N = 400 are as follows:

¡Windows: Yp = 16.09 + .0093(400) = 19.81
¡Linux: Yp= 16.53 + .0096(400) = 20.37

Plugging these Yp values into the interval formula gives us ranges for linearized

90% confident Yp values of:

¡Windows: 17.56, 22.06
¡Linux: 18.92, 21.82

 

Using the formula (y1,y2) = yp -/+ t * s * sqrt(1 + 1/n + (xp – xavg) ^ 2 / Sxx)
 

Windows

42197345

3814716280

Linux

164000000

3000256990

 

At 90% confidence, the ranges are pretty large, the transformation of ln x through e^x gives these values.

 

The appropriateness of the ln x as the transformation might not be the best choice, but gives the most linear result I could find.

 

The values given do in fact contain the measured values however.


Close

The results from this analysis show that yes, the operating system is a significant factor in determining the variance in performance run time.