Monday, August 20, 2012

Oracle load testing - part 3 Hammerora results

I learned something very important when doing testing with Hammerora. The documentation is quite good and has a simple, but important point, the importance of planning and preparation. To quote the documentation for Oracle OLTP testing:

Planning and Preparation is one of the most crucial stages of successful testing but is often overlooked. Firstly you should fully document the configuration of your entire load testing environment including details such as hardware, operating system versions and settings and Oracle version and parameters. Once you have fully documented your configuration you should ensure that the configuration is not changed for an entire series of measured tests. This takes discipline but is an essential component of conducting accurate and measured tests.
From this I conclude that many tests I've seen and done myself have not been accurate. As stated earlier the goal for this testing was just to compare performance before and after migration from EVA to 3PAR.

Since this customer did not have the required license to run AWR I did the following change in the driver scipt to create a Statspack snapshot in place of a AWR snapshot. Search for string containing dbms_workload, replace
set sql1 "BEGIN dbms_workload_repository.create_snapshot(); END;"
with
set sql1 "BEGIN perfstat.statspack.snap; END;"

When testing with Hammerora I decided to run each test three times to see if the numbers where consistent. I recorded the numbers of each run in an spread sheet as shown in the following table for the tests on EVA:

Vusers Run Report tpm nopm Avg_tpm
1 1 1_2 7158 2369 7634
1 2 11_12 7874 2645
1 3 21_22 7868 2804
3 1 31_32 16478 5765 17317
3 2 41_42 17678 6256
3 3 43_44 17794 6130
10 1 45_46 27847 9959 33225
10 2 51_61 32581 11600
10 3 71_81 39248 13701
20 1 91_101 47489 17441 47075
20 2 111_121 63062 22658
20 3 131_141 30674 11116
30 1 151_161 54349 19756 44186
30 2 171_181 45628 17331
30 3 191_201 32581 12733

Vusers is the number of virtual users in Hammerora, Run is 1 - 3 for each new setting of Vusers. Report  refers to the Statspack report created on snapshots before and after. Tpm and nopm as reported from Hammerora and finally Avg_tpm is the average in each group. Compare this to the numbers for the 3PAR:

Vusers Run Report tpm nopm Avg_tpm
1 1 9_10 8246 2815 8262
1 2 11_12 7983 2717
1 3 13_14 8556 2956
3 1 15_16 22652 7854 22881
3 2 17_18 22652 7831
3 3 19_20 23339 7994
10 1 21_22 33539 11767 33191
10 2 25_26 39054 13729
10 3 27_28 26981 9428
20 1 29_30 47134 16462 47356
20 2 31_32 46436 16330
20 3 33_34 48497 17023
30 1 35_36 53197 18902 50788
30 2 37_38 44980 15994
30 3 39_40 54187 19033

The repeated tests for the same number of virtual users do not vary as much on the 3PAR compared to the EVA. Also the numbers for the EVA seemed to improve for each run, maybe due to some caching taking place.

The 3PAR seemed to be more reliable for the same number of virtual users as can be seen in these screen captures, the first for 20 virtual users on EVA:

The peformance on the 3PAR does not change much during the test (20 virtual users):

You'll see that in one instant the EVA seems to perform better, but I rather have stable and less erratic performance with the 3PAR than a system with occasional good performance.

All in all it was very easy to play around with Hammerora, it is very easy to set up so you can spend time on planning and executing the tests. Also I like how you can observe change of performance over time. Clearly Hammerora is a tool I will use more later.