Friday, 14 March 2014

Apache Hama Benchmark Test at LG CNS

In latest version 0.6.4 of Apache Hama, the memory efficiency has been greatly improved. Since PageRank is an good example of communication-intensive processing, we've benchmarked it.

This PageRank Benchmark test performed by Victor Lee and David Min of the Future Tech team at LG CNS. Thanks!

2 Node Hama Cluster Test:
  • H/W : 16 Cores and 14GB memory per node 
  • 6 tasks (opt : -Xmx2048m, -Xmx4096m)
verticesedges
per vertex
Hama 0.6.3Hama 0.6.4
-Xmx2048m-Xmx4096m-Xmx2048m-Xmx4096m
40000500 145.301 seconds121.167 seconds208.423 seconds196.296 seconds
40000600169.238 seconds160.213 seconds253.359 seconds241.447 seconds
40000700232.757 seconds190.421 seconds289.457 seconds271.362 seconds
40000800OutOfMemory220.553 seconds334.555 seconds328.391 seconds
40000900 253.797 seconds376.871 seconds385.595 seconds
400001000 280.575 seconds433.763 seconds418.516 seconds
400001100 323.785 seconds488.072 seconds466.849 seconds
400001200 368.119 seconds575.149 seconds511.822 seconds
400001300 398.843 seconds662.303 seconds554.057 seconds
400001400 469.371 secondsOutOfMemory649.482 seconds
400001500 678.136 seconds 670.641 seconds
400001600 OutOfMemory 729.434 seconds
400001700   780.629 seconds


Single Node Hama Cluster Test:
  • H/W : 16 Cores and 14GB memory per node 
  • 3 tasks (opt : -Xmx2048m, -Xmx4096m)
verticesedges
per vertex
Hama 0.6.3Hama 0.6.4
-Xmx2048m-Xmx4096m-Xmx2048m-Xmx4096m
4000010049.065 seconds49.016 seconds75.981 seconds75.967 seconds
4000020097.189 seconds91.185 seconds148.238 seconds139.115 seconds
40000300160.341 seconds133.155 seconds229.183 seconds226.278 seconds
40000400OutOfMemory193.216 seconds310.489 seconds301.418 seconds
40000500 265.385 seconds412.665 seconds382.675 seconds
40000600 322.875 seconds514.926 seconds460.638 seconds
40000700 441.495 secondsOutOfMemory568.985 seconds
40000800 OutOfMemory 665.094  seconds
40000900   764.596  seconds

No comments:

Post a Comment