Throughput Analysis of E-commerce Servers Using the TPC-W Benchmark

Throughput Analysis of E-commerce Servers Using the TPC-W Benchmark

Javier Garca
Computer Science Department
University of Oviedo (SPAIN)
+34 985 18 22 26
javier@atc.uniovi.es
Daniel F. Garca
Computer Science Department
University of Oviedo (SPAIN)
+34 985 18 20 66
daniel@atc.uniovi.es
Joaqun Entrialgo
Computer Science Department
University of Oviedo (SPAIN)
+34 985 18 19 66
joaquin@atc.uniovi.es
Manuel Garca
Computer Science Department
University of Oviedo (SPAIN)
+34 985 18 25 19
manuel@atc.uniovi.es
Antonio M. Campos
Computer Science Department
University of Oviedo (SPAIN)
+34 985 18 25 18
campos@atc.uniovi.es

ABSTRACT

Performance is a crucial aspect in the quality of service of e-commerce sites. Although many factors may affect performance, in this paper we focus on the role computer servers play in the performance of e-commerce sites. The paper presents an experimental study into how system factors (software and hardware), as well the external authentication service influence the throughput provided by e-commerce servers. The study was carried out using the TPC-W benchmark as a synthetic workload to test the servers. The results obtained from the study highlight certain behavioral aspects of servers which may provide valuable insight useful for the performance studies required to set up e-commerce sites.

Keywords

Benchmarking, TPC-W, E-commerce Servers.

1. THE TPC-W BENCHMARK

The TPC-W e-commerce benchmark [1, 2] has client-server architecture. Figure 1 shows the elements which make up the architecture of the benchmark. The SUT (System Under Test) includes all the components that constitute a typical e-commerce server. The RBEs (Remote Browser Emulators) emulate the workload that real customers would generate. The PGE (Payment Gateway Emulator) emulates the entity which authenticates the users and authorizes the payments. The SUT, the PGE and the RBEs communicate through a dedicated network.

In TPC-W the Workload is modeled as a mix of web interactions. Examples of these interactions are search and browse products, add products to the shopping cart, and buy products. Different mixes of web interactions are used to model the behavior of different types of clients. These mixes are referred to as user profiles. In TPC-W three user profiles are defined: Browsing, Ordering and Shopping.

In addition to the user profile, another crucial configuration parameter of TPC-W is the number of items to sell. Five values can be selected: 103, 104, 105, 106 and 107.

The metric reported by TPC-W is the number of Web Interactions Per Second (WIPS) measured using the shopping user profile as the workload.

Figure 1. The TPC-W benchmark architecture

2. EXPERIMENTAL RESULTS

The experimental work presented in this paper was developed using an Intel Pentium-III tetra-processor running the Linux operating system. The database used was ORACLE and the HTTP server was Apache, connected by the PHP tool. In all the experiments, the benchmark was configured to use the shopping user profile as the workload and 105 items to sell in the database.

2.1 Influence of system factors in WIPS

The system factors, shown in Table 1, can be classified in two broad groups: hardware factors and software factors. The hardware factors are mainly ordered quantitative factors, whose levels are expressed numerically in increasing order. The hardware factors analyzed are the number of CPUs and the amount of RAM.

The software factors are mainly unordered qualitative factors, mainly associated to the version or release of each software component of the system or regarding their configuration parameters.

Sys Factor Intel SC450NX Platform Levels
CPU Pentium III-Xeon 450 MHz 1, 2, 3, 4
RAM EDO ECC 133 MHz 256 ... 1536
DISK SCSI Seagate Cheetah 18 GBytes (10,000 rpm 5.4 ms seek time) --
Op. System Linux, RedHat V6.2, kernel 2.2.18 --
Http server Apache V1.3.14 --
Data Base Oracle 8i V8.1.7 dedicated servers --
CGI PHP V4.0.4 permanent connections --
Table 1. System factors affecting the WIPS

To evaluate the influence of hardware system factors on WIPS, a set of experiments with the number of processors varying between 1 and 4 and the memory installed varying between 128 MB and 1,5 GB was carried out. Figure 2 shows the results of the experiments.

Figure 2. Influence of hardware system factors in WIPS metric

When the minimum amount of memory which allows the database to operate is installed (curve of 128MB), the number of processors used has no influence on WIPS. In other words, under the maximum memory contention conditions, the scalability of the e-commerce server is null.

With a sufficiently high amount of memory (curve of 512MB), the addition of processors always generates an increment of WIPS, although not in a linear manner. On this curve the server works under minimum or unnoticeable memory contention conditions. With more memory installed (curve of 1536MB), there is a total absence of memory contention.

The evaluation of the influence of software system factors involves many configuration parameters of the operating system, web server and database software. However, the default installation values for these parameters generally provide nearly optimum performance. Not so with the data base queries. The optimization of the queries is obtained by indexing the database, which is carried out with the following rule: when a field of a table is referenced in one or more SQL clauses, a single index is created using the field. The results presented in Figure 2 were obtained without optimizing the queries. Figure 3 shows the equivalent evolution of the WIPS metric when single indexes were added to the tables of the database: single indexing of tables allows an increment of WIPS of up to 300%.

Figure 3. Influence of system factors (single indexing) on WIPS metric

2.2 Influence of the authentication service in WIPS

The authentication service is provided by the Payment Gateway Emulator (PGE). To evaluate if the PGE represents a brake in performance, WIPS were measured with and without PGE under two optimization levels of the database, indexed and not indexed. The results are shown in Figure 4.

Figure 4. Influence of authentication services in WIPS metric

When the database is not indexed, the WIPS, with and without the PGE, are the same. They are represented by the two nearly identical curves in the lower part of Figure 4. However, when the database is indexed, the authentication services produce a reduction of 10% in the WIPS.

3. ACKNOWLEDGEMENTS

This work was supported by the Spanish Research, Development, and Innovation Program under project TIC2001-1374-C03-03.

4. REFERENCES

  1. D.A. Menasc. TPC-W: A Benchmark for E-commerce IEEE Internet Computing, vol. 6, no. 3, 2002, pp. 83-87.
  2. Transaction Processing Performance Council. TPC-W Benchmark Specification. August 2001. http://www.tpc.org.