Database Benchmarking

.!.

前阵子跟kaya

聊天,说到benchmark测试和POC,这些都是实际项目中经常会遇到的概念,我们在研发环境很少会遇到,如果有机会实际参加这样的项目体验一把,也未尝不是一件好事。今天转载一篇Oracle Wiki

上的关于Database Benchmarking download flatliners 的文章,当然跟具体的数据库无关。这篇文章从下面几个方面介绍了数据库基准测试(姑且这么翻译):

什么是数据库基准测试

为什么要做数据库基准测试

业界标准的数据库基准测试(也就是传说中的TPC)

数据库基准测试的开源工具和商业工具

数据库基准测试书籍

原文链接

The following is my intro information for database benchmarking…

Reasons for Benchmarking
Benchmarks are performed for various reasons. However, benchmarks are primarily used:

* To compare different hardware configurations
Benchmarks can be used to compare the relative performance of different hardware running the same application. This is generally used to directly compare hardware configurations between two hardware vendors.

* To compare different database vendor software
By running the same benchmark using different database software on the same machine, one can easily compare between different database vendors. This is generally used to make a price/performance decision between vendors such as Oracle, Microsoft, IBM, etc.

* To compare different database software releases
Similar to the above, one can use different versions of the same vendor’s database software to compare the one they want to use or check for performance regressions due to upgrades (i.e. 10g vs. 11g).

Real World Benchmarks
While there are several industry standard benchmarks, most of the time, they don’t reflect a company’s actual workload. As such, you will often find custom benchmarks being performed. The goals of a real-world benchmark are the same reasons mentioned above.

Industry Standard Database Benchmarks
Industry standard benchmarks are generally used by businesses to compare different hardware and software system performance for purchase-related reasons. While industry standard benchmarks are modelled after real-world workloads, they rarely reflect a company’s real workload. However, it is a quick way to compare the performance of various hardware and software combinations operating in a well-defined scenario. The major industry standard benchmarks are listed below.

Transaction Processing Council (TPC)
The TPC is a non-profit corporation which supports a consortium of hardware and database software vendors devoted to defining transaction processing and database-related benchmarks. The primary goal behind TPC benchmarks is the definition of functional requirements which can be run on any database, regardless of the hardware or operating system. This allows vendors to implement their own benchmark kits in order to satisfy the functional requirements. Similarly, after having publicly submitted proof that a benchmark was conducted according to the specification, end-users have more of an assurance that what they are presented with is a valid, apples-to-apples comparison.

* TPC Benchmark C (TPC-C)
The TPC-C simulates an order-entryenvironment where a population of terminal operators executes transactions against a database. The benchmark is comprised of transactions which include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. The most frequent transaction consists of entering a new order which, on average, is comprised of ten different items. Each warehouse tries to maintain stock for the 100,000 items in the Company’s catalog and fill orders from that stock. The performance metric reported by TPC-C measures the number of orders that can be fully processed per minute and is expressed in tpm-C. The TPC-C will soon be deprecated in favor of the TPC-E.

* TPC Benchmark E (TPC-E)
The TPC-E benchmark simulates the OLTP workload of a brokerage firm. The focus of the benchmark is the central database that executes transactions related to the firm’s customer accounts. The TPC-E metric is given in transactions per second (tps). It specifically refers to the number of Trade-Result transactions the server can sustain over a period of time.

* TPC Benchmark H (TPC-H)
TheTPC-H is a decision support benchmark which consists of several business-oriented ad-hoc queries and concurrent data modifications. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions.

The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries.

Standard Performance Evaluation Corporation (SPEC)
The SPEC is a

* SPEC jAppServer
The TPC-C is

Free and Open Source Benchmark Kits
As the benchmark kits used by various database vendors are proprietary, several open source database benchmark kits have been written to implement not only the industry standard workloads, but others as well.

* Open Source Development Labs Database Test Suite
The OSDL DBT suite is the most comprehensive of all open source benchmark kits. While its benchmark implementations are based on TPC standards, they differ in some areas and are not certified; making a comparison between a certified TPC-C result and OSDL’s implementation of TPC-C (DBT-2) impossible. As the original OSDL DBT suite lacks Oracle support, Oracle has added and released it as part of the Oracle Linux Test kit.

* PolePosition
PolePosition is an open source benchmark test suite to compare database engines and object-relational mapping technology.

* jTPCC
jTPCC is an open source Java implementation of the TPC-C benchmark. While it claims compliance with TPC-C, it is not.

* BenchmarkSQL
BenchmarkSQL is an open source, easy to use JDBC benchmark which closely resembles the TPC-C standard for OLTP; it was originally based on jTPCC.

* Bristlecone
Bristlecone is a simple SELECT+INSERT/UPDATE/DELETE throughput and response-time benchmark written in Java.

* FinTime
FinTime is an open source set of data and queries which reflects the needs of financial analysts who are studying patterns in stock market data, but it should appeal to the designers of any system that has pretensions of handling ordered data well.

* Java TPC-W Implementation
This open-source implementation is based on the TPC-W specification version 1.0.1. It strays from the official benchmark specification in a few minor areas, which are specified in documentation included with the distribution. This version itself lacks Oracle support, but it can be found on the web.

* Swingbench
Swingbench is a free load generator (and benchmarks) designed to stress test an Oracle database (9i,10g,11g).

* OpenLink ODBC Bench
OpenLink ODBC Bench is an open-source ODBC Benchmarking tool providing real-time comparative benchmarking for ODBC Drivers, Database Engines, and Operating Systems combinations. The Benchmarks in this application are loosely based on the TPC-A and TPC-C standard benchmarks, with modifications to specifically test the performance of an ODBC Driver and/or Database Engine in a client/server environment.

* OpenLink JDBC Bench
Similar to OpenLink’s ODBC Bench, this is a JDBC version.

* TPCC-UVa
A free, open-source implementation of the TPC-C Benchmark.

* Hammerora
An open source load generation tool.

Commercial Benchmark Kits
There are several industry standard benchmarks.

* Official Oracle Benchmark Kits
Similar to Microsoft and IBM, the official benchmark kits used by Oracle can only be obtained under certain conditions. For more information, contact your Oracle Consulting/Partner/Support representative.

* Real Application Testing
While many wouldn’t think of Real Application Testing as a traditional benchmark kit, it does perform workload execution primarily for the real world benchmarking case.

* Quest Software’s Benchmark Factory for Databases
Benchmark Factory for Databases is a workload simulation and scalability testing tool that stress tests your environment by simulating users and transactions on the database. You can either replay production workload or use synthetic workload to test the limits of your database.

* SPEC jAppServer Benchmark Kit
This can be acquired directly from SPEC.

Benchmarking-related Books
There are several good books regarding benchmarking. The best I’ve found include the following:

* The Benchmark Handbook: For Database and Transaction Processing Systems (Jim Gray)
A great, albeit outdated book. While several of the benchmarks it discusses have been deprecated, it presents good concepts and background information regarding benchmarking.

* Performance Analysis of Transaction Processing Systems (Wilbur H. Highleyman)
While out of print, this book presents both plain-text and math-heavy concepts of queueing and performance analysis techniques oriented for OLTP.

* Database Benchmarking: Practical Methods for Orac

This entry was posted in 数据库 and tagged , . Bookmark the permalink.

4 Responses to Database Benchmarking

  1. Kaya says:

    总结得不错,很多有用的链接,
    P.S., Real World Benchmarks 基本上占我们组工作的1/2…

  2. Jackei says:

    越来越牛X了。呵呵

  3. Jackei says:

    另外,ricky 兄要注意,现在通过RSS订阅这篇文章只能看到标题,看不到任何具体的内容。

  4. ricky.zhu says:

    and now…?
    把rss从feedsky悄悄转到feedburner了

Leave a Reply

Your email address will not be published. Required fields are marked *