.!.
聊天，说到benchmark测试和POC，这些都是实际项目中经常会遇到的概念，我们在研发环境很少会遇到，如果有机会实际参加这样的项目体验一把，也未尝不是一件好事。今天转载一篇Oracle Wiki
The following is my intro information for database benchmarking…
Reasons for Benchmarking
Benchmarks are performed for various reasons. However, benchmarks are primarily used:
* To compare different hardware configurations
Benchmarks can be used to compare the relative performance of different hardware running the same application. This is generally used to directly compare hardware configurations between two hardware vendors.
* To compare different database vendor software
By running the same benchmark using different database software on the same machine, one can easily compare between different database vendors. This is generally used to make a price/performance decision between vendors such as Oracle, Microsoft, IBM, etc.
* To compare different database software releases
Similar to the above, one can use different versions of the same vendor’s database software to compare the one they want to use or check for performance regressions due to upgrades (i.e. 10g vs. 11g).
Real World Benchmarks
While there are several industry standard benchmarks, most of the time, they don’t reflect a company’s actual workload. As such, you will often find custom benchmarks being performed. The goals of a real-world benchmark are the same reasons mentioned above.
Industry Standard Database Benchmarks
Industry standard benchmarks are generally used by businesses to compare different hardware and software system performance for purchase-related reasons. While industry standard benchmarks are modelled after real-world workloads, they rarely reflect a company’s real workload. However, it is a quick way to compare the performance of various hardware and software combinations operating in a well-defined scenario. The major industry standard benchmarks are listed below.
Transaction Processing Council (TPC)
The TPC is a non-profit corporation which supports a consortium of hardware and database software vendors devoted to defining transaction processing and database-related benchmarks. The primary goal behind TPC benchmarks is the definition of functional requirements which can be run on any database, regardless of the hardware or operating system. This allows vendors to implement their own benchmark kits in order to satisfy the functional requirements. Similarly, after having publicly submitted proof that a benchmark was conducted according to the specification, end-users have more of an assurance that what they are presented with is a valid, apples-to-apples comparison.
* TPC Benchmark C (TPC-C)
The TPC-C simulates an order-entryenvironment where a population of terminal operators executes transactions against a database. The benchmark is comprised of transactions which include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. The most frequent transaction consists of entering a new order which, on average, is comprised of ten different items. Each warehouse tries to maintain stock for the 100,000 items in the Company’s catalog and fill orders from that stock. The performance metric reported by TPC-C measures the number of orders that can be fully processed per minute and is expressed in tpm-C. The TPC-C will soon be deprecated in favor of the TPC-E.
* TPC Benchmark E (TPC-E)
The TPC-E benchmark simulates the OLTP workload of a brokerage firm. The focus of the benchmark is the central database that executes transactions related to the firm’s customer accounts. The TPC-E metric is given in transactions per second (tps). It specifically refers to the number of Trade-Result transactions the server can sustain over a period of time.
* TPC Benchmark H (TPC-H)
TheTPC-H is a decision support benchmark which consists of several business-oriented ad-hoc queries and concurrent data modifications. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions.
The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries.
Standard Performance Evaluation Corporation (SPEC)
The SPEC is a
* SPEC jAppServer
The TPC-C is
Free and Open Source Benchmark Kits
As the benchmark kits used by various database vendors are proprietary, several open source database benchmark kits have been written to implement not only the industry standard workloads, but others as well.
* Open Source Development Labs Database Test Suite
The OSDL DBT suite is the most comprehensive of all open source benchmark kits. While its benchmark implementations are based on TPC standards, they differ in some areas and are not certified; making a comparison between a certified TPC-C result and OSDL’s implementation of TPC-C (DBT-2) impossible. As the original OSDL DBT suite lacks Oracle support, Oracle has added and released it as part of the Oracle Linux Test kit.
* PolePosition
PolePosition is an open source benchmark test suite to compare database engines and object-relational mapping technology.
* jTPCC
jTPCC is an open source Java implementation of the TPC-C benchmark. While it claims compliance with TPC-C, it is not.
* BenchmarkSQL
BenchmarkSQL is an open source, easy to use JDBC benchmark which closely resembles the TPC-C standard for OLTP; it was originally based on jTPCC.
* Bristlecone
Bristlecone is a simple SELECT+INSERT/UPDATE/DELETE throughput and response-time benchmark written in Java.
* FinTime
FinTime is an open source set of data and queries which reflects the needs of financial analysts who are studying patterns in stock market data, but it should appeal to the designers of any system that has pretensions of handling ordered data well.
* Java TPC-W Implementation
This open-source implementation is based on the TPC-W specification version 1.0.1. It strays from the official benchmark specification in a few minor areas, which are specified in documentation included with the distribution. This version itself lacks Oracle support, but it can be found on the web.
* Swingbench
Swingbench is a free load generator (and benchmarks) designed to stress test an Oracle database (9i,10g,11g).
* OpenLink ODBC Bench
OpenLink ODBC Bench is an open-source ODBC Benchmarking tool providing real-time comparative benchmarking for ODBC Drivers, Database Engines, and Operating Systems combinations. The Benchmarks in this application are loosely based on the TPC-A and TPC-C standard benchmarks, with modifications to specifically test the performance of an ODBC Driver and/or Database Engine in a client/server environment.
* OpenLink JDBC Bench
Similar to OpenLink’s ODBC Bench, this is a JDBC version.
* TPCC-UVa
A free, open-source implementation of the TPC-C Benchmark.
* Hammerora
An open source load generation tool.
Commercial Benchmark Kits
There are several industry standard benchmarks.
* Official Oracle Benchmark Kits
Similar to Microsoft and IBM, the official benchmark kits used by Oracle can only be obtained under certain conditions. For more information, contact your Oracle Consulting/Partner/Support representative.
* Real Application Testing
While many wouldn’t think of Real Application Testing as a traditional benchmark kit, it does perform workload execution primarily for the real world benchmarking case.
* Quest Software’s Benchmark Factory for Databases
Benchmark Factory for Databases is a workload simulation and scalability testing tool that stress tests your environment by simulating users and transactions on the database. You can either replay production workload or use synthetic workload to test the limits of your database.
* SPEC jAppServer Benchmark Kit
This can be acquired directly from SPEC.
Benchmarking-related Books
There are several good books regarding benchmarking. The best I’ve found include the following:
* The Benchmark Handbook: For Database and Transaction Processing Systems (Jim Gray)
A great, albeit outdated book. While several of the benchmarks it discusses have been deprecated, it presents good concepts and background information regarding benchmarking.
* Performance Analysis of Transaction Processing Systems (Wilbur H. Highleyman)
While out of print, this book presents both plain-text and math-heavy concepts of queueing and performance analysis techniques oriented for OLTP.
* Database Benchmarking: Practical Methods for Orac
install，注意用 chown -Rf，确保子目录的权限也得到修改
Recent Comments