Computer and microprocessor architectures are advancing at an astounding pace. However, increasing demands on performance coupled with a wide variety of specialized operating environments act to slow this pace by complicating the performance evaluation process. Carefully balancing efficiency and accuracy is key to avoid slowdowns, and such a balance can be achieved with an in-depth understanding of the available evaluation methodologies. Performance Evaluation and Benchmarking outlines a variety of evaluation methods and benchmark suites, considering their strengths, weaknesses, and when each is appropriate to use. Following a general overview of important performance analysis techniques, the book surveys contemporary benchmark suites for specific areas, such as Java, embedded systems, CPUs, and Web servers. Subsequent chapters explain how to choose appropriate averages for reporting metrics and provide a detailed treatment of statistical methods, including a summary of statistics, how to apply statistical sampling for simulation, how to apply SimPoint, and a comprehensive overview of statistical simulation. The discussion then turns to benchmark subsetting methodologies and the fundamentals of analytical modeling, including queuing models and Petri nets. Three chapters devoted to hardware performance counters conclude the book. Supplying abundant illustrations, examples, and case studies, Performance Evaluation and Benchmarking offers a firm foundation in evaluation methods along with up-to-date techniques that are necessary to develop next-generation architectures.

The author is one of the prominent researchers in the field of Data Envelopment Analysis (DEA), a powerful data analysis tool that can be used in performance evaluation and benchmarking. This book is based upon the author’s years of research and teaching experiences. It is difficult to evaluate an organization’s performance when multiple performance metrics are present. The difficulties are further enhanced when the relationships among the performance metrics are complex and involve unknown tradeoffs. This book introduces Data Envelopment Analysis (DEA) as a multiple-measure performance evaluation and benchmarking tool. The focus of performance evaluation and benchmarking is shifted from characterizing performance in terms of single measures to evaluating performance as a multidimensional systems perspective. Conventional and new DEA approaches are presented and discussed using Excel spreadsheets — one of the most effective ways to analyze and evaluate decision alternatives. The user can easily develop and customize new DEA models based upon these spreadsheets. DEA models and approaches are presented to deal with performance evaluation problems in a variety of contexts. For example, a context-dependent DEA measures the relative attractiveness of similar operations/processes/products. Sensitivity analysis techniques can be easily applied, and used to identify critical performance measures. Two-stage network efficiency models can be utilized to study performance of supply chain. DEA benchmarking models extend DEA’s ability in performance evaluation. Various cross efficiency approaches are presented to provide peer evaluation scores. This book also provides an easy-to-use DEA software — DEAFrontier. This DEAFrontier is an Add-In for Microsoft® Excel and provides a custom menu of DEA approaches. This version of DEAFrontier is for use with Excel 97-2013 under Windows and can solve up to 50 DMUs, subject to the capacity of Excel Solver. It is an extremely powerful tool that can assist decision-makers in benchmarking and analyzing complex operational performance issues in manufacturing organizations as well as evaluating processes in banking, retail, franchising, health care, public services and many other industries.

This new edition continues to emphasize the use of data envelopment analysis (DEA) to create optimization-based benchmarks within hospitals, physician group practices, health maintenance organizations, nursing homes and other health care delivery organizations. Suitable for graduate students learning DEA applications in health care as well as for practicing administrators, it is divided into two sections covering methods and applications. Section I considers efficiency evaluations using DEA; returns to scale; weight restricted (multiplier) models; non-oriented or slack-based models, including in this edition two versions of non-controllable variable models and categorical variable models; longitudinal (panel) evaluations and the effectiveness dimension of performance evaluation. A new chapter then looks at new and advanced models of DEA, including super-efficiency, congestion DEA, network DEA, and dynamic network models. Mathematical formulations of various DEA models are placed in end-of-chapter appendices. Section II then looks at health care applications within particular settings, chapter-by-chapter, including hospitals, physician practices, nursing homes and health maintenance organizations (HMOs). Other chapters then explore home health care and home health agencies; dialysis centers, community mental health centers, community-based your services, organ procurement organizations, aging agencies and dental providers; DEA models to evaluate provider performance for specific treatments, including stroke, mechanical ventilation and perioperative services. A new chapter then examines international-country-based applications of DEA in health care in 16 different countries, along with OECD and multi-country studies. Most of the existing chapters in this section were expanded with recent applications. Included with the book is online access to a learning version of DEA Solver software, written by Professor Kaoru Tone, which can solve up to 50 DMUs for various DEA models listed in the User’s Guide at the end of the book.

This book constitutes the proceedings of the Second Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2010, held in conjunction with the 36th International Conference on Very Large Data Bases, VLDB 2010, in Singapore, September 13-17, 2010. The 14 full papers and two keynote papers were carefully selected and reviewed from numerous submissions. This book considers issues such as appliance; business intelligence; cloud computing; complex event processing; database optimizations; data compression; energy and space efficiency, green computing; hardware innovations; high speed data generation; hybrid workloads; very large memory systems; and virtualization.

First established in August 1988, the Transaction Processing Performance Council (TPC) has shaped the landscape of modern transaction processing and database benchmarks over two decades. Now, the world is in the midst of an extraordinary information explosion led by rapid growth in the use of the Internet and connected devices. Both user-generated data and enterprise data levels continue to grow ex- nentially. With substantial technological breakthroughs, Moore's law will continue for at least a decade, and the data storage capacities and data transfer speeds will continue to increase exponentially. These have challenged industry experts and researchers to develop innovative techniques to evaluate and benchmark both hardware and software technologies. As a result, the TPC held its First Conference on Performance Evaluation and Benchmarking (TPCTC 2009) on August 24 in Lyon, France in conjunction with the 35th International Conference on Very Large Data Bases (VLDB 2009). TPCTC 2009 provided industry experts and researchers with a forum to present and debate novel ideas and methodologies in performance evaluation, measurement and characteri- tion for 2010 and beyond. This book contains the proceedings of this conference, including 16 papers and keynote papers from Michael Stonebraker and Karl Huppler.

Managers are often under great pressure to improve the performance of their organizations. To improve performance, one needs to constantly evaluate operations or processes related to producing products, providing services, and marketing and selling products. Performance evaluation and benchmarking are a widely used method to identify and adopt best practices as a means to improve performance and increase productivity, and are particularly valuable when no objective or engineered standard is available to define efficient and effective performance. For this reason, benchmarking is often used in managing service operations, because service standards (benchmarks) are more difficult to define than manufacturing standards. Benchmarks can be established but they are somewhat limited as they work with single measurements one at a time. It is difficult to evaluate an organization’s performance when there are multiple inputs and outputs to the system. The difficulties are further enhanced when the relationships between the inputs and the outputs are complex and involve unknown tradeoffs. It is critical to show benchmarks where multiple measurements exist. The current book introduces the methodology of data envelopment analysis (DEA) and its uses in performance evaluation and benchmarking under the context of multiple performance measures.

The book discusses rationales for creating and updating benchmarks, the use of benchmarks in academic research, benchmarking methodologies, the relation of SPEC benchmarks to other benchmarking activities, shortcomings of current benchmarks, and the need for further benchmarking efforts. Performance evaluation and benchmarking are of concern to all computer-related disciplines. A benchmark is a standard program or set of programs that can be run on different computers to give an accurate measure of their performance. This book covers a variety of aspects of computer performance evaluation, with a focus on Standard Performance Evaluation Corporation (SPEC) benchmarks. SPEC is a nonprofit organization whose members represent industry, academia, and other organizations. The book discusses rationales for creating and updating benchmarks, the use of benchmarks in academic research, benchmarking methodologies, the relation of SPEC benchmarks to other benchmarking activities, shortcomings of current benchmarks, and the need for further benchmarking efforts. Contributors Brian Armstrong, Frederica Darema, Edward S. Davidson, Sylvia Dieckmann, Jozo J. Dujmovic, Rudolf Eigenmann, J. Kelly Flanagan, Greg Gaertner, Jonathan Geisler, John Gustafson, Urs Hölzle, Shih-Hao Hung, Kathryn S. McKinley, Reinhard Riedl, Faisal Saied, Frank Sorenson, Mark Straka, Valerie Taylor, Olivier Temam, Rajat Todi, Reinhold Weicker