Statistical Methodology for Comparison of SAT Solvers

6 pages•Published: May 15, 2012

Abstract

Evaluating improvements to modern SAT solvers and comparison of two arbitrary solvers is a challenging and important task. Relative performance of two solvers is usually assessed by running them on a set of SAT instances and comparing the number of solved instances and their running time in a straightforward manner. In this paper we point to shortcomings of this approach and advocate more reliable, statistically founded methodologies that could discriminate better between good and bad ideas. We present one such methodology and illustrate its application.

Keyphrases: comparison, sat

In: Aaron Stump, Geoff Sutcliffe and Cesare Tinelli (editors). EMSQMS 2010. Workshop on Evaluation Methods for Solvers, and Quality Metrics for Solutions, vol 6, pages 33-38.

Links:	https://easychair.org/publications/paper/gW
	https://doi.org/10.29007/bhvj

BibTeX entry

@inproceedings{EMSQMS2010:Statistical_Methodology_Comparison_SAT,
  author    = {Mladen Nikolić},
  title     = {Statistical Methodology for Comparison of SAT Solvers},
  booktitle = {EMSQMS 2010. Workshop on Evaluation Methods for Solvers, and Quality Metrics for Solutions},
  editor    = {Aaron Stump and Geoff Sutcliffe and Cesare Tinelli},
  series    = {EPiC Series in Computing},
  volume    = {6},
  publisher = {EasyChair},
  bibsource = {EasyChair, https://easychair.org},
  issn      = {2398-7340},
  url       = {/publications/paper/gW},
  doi       = {10.29007/bhvj},
  pages     = {33-38},
  year      = {2012}}

Download PDF Open PDF in browser