BACKGROUND: Helical membrane proteins are vital for the interaction of cells 
with their environment. Predicting the location of membrane helices in protein 
amino acid sequences provides substantial understanding of their structure and 
function and identifies membrane proteins in sequenced genomes. Currently there 
is no comprehensive benchmark tool for evaluating prediction methods, and there 
is no publication comparing all available prediction tools. Current benchmark 
literature is outdated, as recently determined membrane protein structures are 
not included. Current literature is also limited to global assessments, as 
specialised benchmarks for predicting specific classes of membrane proteins were 
not previously carried out.
DESCRIPTION: We present a benchmark server at 
http://sydney.edu.au/pharmacy/sbio/software/TMH_benchmark.shtml that uses recent 
high resolution protein structural data to provide a comprehensive assessment of 
the accuracy of existing membrane helix prediction methods. The server further 
allows a user to compare uploaded predictions generated by novel methods, 
permitting the comparison of these novel methods against all existing methods 
compared by the server. Benchmark metrics include sensitivity and specificity of 
predictions for membrane helix location and orientation, and many others. The 
server allows for customised evaluations such as assessing prediction method 
performances for specific helical membrane protein subtypes.We report results 
for custom benchmarks which illustrate how the server may be used for 
specialised benchmarks. Which prediction method is the best performing method 
depends on which measure is being benchmarked. The OCTOPUS membrane helix 
prediction method is consistently one of the highest performing methods across 
all measures in the benchmarks that we performed.
CONCLUSIONS: The benchmark server allows general and specialised assessment of 
existing and novel membrane helix prediction methods. Users can employ this 
benchmark server to determine the most suitable method for the type of 
prediction the user needs to perform, be it general whole-genome annotation or 
the prediction of specific types of helical membrane protein. Creators of novel 
prediction methods can use this benchmark server to evaluate the performance of 
their new methods. The benchmark server will be a valuable tool for researchers 
seeking to extract more sophisticated information from the large and growing 
protein sequence databases.