The most important way of conveying new findings in biomedical research is 
scientific publication. Extraction of protein-protein interactions (PPIs) 
reported in scientific publications is one of the core topics of text mining in 
the life sciences. Recently, a new class of such methods has been proposed - 
convolution kernels that identify PPIs using deep parses of sentences. However, 
comparing published results of different PPI extraction methods is impossible 
due to the use of different evaluation corpora, different evaluation metrics, 
different tuning procedures, etc. In this paper, we study whether the reported 
performance metrics are robust across different corpora and learning settings 
and whether the use of deep parsing actually leads to an increase in extraction 
quality. Our ultimate goal is to identify the one method that performs best in 
real-life scenarios, where information extraction is performed on unseen text 
and not on specifically prepared evaluation data. We performed a comprehensive 
benchmarking of nine different methods for PPI extraction that use convolution 
kernels on rich linguistic information. Methods were evaluated on five different 
public corpora using cross-validation, cross-learning, and cross-corpus 
evaluation. Our study confirms that kernels using dependency trees generally 
outperform kernels based on syntax trees. However, our study also shows that 
only the best kernel methods can compete with a simple rule-based approach when 
the evaluation prevents information leakage between training and test corpora. 
Our results further reveal that the F-score of many approaches drops 
significantly if no corpus-specific parameter optimization is applied and that 
methods reaching a good AUC score often perform much worse in terms of F-score. 
We conclude that for most kernels no sensible estimation of PPI extraction 
performance on new text is possible, given the current heterogeneity in 
evaluation data. Nevertheless, our study shows that three kernels are clearly 
superior to the other methods.