The gene normalization task in BioCreative III

• DOI: 10.1186/1471-2105-12-s8-s2
• Publication Date: 2011-10-05T00:44:22Z
• Authors: Zhiyong Lu; Hung-Yu Kao; Chih-Hsuan Wei; Minlie Huang; Jingchen Liu; Cheng-Ju Kuo; Chun-Nan Hsu; Richard Tzong-Han...; Hong-Jie Dai; Naoaki Okazaki; Han-Cheol Cho; Martin Gerner; Illes Solt; Shashank Agarwal; Feifan Liu; Dina Vishnyakova; Patrick Ruch; Martin Romacker; Fabio Rinaldi; Sanmitra Bhattach...; Padmini Srinivasan; Hongfang Liu; Manabu Torii; Sergio Matos; David Campos; Karin Verspoor; Kevin M Livingston; W John Wilbur
• Keywords: Gold standard (test); Normalization; Ground truth; Maximization

We report the Gene Normalization (GN) challenge in BioCreative III where participating teams were asked to return a ranked list of identifiers genes detected full-text articles. For training, 32 fully and 500 partially annotated articles prepared. A total 507 selected as test set. Due high annotation cost, it was not feasible obtain gold-standard human annotations for all Instead, we developed an Expectation Maximization (EM) algorithm approach choosing small number manual that most capable differentiating team performance. Moreover, same subsequently used inferring ground truth based solely on submissions. performance both gold standard inferred using newly proposed metric called Threshold Average Precision (TAP-k). received 37 runs from 14 different task. When evaluated 50 articles, highest TAP-k scores 0.3297 (k=5), 0.3538 (k=10), 0.3535 (k=20), respectively. Higher 0.4916 (k=5, 10, 20) observed when over full combining results machine learning, best composite system achieved 0.3707 0.4311 0.4477 (k=20) standard, representing improvements 12.4%, 21.8%, 26.6% results, By text being species non-specific, GN task has moved closer real literature curation than similar tasks past presents additional challenges mining community, revealed overall results. evaluating show EM allows submissions be differentiated while keeping effort feasible. Using measures comparative between teams. Finally, by comparing rankings vs. truth, further demonstrate is effective detecting good