Auer, S.; Barone, D.A.C.; Bartz, C.; Cortes, E.G.; Jaradeh, M.Y. et al.: The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. In: Scientific Reports 13 (2023), 7240. DOI: https://doi.org/10.1038/s41598-023-33607-z
Zusammenfassung: | |
Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge. | |
Lizenzbestimmungen: | CC BY 4.0 Unported |
Publikationstyp: | Article |
Publikationsstatus: | publishedVersion |
Erstveröffentlichung: | 2023 |
Die Publikation erscheint in Sammlung(en): | Zentrale Einrichtungen Forschungszentren |
Pos. | Land | Downloads | ||
---|---|---|---|---|
Anzahl | Proz. | |||
1 | Germany | 36 | 60,00% | |
2 | United States | 12 | 20,00% | |
3 | No geo information available | 2 | 3,33% | |
4 | Greece | 2 | 3,33% | |
5 | France | 2 | 3,33% | |
6 | Netherlands | 1 | 1,67% | |
7 | United Kingdom | 1 | 1,67% | |
8 | China | 1 | 1,67% | |
9 | Canada | 1 | 1,67% | |
10 | Australia | 1 | 1,67% | |
andere | 1 | 1,67% |
Hinweis
Zur Erhebung der Downloadstatistiken kommen entsprechend dem „COUNTER Code of Practice for e-Resources“ international anerkannte Regeln und Normen zur Anwendung. COUNTER ist eine internationale Non-Profit-Organisation, in der Bibliotheksverbände, Datenbankanbieter und Verlage gemeinsam an Standards zur Erhebung, Speicherung und Verarbeitung von Nutzungsdaten elektronischer Ressourcen arbeiten, welche so Objektivität und Vergleichbarkeit gewährleisten sollen. Es werden hierbei ausschließlich Zugriffe auf die entsprechenden Volltexte ausgewertet, keine Aufrufe der Website an sich.