For tasks 1 and 2, all our evaluations ran on a server with an AMD Ryzen Threadripper 2990WX
32-Core Processor and 128GB of RAM. The server ran on a 5.3.0-46-generic
#38 18.04.1-Ubuntu SMP operating system. We use Java 1.8 and OWL API
5.1.11 in the experiments. We set the heap space to 24GB. The time-out was set to 30 minutes. We
report the average time taken in 2 independent runs for the classification task.
Task-1
We got 4 submissions in this category (arranged in the order of ranking). We evaluated each ontology in terms of the time-taken(in seconds) by different OWL 2 DL reasoners (HermiT 1.4.5.456, JFact 5.0.05, Konclude 0.7.0, Openllet 2.6.46
, and Pellet 2.3.6).
LE (Loading Error), PE (Parsing Error), ME (Memory Error), TO (Time-out)
| Submissions | Submitted Ontologies | Expressivity | Logical Axioms | HermiT | Pellet | Openllet | JFact | Konclude |
| CaLiGraph | clg_10 | SOI(D) | 15 | 0.2 | 0.1 | 0.16 | 0.21 | 0.002 |
| clg_10e2 | SOI(D) | 226 | 0.2 | 0.1 | 0.1 | 0.90 | 0.006 |
| clg_10e3 | SOI(D) | 22,797 | 265 | 11.75 | 9.8 | 86.70 | 0.137 |
| clg_10e4 | SOI(D) | 157,825 | TO | TO | TO | 1541 | 1.2 |
| clg_10e5 | SOI(D) | 2,436,382 | TO | TO | TO | TO | 10.8 |
| clg_full | SOI(D) | - | ME | ME | ME | ME | TO |
| Micro-electronics | Genial | SHOIQ(D) | 396 | 0.2 | 0.3 | 0.47 | 4.99 | 0.032 |
| DACOC3 | ExtruOnt | SIF | 281 | Error | TO | TO | Inconsistent | 0.004 |
| COSMO | SHOIN(D) | 108,525 | Error | TO | TO | LE | Inconsistent |
| uPheno | - | - | Error | TO | TO | PE | LE |
| Gene Variant Data | variant1 | SI | 617,565 | Error | Inconsistent | Inconsistent | Inconsistent | 2.41 |
| variant2 | SI | 199,845 | Error | Inconsistent | Inconsistent | Inconsistent | 0.78 |
| variant3 | SI | 305,595 | Error | TO | 253 | TO | 1.1 |
| variant4 | SI | 651,723 | Error | Inconsistent | Inconsistent | Inconsistent | 2.72 |
| variant5 | SI | 617,924 | Error | Inconsistent | Inconsistent | Inconsistent | 2.43 |
Task-2
Time-taken(in seconds) and Memory Consumed (in GB) by submitted description logic reasoner: Konclude 0.7.0. Test ontologies are available here.
| Test Ontologies | Expressivity | Logical Axioms | Time-taken (sec) | Memory Consumed (GB) |
| OWL2DL-1 | SROIQ(D) | 50,127 | 37 | 4.5 |
| OWL2DL-2 | SROIQ(D) | 99,080 | 243 | 11 |
| OWL2DL-5 | SROIQ(D) | 325,408 | TO | NA |
| OWL2DL-10 | SROIQ(D) | 711,017 | TO | NA |
Task-3
Results of the submitted Neuro-symbolic reasoner: EmELvar. Test ontologies are available here.
| Test Ontologies | Top1 | Top10 | Top100 | Median | 90th Percentile |
| OWL2EL-1 | NA | NA | NA | NA | NA |
| OWL2EL-2 | 0 | 0 | 0 | 35028 | 92741 |
| OWL2EL-3 | 0.08 | 0.14 | 0.15 | 49430 | 247953 |
| OWL2EL-4 | 0 | 0 | 0.4 | 92 | 227 |
| OWL2EL-5 | 0 | 0.07 | 0.4 | 112.5 | 287 |