Taiwan AI Labs, a leading artificial intelligence research institution, has announced a significant milestone in its global expansion efforts with the signing of a memorandum of cooperation with Turing College, a pioneering educational institution based in Lithuania. The collaboration aims to facilitate intensive exchanges in areas such as federated learning and will also encompass internship programs, fostering a mutually beneficial partnership.

Turing College is a renowned online educational institution offering cutting-edge artificial intelligence courses to working professionals. Students have the unique opportunity to learn from experts in the field, including AI industry leaders such as Google, Meta, Amazon, and now, Taiwan AI Labs, as well as prestigious research universities like Cambridge and Stanford. Currently, the college has over 500 students from more than 20 countries, with the ambitious goal of becoming one of the largest AI academies in the European Union by 2025.

Founder of Taiwan AI Labs, Mr. Ethan Tu, expressed his excitement about the collaboration, stating, “This partnership provides Taiwan AI Labs with an exceptional opportunity to showcase Taiwan’s technological prowess and jointly develop trustworthy AI technologies that align with the principles of privacy and human rights in the European Union. We also anticipate that this collaboration will foster greater industry partnerships in the EU region, shining a spotlight on Taiwan’s contributions to the global AI landscape.”

Mr. Jason Huang, Representative of the Taiwanese Representative Office in Lithuania, extended his congratulations, highlighting the synergies between Taiwan and Lithuania as both economies thrive on advanced technology. He emphasized the immense potential for collaboration in the burgeoning AI industry, stating, “Both Taiwan and Lithuania are leaders in advanced technology, and together, we have substantial room for cooperation in the realm of AI knowledge.”

The collaboration between Taiwan AI Labs and Turing College was formalized following intensive discussions and consensus on various aspects, including the development of AI education, utilization of federated learning methodologies, establishment of best practices, and the initiation of an internship exchange program. The memorandum of cooperation was signed in April, marking the commencement of a fruitful partnership in the field of AI application and academia.

Lukas Kaminskis, CEO of Turing College, expressed his enthusiasm for the collaboration, stating, “Partnering with Taiwan AI Labs provides our students with an exceptional opportunity to learn from one of the leading AI institutions in Asia. We are excited about the possibilities this collaboration holds for their educational journey.”

With this collaboration, Taiwan AI Labs and Turing College aim to foster knowledge exchange, nurture AI talent, and contribute to the advancement of the global AI ecosystem. 

About Taiwan AI Labs:

Taiwan AI Labs is a renowned research institution at the forefront of artificial intelligence technology. With a mission to push the boundaries of AI innovation, Taiwan AI Labs focuses on groundbreaking research, development of cutting-edge AI applications, and fostering partnerships to create a positive impact on society. For more information, please visit [website link].

About Turing College

Turing College is an online educational institution dedicated to providing high-quality AI courses to professionals worldwide. Through its industry partnerships and expert-led curriculum, Turing College aims to empower individuals with the skills and knowledge required to thrive in the AI-driven world. To learn more about Turing College, please visit [website link].

Press Contact

Mr. Barrett Tsai

PR Account Executive | Taiwan AI Labs

barrett.tsai@ailabs.tw 

Eunice You-Chi Liu

• Introduction

pubmedKB is “a novel literature search engine that combines a large number of state-of-the-art text-mining tools optimized to automatically identify the complex relationships between biomedical entities—variants, genes, diseases, and chemicals—in PubMed abstracts” (Li et al.).  Currently, the pubmedKB Relation Extraction (RE) module comprises three submodules—Relational Phrases (algorithm developed by applying spaCy which is an open-source library that pioneers syntactic-dependency syntax parser), Relational Facts (model advanced by integrating R-BERT relation classification framework and BioBert) and Odds Ratio Info (tool progressed from a pre-trained general-purpose sequence-to-sequence model T5). 

• Motivation

Indeed, through powerful literature mining, pubmedKB allows researchers and medical practitioners to more effectively gain extensive knowledge and insights on any biomedical entity. As this literature search engine takes all PubMed abstracts into account (currently, pubmedKB currently contains relations for 10.8M PubMed abstracts and 1.7M PubMed full texts), the relations extracted can be excessive and superfluous especially if the users want to productively identify the most pertinent relations or concisely present what relations a biomedical entity or entities pair entails. For instance, I am interested in learning the relations between HGVS:p.V600E and MESH:D009369. There are more than 2000 relations extracted for this entity pair (Figure 1). Depending on the needs of the users, scrolling over a sea of relations for this given pair might not be the most effective. Specifically, the vast majority of the relations are relational facts (denoted as rbert_cre). Moreover, when we look further into the relations extracted that are rbert_cre, there are more than 1500 relations implying similar meaning: MESH:D009369 patients that carry HGVS:p.V600E, which can be redundant and difficult to grasp what is important (Figure 2). In this case, we ask given any entity or PubMed Unique Identifier (PMID), how can we extract and demonstrate the most relevant and significant biomedical relations? 

Figure 1: Using the Web GUI to search for the relations between the biomedical entities pair: HGVS: p.V600E and MESH: D009369, the number of relations found for each submodule is shown.

Figure 2: A demonstration of a small portion of the in-patient relations extracted between HGVS: p.V600E and MESH: D009369.

• Methods

To answer the research question posed in the motivation section, the methods can be majorly divided into two parts which are 1) “extracting” and 2) “demonstrating” the most relevant and significant biomedical relations. With the goal of acquiring the most relevant and significant relations of given entities, we develop scispaCy open relation extractions (ORE) and research on applying and developing neural open relation extractions (ORE). After finding the most accurate relations, we develop the evidence summary in order to present selective relations in a concise and informative way.

1) Develop scispaCy ORE

We decide to develop scispaCy ORE because it is a spaCy model specifically for biomedical text processing, which fits our main interest in working with biomedical literature and extracting significant relations from them. scispaCy dependency parsers are trained on GENIA 1.0 and then are transformed into universal dependencies, UD, format (Neumann et al.). According to spaCy’s official documentation, spaCy is trained in Clear format (English · spacy models documentation). scispaCy dependency parsers are trained on GENIA 1.0 and then are transformed into universal dependencies, UD, format (Neumann et al.). The difference in how the dependency is generated and trained might have resulted in syntactic parsing and part-of-speech (POS) tagging differences between scispaCy and spaCy (Figure 3).

Figure 3: For a very simple sentence that entails a relation between two biomedical entities, “All of ENT0ITY relates to ENT1ITY,” in which ENT0ITY and ENT1ITY infer two different biomedical entities. The upper part (labeled as a) is the dependency parsing using spaCy and the lower part (labeled as b) is the dependency parsing applying scispaCy. Notably, the dependency parse of the same sentence (shown by arrow) and the part-of-speech (POS) tagging (marked as capitalized abbreviation under the word) are distinct in part a and b. 

Using the same grammatical structure and the defined relationships between words as spaCy’s can result in misleading relations being identified. For example, as shown in Figure 3b, the main verb “relates” are directly linked to the object “ENT1ITY” rather than links to the preposition “to” in Figure 3a.

To ensure that the most relevant and important relations are extracted, understanding and redefining scispaCy’s head-dependent relationships are critical as they oftentimes “directly encode important information that is often buried in the more complex phrase-structure parses” (Jurafsky and Martin 310-334). In this case, we redefine scispaCy’s active and passive sentence structure capturing the subject, predicate, meaningful preposition, and object, and ensure that prepositions following the predicate and before entities in a noun modifier (nmod) dependency are captured. We also created a customized noun chunk extraction and set it as an extension as the default noun chunk extraction does not correspond to scispaCy’s dependency parsing. Besides the difference in dependency parsing, scispaCy and spaCy’s part-of-speech tagging diverges. In this case, we re-tag certain groups of words to ensure that the complete entities are pinpointed because for instance in Figure 3, the first entity should be “All of ENT0ITY” instead of only “ENT0ITY” because if today the determiner is changed from “all” to “none,” then the relations extracted can be inaccurate. 

The relations extracted using spaCy or scispaCy ORE are called triplets as they all include one head mention, one predicate, and one tail mention. After the triplets are identified, we calculate the number of triplets extracted and randomly select 20,000 sentences with or without triplets and manually label them. In order to be counted as a valid triplet, the triplet has to match the original meaning of the sentence, should be an assertion, and be understandable (unambiguous) without looking at the original sentence. 

2) Research on neural ORE

The current pubmedKB’s ORE submodule–spaCy–and the newly developed scispaCy ORE are all rule-based and can extract relations on a sentence level. As our goal is to extract the most significant and relevant relations, focusing on the sentence level might not be sufficient. We decide to research and develop neural ORE which can enable us to extract relations from paragraphs, sections, and hopefully the whole paper. Inspired by the paper “OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction,” we adopt and applied the OpenIE6 model to our own data of PubMed abstracts and full texts. OpenIE6 model contains a series of different pipelines, which includes a BERT encoder that “computes contextualized embeddings for each word” and a 2-layer transformer that attains the contextual embeddings at every level (Kolluru et al.). Extractions, in other words, triplets, are generated at every iteration after passing in the contextualized embeddings to produce words’ labels at a specific level (Kolluru et al.).  

The OpenIE6 model is trained on Wikipedia sentences, which is the same dataset as the one used for OpenIE4 (Kolluru et al.). We prepared the biomedical sentences from PubMed abstracts and full texts to get extractions. In order to evaluate the performance of the model on our biomedical sentences, we also prepared gold standard data derived from spaCy’s triplets that match the format of OpenIE6’s testing data. 

3) Develop and generate the evidence summary

After gathering the most relevant and significant relations, we hope to demonstrate them in a clear and intuitive way.  Figure 4 demonstrates an overview of how the evidence summary is generated. Aiming to be as diverse and informative as possible, we design that the summary generated would contain three different kinds of relations which are comprised of odds ratio, relational fact, and relational phrase via ORE. To achieve the concise aspect, we skillfully selected the relations with the highest confidence value–highest odds ratio and most supported relational phrases. In particular for the relational facts, based on the level of informativity, if the confidence value is identical, the causal relation between genetic variants and diseases is favored over the relation that entails certain patients carrying some specific diseases and the appositional relation of genetic variants and diseases. In terms of which relational phrases to feature, at most two different relational phrases from distinct sentences are included in the summary. Particularly, the relational phrases are selected based on the number of relations extracted from a single sentence as it is highly possible to infer that these sentences contain crucial information. 

Figure 4: A quick overview and explanation of how the evidence summary is being designed and produced. 

• Results

Using the newly developed scispaCy ORE, among 95,441 unique sentences extracted from various PubMed abstracts, we are able to identify and extract 1000 more triplets compared to employing spaCy ORE (Table 1). 

Table 1: Among 95,441 individual sentences extracted from various PubMed abstracts, the number of sentences containing identified triplets and the number of identified triplets are both higher when applying the newly developed scispaCy ORE.  

We proceed into qualitative evaluation by manually looking over triplets that occurred in the randomly selected 20,000 sentences. Based on Table 2, by all means, the yield when applying scispaCy ORE is higher because a larger number of triplets are identified. The precision of the correct triplets (those who fit the criterion mentioned in the method section) extracted from scispaCy and spaCy is similar (Table 2). We can not really calculate recall because it would be nearly impossible to identify false negative triplets. 

Furthermore, we discover that scispaCy ORE allows us to generate triplets in sentences that have possessive relationships and adjective comparisons which current spaCy ORE is not yet able to achieve. To illustrate, for the sentence “Our results indicate that CYFRA 21-1 may be a useful tumor marker in NSCLC, especially in carcinoma planocellulare,” the triplet–“CYFRA 21-1, marker in, NSCLC”–is identified and extracted using scispaCy ORE but not spaCy ORE. In this case, scispaCy ORE is capable of identifying and incorporating relations other than action verbs and these other relations are also common and critical in biomedical literature besides those focused on an action verb. 

Table 2:  After randomly selecting 20,000 sentences and labeling them, the number of correct triplets extracted in scispaCy is 382 in spaCy is 244. Among the number of correct triplets extracted in scispaCy ORE and spaCy ORE, 190 triplets are identical. The precision of the correct triplets extracted from scispaCy and spaCy is similar, 0.75 and 0.79 respectively. 

Besides having scispaCy ORE that extracts and identifies important triplets which are missing in spaCy ORE, we recognize that the performance of neural ORE, specifically the OpenIE6 model, is also promising. By creating one-to-one mapping between the predicted extractions (generated by the OpenIE6 model) and the spaCy original extracted triplets, we evalutate the precision and recall. Notably, the AUC and F1 score, 49.0 and 62.9, are comparatively the same as the testing results using the standard CaRB dataset –a large-scale Open IE benchmark annotation that aims to contribute to the standardization of Open IE evaluation (Stanovsky and Dagan).

After developing and researching the most suitable ORE to extract the most significant and relevant relations, we compiled those information into a well-organized and informative paragraph (Figure 5). Moreover, the relations annotated by each annotator, including odds ratio, relational phrase, and relational fact are clearly labeled in distinct colors for easier comprehension (Figure 5). 

Figure 5: The summary of the relations between HGVS: p.V600E and MESH: D009369. 

• Discussion

In this study, we ask and answer the question: given any entity or PubMed Unique Identifier (PMID), how can we extract and demonstrate the most relevant and significant biomedical relations? We add on to previous work especially to the development of pubmedKB by developing a new biomedical-specific sicspaCy ORE and researching the application of neural ORE. Moreover, we organize and present those selected relations into a well-informative paragraph ready for the usage of demonstrating in academic and professional contexts. Future studies can further expand on the development of neural ORE given the positive performance of the OpenIE6 model. Currently, the OpenIE6 model is trained on sentences from Wikipedia. In this case, training the OpenIE6 model with both biomedical sentences from PubMed and general sentences from Wikipedia can potentially yield greater performance. In terms of the scispaCy ORE, although it is a comparatively developed and mature submodule ready to be used. It might also be interesting to explore scispaCy’s named-entity recognition (NER) module and see how it can be integrated into the ORE. 

Works Cited

“Dependency Parsing.” Speech and Language Processing: An Introduction to 

Natural Language Processing, Computational Linguistics, and Speech 

Recognition, by Dan Jurafsky and James H. Martin, Pearson, 2022, pp. 

310–334. 

“English · Spacy Models Documentation.” English, 

spacy.io/models/en?fbclid=IwAR2_DCNpcRx8qp_iB0b5_QdTjljmL2oty

pUMTLm4p-wGP_8CYuvLnlfH6os#en_core_web_sm. 

Kolluru, Keshav, et al. “OpenIE6: Iterative Grid Labeling and Coordination 

Analysis for Open Information Extraction.” Proceedings of the 2020 

Conference on Empirical Methods in Natural Language Processing 

(EMNLP), 2020, doi:10.18653/v1/2020.emnlp-main.306. 

Li, Peng-Hsuan, et al. “PubmedKB: An Interactive Web Server for Exploring 

Biomedical Entity Relations in the Biomedical Literature.” Nucleic Acids 

Research, vol. 50, no. W1, 2022, doi:10.1093/nar/gkac310. 

Neumann, Mark, et al. “ScispaCy: Fast and Robust Models for Biomedical 

Natural Language Processing.” Proceedings of the 18th BioNLP 

Workshop and Shared Task, 2019, doi:10.18653/v1/w19-5034. 

Stanovsky, Gabriel, and Ido Dagan. “Creating a Large Benchmark for Open 

Information Extraction.” Proceedings of the 2016 Conference on 

Empirical Methods in Natural Language Processing, 2016, 

doi:10.18653/v1/d16-1252. 

Introduce Azoospermia

Azoospermia is a medical term implying the condition of no measurable sperm in a man’s semen. It is also the main challenge in male infertility. Azoospermia could be divided into two classes, including obstructive azoospermia(OA) and non-obstructive azoospermia(NOA). For OA, the testicular size and serum hormone profile are normal. On the other hand, NOA means the process of spermatogenesis is unusual, and to make a further diagnosis the doctor has to check the cell findings of testicular specimens. Originally, it took 2-3 days to complete a pathological diagnosis. In order to make the process more efficient, the Department of Urology of Taipei Veterans General Hospital has developed a standard process using the testicular touch print smear(TPS) technique to make real-time diagnoses. However, it would take a lot of time to learn the interpretation of TPS. Machine learning or deep learning technologies have applied to different kinds of medical images and have become a new field with many researchers involved. AI Labs cooperates with Dr. William J. Huang(黃志賢醫師) from Taipei Veterans General Hospital, to make AI assist surgeons in the reading of TPS slides. We aim to perform object detection techniques on testicular specimens to find 6 cells, including Sertoli cell, primary spermatocyte, round spermatid, elongated spermatid, immature sperm, and mature sperm. These cells are essential to determine different stages of azoospermia. 

Implementation

In this task, our goal is to detect cells among the above 6 classes in one individual image, the desired outputs are accurate bounding boxes and their labels. Since there are no existing open-dataset for this work, we have to build our own dataset. Provided input images are 2D testicular specimens captured from an electron microscope, with ground truth boxes and labels. The cell dataset currently contains 120 images with over 4,500 cells, which are annotated and reviewed by Dr. Huang and his assistant. Considering input size, class number and performances, we use EfficientDet as our model to train a network to detect cells in TPS. 

Example of input image with annotations. Different color means different class.

What is EfficientDet?

EfficientDet is the current state-of-the-art network for object detection. It is consists of a backbone network and a feature network. Inputs are feed into the backbone network; features would be extracted from different layers in backbone and sent to feature networks. Feature maps would be combined in different strategies depending on the network you used. At the end of the feature network, there are two heads with several layers that are used to predict final bounding box positions and class labels. In our setting, the EfficientDet uses EfficientNet pretrained on ImageNet as backbone and BiFPN as feature networks. 

EfficicientDet Model Structure.

EfficientNet is a model that applies the compound scaling strategy to improve accuracy. While pursuing higher performance, researchers often scale up model width, depth or resolution. However, the results are often contrary to expectation if the model becomes too complicated. The authors of EfficienNet combined different scaling through Neural Architecture Search to find suitable composite, thus it is called compound scaling. There are 8 levels of EfficientNet in total, and we choose EfficientNet-B3 as our backbone considering the difficulty of our task, input size, and model size.

Illustration of compound scaling strategy.

BiFPN is a feature pyramid network(FPN) with both top down and bottom up paths to combine feature maps, while the original FPN has only top down path. The purpose is to enhance the feature expression, so the bounding box regression and classification could perform better.

Comparison of different FPNs.

Implementation Details

There are some modifications that we have done to apply EfficientDet on our data. First, we reduce the size of anchor and the intersection of union(IoU) threshold for finding anchors correspond to ground truth boxes. The reason is that the smallest cell size is only around 8px, which is much smaller than the default base anchor size 32px. Also, since the boxes are quite small, matching ground truth boxes and anchors under a looser condition would make it easier to learn. Furthermore, we sample K matched anchors instead of all candidates for computing losses and update, K=20. Image size is set to 768×1024. 

For the small dataset with 108 training images and 12 testing images, we are able to reach a mAP 71%, recall 76% performance.

Figure of input ground truth data(bold frames) and predicted boxes(thin frames). The number is confidence score.

Conclusion

We demonstrate that modern machine learning/deep learning methods could apply to medical images, and are able to achieve satisfying performance. This model would help the surgeon to interpret the smear more easily, and even speed up the surgery as we are actively working on improving the model with more data. 

Ref:

[1] Tan, M. & Le, Q.. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:6105-6114

[2] M. Tan, R. Pang and Q. V. Le, “EfficientDet: Scalable and Efficient Object Detection,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 10778-10787, doi: 10.1109/CVPR42600.2020.01079.