[PhD topic] Characterising and exploring the space of neural network models

PhD subject. Supervisors: Mathieu d’Aquin and Emmanuel Nauer

This a call for applications to a PhD studentship, subject to funding approval, within the K Team of the LORIA Laboratory. Please contact Mathieu d’Aquin (mathieu dot daquin at loria dot fr) and Emmanuel Nauer (emmanuel dot nauer at loria dot fr) if interested to apply (before May 15th 2022).

Context

There has been a dramatic increase in the last few years in the use of machine learning techniques in all sorts of applications. As a result, libraries such as Keras [4] have been created and have quickly become de facto standards in the creation of neural network models for machine learning. Also, besides code shared on common platforms such as github, new online tools for sharing and evaluating such code have become increasingly popular, with platforms such as Kaggle therefore hosting the code, data and results for thousands of different models.

With the practices of creating models for specific tasks and data, an increasing need has also emerged for model management [6]. Indeed, with each machine learning project comes numerous versions of
a single model based on testing different configurations, data engineering methods and applying hyper-parameter optimisation [2]. On an example such as [1], hundreds of models are trained based on combining models, using different data as input, finding optimal parameters and carrying out an ablation study. Some of those models (or at least the code to produce them) might appear online, and some of the results will get published, but much of the knowledge encapsulated in those will be lost, possibly to be re-discovered, multiple times, at a later stage.

Some tools already exist to help manage the proliferation of models in a given project (see for example neptune.ai) or to support finding models that have been shared (see for example [7]). However, those tools remain limited in scope and do not allow the users to explore the space of existing models with precise questions such as: “What is the most common structure of a neural network for this kind of tasks?”, “What kind of model has been used on that kind of data?”, or ”Have other models found results like mine, and what does it mean?” This can only be realised through semantically describing models in a way that can be interpreted and explored.

Objective of the PhD

The overall objective of this PhD is therefore to study, research and resolve the various problems and issues that are raised by the construction of a knowledge graph of neural network models. A knowledge graph is a semantic description of entities according to an ontology [3] in such a way that the data about those entities can be navigated and explored meaningfully. Some of the anticipated contributions of this PhD therefore include:

  • Establishing a process to identify and extract models from online repositories such as Kaggle.
  • Identifying the characteristics useful to describe those models (and their results, data, etc.) and define those in an ontology.
  • Devising methods to automatically extract such characteristics for example from code available online.
  • Devising methods for searching and exploring the knowledge graph according to those characteristics.
  • Study applications of such methods, for example for model interpretation [5] and model reuse, in specific domains, including healthcare and the digital humanities.

References

[1] Lucas Azevedo, Mathieu d’Aquin, Brian Davis, and Manel Zarrouk. Lux (linguistic aspects under examination): Discourse analysis for automatic fake news classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 41–56, 2021.


[2] Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated machine learning, pages 3–33. Springer, Cham, 2019.


[3] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, Jos ́e Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. Knowledge graphs. Synthesis Lectures on Data, Semantics, and Knowledge, 12(2):1–257, 2021.


[4] Nikhil Ketkar. Introduction to keras. In Deep learning with Python, pages 97–111. Springer, 2017.


[5] Andriy Nikolov and Mathieu d’Aquin. Uncovering semantic bias in neural network models using a knowledge graph. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1175–1184, 2020.


[6] Sebastian Schelter, Felix Biessmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. On challenges in machine learning model management. IEEE Data Eng. Bull., 2018.


[7] Manasi Vartak, Harihar Subramanyam, Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. Modeldb: a system for machine learning model management. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, pages 1–3, 2016.

Logo d'Inria