BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//LORIA - ECPv6.17.2//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:LORIA
X-ORIGINAL-URL:https://www.loria.fr
X-WR-CALDESC:Évènements pour LORIA
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:Europe/Paris
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20200329T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20201025T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20210328T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20211031T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20220327T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20221030T010000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Paris:20210409T110000
DTEND;TZID=Europe/Paris:20210409T120000
DTSTAMP:20210407T092250Z
CREATED:20210407T092250Z
LAST-MODIFIED:20210407T092250Z
UID:11971-1617966000-1617969600@www.loria.fr
SUMMARY:MALOTEC Séminaire : Evaluating Local Explanation Methods on Ground Truth
DESCRIPTION:Evaluating local explanation methods is a difficult task due to the lack of a shared and universally accepted definition of explanation. In the literature\, one of the most common ways to assess the performance of an explanation method is to measure the fidelity of the explanation with respect to the classification of a black box model adopted by an Artificial Intelligent system for making a decision. However\, this kind of evaluation only measures the degree of adherence of the local explainer in reproducing the behavior of the black box classifier with respect to the final decision. Therefore\, the explanation provided by the local explainer could be different in the content even though it leads to the same decision of the AI system. We propose an approach that allows to measure to which extent the explanations returned by local explanation methods are correct with respect to a synthetic ground truth explanation. Indeed\, the proposed methodology enables the generation of synthetic transparent classifiers for which the reason for the decision taken\, i.e.\, a synthetic ground truth explanation\, is available by design. Experimental results show how the proposed approach allows to easily evaluate local explanations on the ground truth and to characterize the quality of local explanation methods. \n\n\nSpeaker:\nRiccardo Guidotti\, Assistant Professor at the Department of Computer Science (University of Pisa) and a member of the Knowledge Discovery and Data Mining Laboratory (KDDLab).\n\nOn TEAMS. More information at:\nhttps://malotec.loria.fr
URL:https://www.loria.fr/event/malotec-seminaire-evaluating-local-explanation-methods-on-ground-truth/
LOCATION:online
CATEGORIES:Séminaire
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=Europe/Paris:20210415T160000
DTEND;TZID=Europe/Paris:20210415T170000
DTSTAMP:20210401T090927Z
CREATED:20210401T090802Z
LAST-MODIFIED:20210401T090927Z
UID:11954-1618502400-1618506000@www.loria.fr
SUMMARY:DigiTrust Webinar : Enka Blanchard
DESCRIPTION:The second webinar of LUE IMPACT project DigiTrust will take place on Thursday\, 15th April at 4pm. \nPostdoctoral Researcher Enka Blanchard\, working in the DigiTrust consortium\, will give a presentation entitled “Securing everyday voting with low-tech systems” \nThe webinar will take place on Teams. \nAbstract: \nVoting is often seen as a solemn activity\, with voters exercising their rights every few years in secure conditions. However\, many voting activities happen in much more common situations\, from company boardrooms to homeowners associations or employee breakrooms. Those votes often happen by raising one’s hand or at best writing down a name on a piece of paper and putting it into a hat. With the pandemic\, those votes have been harder to organise\, creating a vacuum for new systems to take hold\, and potentially presenting new opportunities.\n\nThe talk will start with an introduction to low-tech verifiable voting systems. We will then go over the impact of the pandemic and how it led us to propose a solution that is currently in use at the University of Maryland\, Baltimore County\, as well as the details of this low-tech (non-cryptographic) verifiable voting system. Finally\, I will discuss the implications this has not only for the development but more importantly for the deployment of new voting systems.\n\n\nThe talk is based on research I recently pursued with colleagues from UMBC and LaBRI. The relevant papers and preprints are available below:\n\nEnka Blanchard and Ted Selker. Origami voting: a non-cryptographic approach to transparent ballot verification. In VOTING – 5th Workshop on Advances in Secure Electronic\nVoting (https://hal.archives-ouvertes.fr/hal-02550738)\nEnka Blanchard\, Ryan Robucci\, Ted Selker\, and Alan T. Sherman. Phrase-verified voting: Verifiable low-tech remote boardroom voting. under review\nEnka Blanchard\, Sébastien Bouchard\, and Ted Selker. Visual secrets: A recognition-based security primitive and its use for boardroom voting. under review
URL:https://www.loria.fr/event/digitrust-webinar-enka-blanchard/
LOCATION:online
CATEGORIES:Séminaire
ATTACH;FMTTYPE=image/png:https://www.loria.fr/wp-content/uploads/2021/04/Webinaire-Enka-Blanchard.png
END:VEVENT
BEGIN:VEVENT
DTSTART;VALUE=DATE:20210419
DTEND;VALUE=DATE:20210421
DTSTAMP:20201203T101217Z
CREATED:20201203T101217Z
LAST-MODIFIED:20201203T101217Z
UID:11302-1618790400-1618963199@www.loria.fr
SUMMARY:Workshop on Human Evaluation of NLP Systems (HumEval)
DESCRIPTION:EACL’21\, Kiev\, Ukraine\, 19-20 April 2021\nFirst Call for Papers\nThe HumEval Workshop invites the submission of long and short papers on substantial\, original\, and unpublished research on all aspects of human evaluation of NLP systems\, both intrinsic and extrinsic\, including but by no means limited to NLP systems whose output is language. More on: humeval.github.io. \nInvited Speakers\n\nMohit Bansal\, UNC Chapel Hill\, US\nMargaret Mitchell\, Google\, US\nLucia Specia\, UCL\, UK\n\nImportant Dates\n\nDec 2: First Call for Workshop Papers\nDec 18: Second Call for Workshop Papers\nJan 18: Workshop Paper Due Date\nFeb 18: Notification of Acceptance\nMar 01: Camera-ready papers due\nApr 19-20: Workshop Dates\nAll deadlines are 11.59 pm UTC-12.\n\nWorkshop Topic and Content\nHuman evaluation plays a central role in NLP\, from the large-scale crowd-sourced evaluations carried out e.g. by the WMT workshops\, to the much smaller experiments routinely encountered in conference papers. Moreover\, while NLP embraced automatic evaluation metrics from BLEU (Papineni et al\, 2001) onwards\, the field has always been acutely aware of their limitations (Callison-Burch et al.\, 2006; Reiter and Belz\, 2009; Novikova et al.\, 2017; Reiter\, 2018)\, and has gauged their trustworthiness in terms of how well\, and how consistently\, they correlate with human evaluation scores (Over et al.\, 2007; Gatt and Belz\, 2008; Bojar et al.\, 2016; Shimorina\, 2018; Ma et al.\, 2019; Mille et al.\, 2019; Dušek et al.\, 2020). \nYet there is growing unease about how human evaluations are conducted in NLP. Researchers have pointed out the less than perfect experimental and reporting standards that prevail (van der Lee et al.\, 2019). Only a small proportion of papers provide enough detail for reproduction of human evaluations\, and in many cases the information provided is not even enough to support the conclusions drawn. More than 200 different quality criteria (Fluency\, Grammaticality\, etc.) have been used in NLP  (Howcroft et al.\, 2020). Different papers use the same quality criterion name with different definitions\, and the same definition with different names. As a result\, we currently do not have a way of determining whether two evaluations assess the same thing which poses problems for both meta-evaluation and reproducibility assessments (Belz et al.\, 2020). \nReproducibility in the context of automatically computed system scores has recently attracted a lot of attention\, against the background of a troubling history (Pedersen\, 2008; Mieskes et al.\, 2019)\, where reproduction is perceived as failing in 24.9% of cases for own results\, and in 56.7% for another team’s (Mieskes et al.\, 2019). Initiatives have included the Reproducibility Challenge (Pineau et al.\, 2019\, Sinha et al.\, 2020); the Reproduction Paper special category at COLING’18; the reproducibility programme at NeurIPS’19 comprising code submission\, a reproducibility challenge\, and the ML Reproducibility checklist\, also adopted by EMNLP’20 and AAAI’21; and the REPROLANG shared task at LREC’20 (Branco et al.\, 2020). \nHowever\, reproducibility in the context of system scores obtained via human evaluations has barely been addressed at all\, with a tiny number of papers (e.g. Belz & Kow\, 2010; Cooper & Shardlow\, 2020) reporting attempted reproductions of results. The developments in reproducibility of automatically computed scores listed above are important\, but it is concerning that not a single one of the initiatives and events above addresses human evaluations. E.g. if a paper fully complies with all of the NeurIPS’19/EMNLP’20 reproducibility criteria\, any human evaluation results reported in it may not be reproducible to any degree\, simply because the criteria do not address human evaluation in any way. \nWith this workshop we wish to create a forum for current human evaluation research and future directions\, a space for researchers working with human evaluations to exchange ideas and begin to address the issues that human evaluation in NLP currently faces\, including aspects of experimental design\, reporting standards\, meta-evaluation and reproducibility. We invite papers on topics including\, but not limited to\, the following: \n\n\nExperimental design for human evaluations \n\n\nReproducibility of human evaluations \n\n\nEthical considerations in human evaluation of computational systems \n\n\nQuality assurance for human evaluation \n\n\nCrowdsourcing for human evaluation \n\n\nIssues in meta-evaluation of automatic metrics by correlation with human evaluations \n\n\nAlternative forms of meta-evaluation and validation of human evaluations \n\n\nComparability of different human evaluations \n\n\nMethods for assessing the quality of human evaluations \n\n\nMethods for assessing the reliability of human evaluations \n\n\nWork on measuring inter-evaluator and intra-evaluator agreement \n\n\nFrameworks\, model cards and checklists for human evaluation \n\n\nExplorations of the role of human evaluation in the context of Responsible AI and Accountable AI \n\n\nProtocols for human evaluation experiments in NLP \n\n\nWe welcome work on the above topics and more from any subfield of NLP (and ML/AI more generally)\, with a particular focus on evaluation of systems that produce language as output. We explicitly encourage the submission of work on both intrinsic and extrinsic evaluation. \nPaper Submission Information\nLong Papers:\nLong papers must describe substantial\, original\, completed and unpublished work. Wherever appropriate\, concrete evaluation and analysis should be included. \nLong papers may consist of up to eight (8) pages of content\, plus unlimited pages of references. Final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers’ comments can be taken into account. \nLong papers will be presented orally or as posters as determined by the programme committee. Cecisions as to which papers will be presented orally and which as posters will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between long papers presented orally and as posters. \nShort Papers:\nShort paper submissions must describe original and unpublished work. Short papers should have a point that can be made in a few pages. Examples of short papers are a focused contribution\, a negative result\, an opinion piece\, an interesting application nugget\, a small set of interesting results. \nShort papers may consist of up to four (4) pages of content\, plus unlimited pages of references. Final versions of short papers will be given one additional page of content (up to 5 pages) so that reviewers’ comments can be taken into account. \nShort papers will be presented orally or as posters as determined by the programme committee. While short papers will be distinguished from long papers in the proceedings\, there will be no distinction in the proceedings between short papers presented orally and as posters. \nReview forms will be made available prior to the deadlines. For more information on applicable policies\, see the ACL Policies for Submission\, Review\, and Citation. \nMultiple Submission Policy\nHumEval’21 allows multiple submissions. However\, if a submission has already been\, or is planned to be\, submitted to another event\, this must be clearly stated in the submission \nEthics Policy\nAuthors are required to honour the ethical code set out in the ACL Code of Ethics. \nThe consideration of the ethical impact of our research\, use of data\, and potential applications of our work has always been an important consideration\, and as artificial intelligence is becoming more mainstream\, these issues are increasingly pertinent. We ask that all authors read the code\, and ensure that their work is conformant to this code. Where a paper may raise ethical issues\, we ask that you include in the paper an explicit discussion of these issues\, which will be taken into account in the review process. We reserve the right to reject papers on ethical grounds\, where the authors are judged to have operated counter to the ACL Code of Ethics\, or have inadequately addressed legitimate ethical concerns with their work. \nPaper Submission and Templates\nSubmission is electronic\, using the Softconf START conference management system. For electronic submission of all papers\, please use: https://www.softconf.com/eacl2021/HumEval2021. Both long and short papers must follow the ACL Author Guidelines\, and must use the EACL’21 templates. You can find the EACL-2021 LaTeX template here or download the zip file. \nOrganisers\n\nAnya Belz\, University of Brighton\, UK\nShubham Agarwal\, Heriot Watt University\, UK\nYvette Graham\, Trinity College Dublin\, Ireland\nEhud Reiter\, University of Aberdeen\nAnastasia Shimorina\, Université de Lorraine / LORIA\n\nPC Members\n\n\n\n\n\n\n\n\nMohit Bansal\, UNC Chapel Hill\, US \n\n\nSaad Mahamood\, Trivago\, DE \n\n\n\n\nKevin B. Cohen\, University of Colorado\, US \n\n\nNitika Mathur\, University of Melbourne\, Australia \n\n\n\n\nKees van Deemter\, Utrecht University\, NL \n\n\nMargot Mieskes\, UAS Darmstadt\, DE \n\n\n\n\nOndrej Dusek\, Charles University\, Czechia \n\n\nEmiel van Miltenburg\, Tilburg University\, NL \n\n\n\n\nKarën Fort\, Sorbonne University\, France \n\n\nMargaret Mitchell\, Google\, US \n\n\n\n\nAnette Frank\, University of Heidelberg\, DE \n\n\nMathias Mueller\, University of Zurich\, CH \n\n\n\n\nClaire Gardent\, CNRS/LORIA Nancy\, France \n\n\nMalvina Nissim\, Groningen University\, NL \n\n\n\n\nAlbert Gatt\, Malta University\, Malta \n\n\nJuri Opitz\, University of Heidelberg\, DE \n\n\n\n\nDimitra Gkatzia\, Edinburgh Napier University\, UK \n\n\nRamakanth Pasunuru\, UNC Chapel Hill\, US \n\n\n\n\nHelen Hastie\, Heriot-Watt University\, UK \n\n\nMaxime Peyrard\, EPFL\, CH \n\n\n\n\nDavid Howcroft\, Heriot Watt University\, UK \n\n\nInioluwa Deborah Raji\, Ai Now Institute\, US \n\n\n\n\nJackie Chi Kit Cheung\, McGill University\, Canada \n\n\nVerena Rieser\, Heriot Watt University\, UK \n\n\n\n\nSamuel Läubli\, University of Zurich\, CH \n\n\nSamira Shaikh\, UNC\, US \n\n\n\n\nChris van der Lee\, Tilburg University\, NL \n\n\nLucia Specia\, UCL\, UK \n\n\n\n\nNelson Liu\, Washington University\, US \n\n\nWei Zhao\, TU Darmstadt\, DE \n\n\n\n\nQun Liu\, Huawei Noah’s Ark Lab\, China \n\n\n\n\n\n\nContact Information\nhumeval.ws@gmail.com \nhttps://humeval.github.io
URL:https://www.loria.fr/event/workshop-on-human-evaluation-of-nlp-systems-humeval/
CATEGORIES:Séminaire
END:VEVENT
END:VCALENDAR