Skip to main content

Table 1 Reviewed methods and their total scores

From: A survey on artificial intelligence assurance

Year

First author's last name and citation

Publishing venue

AI subarea

Total score

2020

D'Alterio [50]

FUZZ-IEEE

XAI

10

2019

Tao [208]

IEEE Access

Generic

10

2020

Anderson [11]

ACM TIIS

RL

9

2020

Birkenbihl [29]

EPMA

ML

9

2020

Checco [39]

JAIR

DS

9

2020

Chen [40]

IEEE Access

XAI

9

2020

Cluzeau [43]

EASA

DL

9

2019

Kaur [109]

WAINA

XAI

9

2020

Kulkarni [117]

Academic Press

DS

9

2020

Kuppa [118]

IEEE IJCNN

XAI

9

2020

Kuzlu [120]

IEEE Access

XAI

9

2021

Massoli [145]

CVIU

DL

9

2020

Spinner [201]

IEEE TVCG

XAI

9

2016

Veeramachaneni [226]

IEEE HPSC

DS

9

2018

Wei [230]

AS

RL

9

2020

Winkel [236]

EJR

RL

9

2014

Ali [8]

GISci

DS

8

2018

Alves [9]

NASA ARIAS

ABS

8

2019

Batarseh [24]

EDML

DS

8

2016

Gao [71]

SEKE

DS

8

2020

Gardiner [72]

Nature Sci Rep

ML

8

2016

Gulshan [81]

JAMA

CV

8

2020

Guo [82]

IEEE ICCVW

XAI

8

2020

Han [87]

IET JoE

XAI

8

2016

Heaney [93]

OD

GA

8

2019

Huber [97]

KI AAI

RL

8

2019

Keneni [112]

IEEE Access

XAI

8

2020

Kohlbrenner [116]

IEEE IJCNN

XAI

8

2019

Maloca [134]

PLoS ONE

DL

8

2020

Malolan [136]

IEEE ICICT

XAI

8

2020

Payrovnaziri [168]

JAMIA

ML

8

2008

Peppler [170]

OASJ

DS

8

2020

Sequeira [196]

SciDir AI

RL

8

2020

Sivamani [199]

IEEE LCS

DL

8

2020

Tan [207]

IEEE IJCNN

XAI

8

2020

Tao [209]

IEEE CoG

XAI

8

2020

Welch [231]

PhysMedBiol

DL

8

2020

Xiao [239]

IS

DL

8

2016

Aitken [6]

UC

ABS

7

2019

Barredo-Arrieta [19]

IEEE ITSC

XAI

7

2013

Batarseh [20]

IEEE TSMCS

KBS

7

2001

Berndt [27]

COMP

DS

7

2010

Bone [30]

CEUS

RL

7

2016

Celis [38]

PrePrint

ML

7

2019

Chittajallu [42]

IEEE ISBI

XAI

7

2018

Elsayed [67]

NIPS

CV

7

2019

Ferreyra [69]

FUZZ-IEEE

XAI

7

2006

Forster [70]

Uni of South Africa

AGI

7

1985

Ginsberg [74]

IJCAI

KBS

7

2018

Go [76]

ACM CCS

DL

7

2020

Halliwell [86]

PrePrint

DL

7

2015

He [90]

MPE

GA

7

2020

Heuer [94]

ACM UMAP

ML

7

2016

Jiang [102]

PMLR

RL

7

2020

Kaur [111]

AINA

XAI

7

2016

Kianifar [113]

SC

GA

7

2019

Lee [121]

IEEE ICTC

XAI

7

2017

Liang [126]

MILCOM

DS

7

2020

Mackowiak [132]

PrePrint

CV

7

2018

Mason [143]

AHIM

RL

7

2018

Murray [157]

FUZZ-IEEE

XAI

7

2019

Naqa [66]

MedPhys

ML

7

2019

Prentzas [174]

IEEE BIBE

XAI

7

2018

Pynadath [177]

Springer HCIS

ML

7

2020

Ragot [179]

CHI

ML

7

2020

Rotman [184]

PrePrint

RL

7

2015

Rovcanin [185]

WN

RL

7

2020

Sarathy [188]

IEEE SISY

XAI

7

2018

Stock [203]

ECCV

CV

7

2009

Tadj [206]

SCI

KBS

7

1999

Thomas [214]

AAAI

GA

7

2020

Uslu [220]

AINA

XAI

7

2018

Xu [240]

PrePrint

DL

7

2019

Bellamy [26]

IBM JRD

XAI

6

2019

Beyret [28]

IEEE IROS

RL

6

2018

Cao [35]

JAIHC

ML

6

2020

Cruz [47]

PrePrint

RL

6

2001

Halkidi [85]

JIIS

ML

6

2020

He [91, 92]

PrePrint

RL

6

2020

Islam [98]

IEEE TFS

XAI

6

2005

Liu [128]

AI2005

DL

6

2019

Madumal [133]

PrePrint

RL

6

1996

Martin [138]

ERIC

DS

6

2007

Martín-Guerrero [141]

AJCAI

RL

6

2000

Mosqueira-Rey [155]

ESA

KBS

6

2020

Mynuddin [160]

IETITS

RL

6

2020

Puiutta [175]

CD-MAKE

RL

6

2018

Ruan [187]

IJCAI

DL

6

2019

Schlegel [193]

IEEE ICCVW

XAI

6

2020

Toreini [216, 217]

ACM FAT

ML

6

2020

Toreini [216, 217]

PrePrint

ML

6

2019

Vabalas [222]

PLoS ONE

ML

6

2010

Winkler [237]

IEEE SUTC

CV

6

2002

Wu [238]

IJHCS

KBS

6

2019

Zhu [246]

ACM PLDI

RL

6

1992

Andert [12]

IJM

KBS

5

2018

Antunes [14]

IEEE DSN-W

ML

5

1989

Becker [25]

NASA

KBS

5

2019

Chen [41]

CS

RL

5

2019

Cruz [46]

AI 2019 AAI

RL

5

2020

Diallo [57]

IEEE ACSOS-C

XAI

5

2010

Dong [62]

IEEE ICWIIAT

GA

5

2019

Dupuis [64]

UoG

XAI

5

2015

Goodfellow [79]

PrePrint

ML

5

2020

Guo [82]

IEEE CM

XAI

5

2020

Haverinen [89]

Uni of Jyväskylä

XAI

5

1997

Jones [104]

JMB

GA

5

2019

Joo [106]

IEEE CoG

RL

5

2020

Katell [107]

ACM FAT

XAI

5

2007

Knauf [115]

IEEE TSMC

KBS

5

1995

Lockwood [130]

AES

KBS

5

2000

Marcos [137]

IEE Proc

KBS

5

2017

Mason [144]

WhiteRose

RL

5

1988

Morell [154]

IEA/AIE

KBS

5

2020

Murray [158]

IEEE TETCI

XAI

5

2010

Niazi [162]

SpringSim

ABS

5

2000

Onoyama [166]

JETAI

KBS

5

2019

Ren [182]

PrePrint

DL

5

2013

Sargent [189]

JoS

ABS

5

2003

Schumann [195]

EANN

DL

5

1995

Singer [198]

POQ

DS

5

2019

Srivastava [202]

AAAI AIES

NLP

5

2006

Taylor [211]

Springer

DL

5

2020

Taylor [213]

IEEE CVPRW

XAI

5

2020

Tjoa [215]

IEEE TNNLS

ML

5

2020

Uslu [221]

BWCCA

XAI

5

2020

Varshney [225]

IEEE CISS

ML

5

2018

Volz [228]

IEEE CIG

XAI

5

2020

Wieringa [234]

ACM FAT

XAI

5

2020

Wing [235]

PrePrint

ML

5

2019

Yoon [242]

IEEE ICCVW

XAI

5

2019

Zhou [245]

IJCAI XAI

ML

5

1994

Zlatareva [249]

ESA

KBS

5

2018

AI Now (Algorithmic Accountability Policy Tooklit) [7]

AI Now

XAI

4

2015

Arifin [15]

Springer

ABS

4

2015

Batarseh [21]

AIR

KBS

4

2007

Brancovici [31]

IEEE CEC

XAI

4

1987

Castore [37]

NASA STI

KBS

4

2013

Cohen [45]

EternalS

NLP

4

2020

Das [51]

PrePrint

XAI

4

2013

David [52]

UCS

ABS

4

2018

Došilović [63]

MIPRO

ML

4

2000

Edwards [65]

Oxford

DS

4

2018

EY (Assurance in the Age of AI) [17]

EY

ML

4

2019

Guidotti [80]

ACM CS

XAI

4

2018

Jilk [103]

PrePrint

ABS

4

2017

Leibovici [123]

ISPRS Int J. Geo-Inf

DS

4

2020

Li [125]

IEEE TKDE

XAI

4

2019

Mehrabi [146]

PrePrint

ML

4

2019

Meskauskas [150]

FUZZ-IEEE

XAI

4

1998

Miller [151]

MS

GA

4

2019

Nassar [161]

WIREs DMKD

XAI

4

1992

Preece [173]

ESA

KBS

4

2019

Qiu [178]

AS

Generic

4

1984

Sargent [190]

IEEE WSC

ABS

4

2003

Taylor [212]

SPIE

DL

4

1999

Tsai [218]

IEEE TKDE

KBS

4

1991

Vinze [227]

IM

KBS

4

2019

Wang [229]

ACM CHI

XAI

4

1993

Wells [232]

AAAI

KBS

4

2018

Zhu [247]

IEEE CIG

XAI

4

1998

Zlatareva [248]

DBLP

KBS

4

2018

Abdollahi [1]

Springer

ML

3

1997

Abel [2]

FLAIRS Conference

KBS

3

2018

Adadi [4]

IEEE Access

XAI

3

2018

Agarwal [5]

PrePrint

Generic

3

2016

Amodei [10]

PrePrint

ML

3

2019

Breck [32]

SysML

ML

3

1996

Carley [36]

CASOS

KBS

3

2000

Coenen [44]

CUP

KBS

3

1987

Culbert [48]

NASA SOAR

KBS

3

2020

Dağlarli [49]

ADL

XAI

3

1992

Davis [53]

RAND

ABS

3

2020

Dodge [61]

ExSS-ATEC

XAI

3

2018

Everitt [68]

IJCAI

AGI

3

1991

Gilstrap [73]

TI

KBS

3

2019

Glomsrud [75]

ISSAV

XAI

3

1996

Gonzalez [78]

EAAI

KBS

3

1997

Harmelen [88]

EUROVAV

KBS

3

2019

He [91, 92]

PrePrint

DL

3

2020

Heuillet [95]

PrePrint

RL

3

2009

Hibbard [96]

AGI

AGI

3

2019

Israelsen [99]

ACM CSUR

Generic

3

2019

Jha [101]

NeurIPS

DL

3

2002

Knauf [114]

IEEE TSMC

KBS

3

2017

de Laat [54]

PhilosTechnol

ML

3

1994

Lee [122]

IEEE TSMC

KBS

3

2004

Liu [127]

IEEE MLC

ABS

3

1997

Lowry [131]

ISMIS

Generic

3

2012

Martinez-Balleste [139]

IEEE SIPC

CV

3

2020

Martinez-Fernandez [140]

PrePrint

XAI

3

2017

Mason [142]

DCAART

RL

3

1993

Mengshoel [148]

IEEE exp

KBS

3

2005

Menzies [149]

AC

Generic

3

2007

Min [152]

WSC

KBS

3

1997

Murrell [159]

DSS

KBS

3

1987

O'Keefe [164]

IEEE exp

KBS

3

2020

Putzer [176]

PrePrint

XAI

3

1991

De Raedt [55]

JWS

KBS

3

2020

Raji [180]

ACM FAT

XAI

3

2004

Sargent [191]

IEEE WSC

ABS

3

1990

Suen [204]

ESA

KBS

3

2019

Sun [205]

IEEE VTC

XAI

3

2006

Yilmaz [241]

CMOT

ABS

3

1997

Zaidi [243]

Automatica

KBS

3

1996

Abel [3]

FLAIRS Conference

KBS

2

2016

Aitken [6]

PrePrint

ABS

2

1998

Antoniou [13]

AI Magazine

KBS

2

2019

Arrieta [16]

SciDir IF

XAI

2

2018

Bride [34]

ICFEM

XAI

2

2020

Dghaym [56]

AU SSAV

XAI

2

2015

Dobson [59]

JCLS

ML

2

2018

Hagras [83]

IEEE Comp

XAI

2

1999

Hailu [84]

IEEE SMC

RL

2

2020

He [91, 92]

IEEE IRCE

XAI

2

2016

Janssen [100]

GIQ

DS

2

2020

Kaur [110]

NBiS

XAI

2

2008

Liu [129]

IEEE SSSC

ABS

2

2006

Min [153]

ICMLC

KBS

2

2019

Mueller [156]

PrePrint

XAI

2

1996

Nourani [163]

ACM SIGSOFT

Generic

2

2020

Pawar [167]

IEEE CyberSA

XAI

2

2009

Pèpe [169]

JCG

GA

2

2013

Pitchforth [171]

ESA

DL

2

2017

Protiviti (Validation of Machine Learning Models) [223]

Protiviti

ML

2

2010

Sargent [192]

WSC

ABS

2

2019

Spada [200]

AIAI

XAI

2

2005

Taylor [210]

IEEE IJCNN

DL

2

2016

Zeigler [244]

JDMS

ABS

2

2001

Barr [18]

ACL

NLP

1

2020

Brennen [33]

ACM CHI EA

XAI

1

2006

Dibie-Barthélemy [58]

KBS

KBS

1

2020

European Commission (A European Approach to Excellence and Trust) [165]

European Commission

XAI

1

2000

Gonzalez [77]

JETAI

Generic

1

2018

Kaul [108]

ACM AIES

ML

1

2003

Kurd [119]

SAFECOMP

DL

1

2017

Lepri [124]

PhilosTechnology

ML

1

2018

Mehri [147]

ACM ARES

DL

1

2019

Pocius [172]

AAAI-19

RL

1

2019

Rossi [183]

JIA

XAI

1

2010

Schumann [194]

NASA SCI

DL

1

2018

Sileno [197]

PrePrint

XAI

1

2019

Varshney [224]

ACM XRDS

ML

1

2016

Wickramage [233]

FTC

DS

1