Expert Evaluations: Round 1

Author

T. Eerola and C. Kirts

This analysis details the Round 1 of the expert ratings. The data consist of ratings of 329 items across 24 constructs rated by 14 experts.

Show the code 
rm(list=ls())  # clear memory
library(tidyverse) 
library(stringr)
library(psychometric)
library(ggplot2)
library(CVI)
#packageVersion("CVI") # Should be 0.1.4.0 at least

source('scr/read_episode_data.R') # requires ToDo! 
source('scr/preprocess_qualtrics.R') # Not Finished, Complete
source('scr/analyse_episode_data.R') # Not tested
source('scr/add_constructs.R') #Needs work

# write the data to file (as an option)
file_write <- FALSE

Read and preprocess the data

Show the code 
FM  <- read_episode_data("data/FM_October312024.csv")
CB  <- read_episode_data("data/CB_October312024.csv")
EDR <- read_episode_data("data/EDR_October312024.csv")
AIA <- read_episode_data("data/AIA_October312024.csv")
PEP <- read_episode_data("data/PEP_October312024.csv")
LMA <- read_episode_data("data/LMA_October312024.csv")
MM  <- read_episode_data("data/MM_October312024.csv")
QRE <- read_episode_data("data/QRE_October312024.csv")

FM <- preprocess_qualtrics(FM) 
CB <- preprocess_qualtrics(CB)
EDR <- preprocess_qualtrics(EDR)
AIA <- preprocess_qualtrics(AIA)
PEP <- preprocess_qualtrics(PEP)
LMA <- preprocess_qualtrics(LMA)
MM <- preprocess_qualtrics(MM)
QRE <- preprocess_qualtrics(QRE)

#source('scr/coder_initials.R') # Anonymise the experts in the end

Define constructs and merge into a data frame

Show the code 
FM_constructs <- add_constructs(FM$rating,'FM')
CB_constructs <- add_constructs(CB$rating,'CB')
EDR_constructs <- add_constructs(EDR$rating,'EDR')
AIA_constructs <- add_constructs(AIA$rating,'AIA')
PEP_constructs <- add_constructs(PEP$rating,'PEP')
LMA_constructs <- add_constructs(LMA$rating,'LMA')
MM_constructs <- add_constructs(MM$rating,'MM')
QRE_constructs <- add_constructs(QRE$rating,'QRE')

# combine all constructs to a single data frame
DF <- rbind(FM_constructs,
            CB_constructs,
            EDR_constructs,
            AIA_constructs,
            PEP_constructs,
            LMA_constructs,
            MM_constructs,
            QRE_constructs)

DF$IDCode[DF$IDCode==""]<-"Unknown" # replace empty IDCode with "Unknown

Summarise the observations

Show the number of ratings across constructs and experts. As we can observe, we have some minor gaps in the data as not all the experts have completed the ratings for all the constructs.

Show the code 
#knitr::kable(table(DF$construct,DF$IDCode))
DF <- dplyr::filter(DF,IDCode!="Unknown") # remove IDCode = Unknown
DF <- dplyr::filter(DF,IDCode!="KDU") # remove IDCode = KDU
# remove double coded expert at coder_initials,R
# length(unique(DF$IDCode)) # number of experts

DF$IDCode <- factor(DF$IDCode,labels = paste0('Ex',sprintf('%0.2d',1:length(unique(DF$IDCode)))))
knitr::kable(table(DF$construct,DF$IDCode))
Ex01 Ex02 Ex03 Ex04 Ex05 Ex06 Ex07 Ex08 Ex09 Ex10 Ex11 Ex12 Ex13 Ex14
Aesthetics 14 14 0 14 14 14 14 14 14 0 14 14 14 14
Associative 18 18 18 18 18 36 18 0 18 0 18 18 18 18
Belonging 15 15 0 15 15 15 15 15 15 0 15 15 15 15
Bodily 18 18 18 18 18 36 18 0 18 0 18 18 18 18
Connection 17 17 0 17 17 17 17 17 17 0 17 17 17 17
Coping 14 14 0 14 14 14 14 14 0 14 14 14 14 14
Curiosity 15 15 0 15 15 15 15 15 15 0 15 15 15 15
Diffuse 14 14 14 14 14 28 14 0 14 0 14 14 14 14
Distraction 17 17 0 17 17 17 17 17 17 0 17 17 17 17
Emotional 13 13 13 13 13 26 13 0 13 0 13 13 13 13
Energy Control 17 17 0 17 17 17 17 17 17 0 17 17 17 17
Enjoyment 13 13 0 13 13 13 13 13 13 0 13 13 13 13
Familiar 6 6 0 6 6 6 6 6 6 0 6 0 6 6
Feeling 16 16 0 16 16 16 16 16 0 16 16 16 16 16
Focus 14 14 0 14 14 14 14 14 14 0 14 14 14 14
Moved 12 12 0 12 12 12 12 12 12 0 12 12 12 12
Preference 8 8 0 8 8 8 8 8 8 0 8 0 8 8
Qualia 6 6 0 6 6 6 6 6 6 0 6 0 6 6
Reduced 11 11 11 11 11 22 11 0 11 0 11 11 11 11
Relaxation 16 16 0 16 16 16 16 16 16 0 16 16 16 16
Self 19 19 0 19 19 38 19 19 0 19 19 19 19 19
Source 19 19 0 19 19 38 19 19 0 19 19 19 19 19
Structural 8 8 8 8 8 16 8 0 8 0 8 8 8 8
Structure 16 16 0 16 16 32 16 16 0 16 16 16 16 16
Show the code 
# rename RC to CC in Coping
library(stringr)
DF$name[DF$construct=='Coping']<-str_replace(DF$name[DF$construct=='Coping'],'RC','CC')
#knitr::kable(table(DF$name,DF$construct))

Calculate Content Validity

Here we use the CVI package. In the analysis of the items in Round 1, we rely on CVI (adjusted) and the same for the figures.

Show the code 
# Calculate CVI for each construct 
Constructs <- unique(DF$construct)
Items<-list()
Items_trimmed <- list()
Scales_full <- list()
Scales_trimmed <- list()
Item_N <- NULL
Item_N_trimmed <- NULL
for (i in 1:length(Constructs)){
  Items[[i]] <- CVI_item(DF[DF$construct==Constructs[i],2:4])
  Item_N[i] <- nrow(Items[[i]])
  Items_trimmed[[i]] <- Items[[i]][Items[[i]]$CVIFit=='Appropriate',]
  Item_N_trimmed[i] <- nrow(Items_trimmed[[i]])
  Scales_full[[i]] <- CVI_scale(Items[[i]])  
  Scales_trimmed[[i]] <- CVI_scale(Items_trimmed[[i]])  
}
Construct_table <- data.frame(Constructs = Constructs, 
                             Item_n = Item_N,
                             SCVI_Ave = unlist(Scales_full),
                             Item_n_T = Item_N_trimmed,
                             SCVI_AveT = unlist(Scales_trimmed)
)
knitr::kable(Construct_table,digits=2)
Constructs Item_n SCVI_Ave Item_n_T SCVI_AveT
Energy Control 17 0.63 5 0.92
Focus 14 0.72 5 0.95
Connection 17 0.85 11 0.95
Belonging 15 0.80 11 0.92
Relaxation 16 0.91 15 0.93
Enjoyment 13 0.72 8 0.89
Distraction 17 0.76 9 0.89
Moved 12 0.83 8 0.96
Curiosity 15 0.72 9 0.88
Aesthetics 14 0.74 6 0.96
Coping 14 0.85 9 0.98
Feeling 16 0.80 10 0.92
Diffuse 14 0.77 7 0.90
Bodily 18 0.80 11 0.94
Emotional 13 0.83 9 0.90
Associative 18 0.83 12 0.93
Structural 8 0.84 7 0.93
Reduced 11 0.85 8 0.92
Structure 16 0.82 8 0.91
Self 19 0.87 14 0.96
Source 19 0.91 16 0.95
Qualia 6 0.97 6 0.97
Preference 8 0.67 3 0.94
Familiar 6 0.91 5 0.95

Visualise item relevance across constructs and sub-constructs

Enjoyment-Distraction-Relaxation

Enjoyment
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
RX1 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX3 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
RX6 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
RX7 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX8 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX9 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX11 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
RX12 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
RX13 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX15 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
RX18 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
RX21 8 12 0.67 0.33 0.62 Eliminated Good Reject
RX22 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
RX24 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
RX25 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
RX26 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
Distraction
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
EJ1 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EJ2 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EJ5 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
EJ7 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
EJ8 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
EJ9 7 12 0.58 0.17 0.48 Eliminated Fair Reject
EJ10 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EJ12 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
EJ13 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EJ16 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EJ17 7 12 0.58 0.17 0.48 Eliminated Fair Reject
EJ20 2 11 0.18 -0.64 0.16 Eliminated Poor Reject
EJ22 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
Relaxation
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
DT1 8 12 0.67 0.33 0.62 Eliminated Good Reject
DT2 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
DT3 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
DT5 9 12 0.75 0.50 0.74 Needs revision Good Reject
DT7 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
DT8 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
DT9 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
DT10 9 12 0.75 0.50 0.74 Needs revision Good Reject
DT11 7 10 0.70 0.40 0.66 Needs revision Good Reject
DT14 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
DT15 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
DT16 8 12 0.67 0.33 0.62 Eliminated Good Reject
DT18 7 12 0.58 0.17 0.48 Eliminated Fair Reject
DT19 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
DT20 6 12 0.50 0.00 0.35 Eliminated Poor Reject
DT21 4 12 0.33 -0.33 0.24 Eliminated Poor Reject
DT23 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67

Connection-Belonging

Connection
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
GC1 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC2 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC3 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC4 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
GC6 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC8 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC10 9 12 0.75 0.50 0.74 Needs revision Good Reject
GC11 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
GC12 8 12 0.67 0.33 0.62 Eliminated Good Reject
GC13 9 12 0.75 0.50 0.74 Needs revision Good Reject
GC14 8 12 0.67 0.33 0.62 Eliminated Good Reject
GC16 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
GC18 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
GC19 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
GC20 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
GC22 7 12 0.58 0.17 0.48 Eliminated Fair Reject
GC24 7 12 0.58 0.17 0.48 Eliminated Fair Reject
Belonging
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
SL1 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
SL2 9 12 0.75 0.50 0.74 Needs revision Good Reject
SL3 7 12 0.58 0.17 0.48 Eliminated Fair Reject
SL4 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
SL6 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
SL7 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
SL9 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
SL10 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
SL12 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
SL15 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
SL17 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
SL18 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
SL19 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
SL20 2 12 0.17 -0.67 0.15 Eliminated Poor Reject
SL22 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67

Focus-Motivation

Energy
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
MF2 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MF3 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
MF4 6 12 0.50 0.00 0.35 Eliminated Poor Reject
MF5 9 12 0.75 0.50 0.74 Needs revision Good Reject
MF6 7 12 0.58 0.17 0.48 Eliminated Fair Reject
MF7 6 11 0.55 0.09 0.41 Eliminated Fair Reject
MF8 8 12 0.67 0.33 0.62 Eliminated Good Reject
MF11 9 12 0.75 0.50 0.74 Needs revision Good Reject
MF12 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MF15 4 12 0.33 -0.33 0.24 Eliminated Poor Reject
MF18 7 12 0.58 0.17 0.48 Eliminated Fair Reject
MF20 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
MF21 8 12 0.67 0.33 0.62 Eliminated Good Reject
MF23 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67

Table: Focus

Personal Emotional Processing

Coping
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
CC1 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC3 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC4 6 12 0.50 0.00 0.35 Eliminated Poor Reject
CC5 9 12 0.75 0.50 0.74 Needs revision Good Reject
CC6 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC7 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
CC9 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC13 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC17 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC20 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
CC22 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
CC25 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CC18 7 10 0.70 0.40 0.66 Needs revision Good Reject
CC26 8 11 0.73 0.45 0.70 Needs revision Good Reject
Feeling
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
EF2 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EF3 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EF4 9 12 0.75 0.50 0.74 Needs revision Good Reject
EF6 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
EF7 8 12 0.67 0.33 0.62 Eliminated Good Reject
EF9 6 12 0.50 0.00 0.35 Eliminated Poor Reject
EF10 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EF12 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
EF14 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
EF15 9 12 0.75 0.50 0.74 Needs revision Good Reject
EF17 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EF19 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
EF24 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
EF20 7 12 0.58 0.17 0.48 Eliminated Fair Reject
EF22 4 12 0.33 -0.33 0.24 Eliminated Poor Reject
EF23 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67

Aesthetic–Interest–Awe

Moved
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
MS1 9 12 0.75 0.50 0.74 Needs revision Good Reject
MS2 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MS3 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
MS4 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MS7 9 12 0.75 0.50 0.74 Needs revision Good Reject
MS9 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
MS11 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
MS12 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MS14 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
MS15 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
MS17 7 12 0.58 0.17 0.48 Eliminated Fair Reject
MS20 3 12 0.25 -0.50 0.21 Eliminated Poor Reject
Curiosity
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
CU1 7 12 0.58 0.17 0.48 Eliminated Fair Reject
CU2 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
CU3 7 12 0.58 0.17 0.48 Eliminated Fair Reject
CU4 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
CU5 8 12 0.67 0.33 0.62 Eliminated Good Reject
CU6 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
CU8 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CU10 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
CU11 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
CU12 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
CU13 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
CU14 2 12 0.17 -0.67 0.15 Eliminated Poor Reject
CU16 6 12 0.50 0.00 0.35 Eliminated Poor Reject
CU17 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
CU18 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
Aesthetics
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
AE3 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
AE4 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
AE5 8 12 0.67 0.33 0.62 Eliminated Good Reject
AE6 7 12 0.58 0.17 0.48 Eliminated Fair Reject
AE7 9 12 0.75 0.50 0.74 Needs revision Good Reject
AE8 7 12 0.58 0.17 0.48 Eliminated Fair Reject
AE9 8 12 0.67 0.33 0.62 Eliminated Good Reject
AE10 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
AE12 6 12 0.50 0.00 0.35 Eliminated Poor Reject
AE14 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
AE15 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
AE17 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
AE19 6 12 0.50 0.00 0.35 Eliminated Poor Reject
AE1 9 11 0.82 0.64 0.81 Appropriate Excellent Reject

Listening Modes and Agency

Diffuse
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
DF2 9 13 0.69 0.38 0.66 Eliminated Good Reject
DF3 7 12 0.58 0.17 0.48 Eliminated Fair Reject
DF5 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
DF6 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
DF7 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
DF8 9 13 0.69 0.38 0.66 Eliminated Good Reject
DF9 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
DF10 8 13 0.62 0.23 0.54 Eliminated Fair Reject
DF11 9 13 0.69 0.38 0.66 Eliminated Good Reject
DF12 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
DF13 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
DF15 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
DF16 5 13 0.38 -0.23 0.27 Eliminated Poor Reject
DF17 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
Bodily
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
BA1 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
BA2 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
BA3 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
BA4 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
BA5 7 12 0.58 0.17 0.48 Eliminated Fair Reject
BA6 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
BA9 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
BA10 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
BA11 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
BA12 7 13 0.54 0.08 0.42 Eliminated Fair Reject
BA13 5 13 0.38 -0.23 0.27 Eliminated Poor Reject
BA14 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
BA15 5 13 0.38 -0.23 0.27 Eliminated Poor Reject
BA18 8 13 0.62 0.23 0.54 Eliminated Fair Reject
BA19 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
BA20 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
BA21 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
BA22 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
Emotional
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
EA1 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
EA2 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
EA3 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
EA7 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
EA8 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
EA9 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
EA10 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
EA13 6 13 0.46 -0.08 0.32 Eliminated Poor Reject
EA16 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
EA17 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
EA19 8 12 0.67 0.33 0.62 Eliminated Good Reject
EA20 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
EA21 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
Associative
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
AA1 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
AA2 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
AA3 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
AA4 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
AA5 7 13 0.54 0.08 0.42 Eliminated Fair Reject
AA6 12 12 1.00 1.00 1.00 Appropriate Excellent Accept > 0.67
AA7 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
AA9 9 13 0.69 0.38 0.66 Eliminated Good Reject
AA10 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
AA13 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
AA14 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
AA15 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
AA16 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
AA17 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
AA18 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
AA21 6 13 0.46 -0.08 0.32 Eliminated Poor Reject
AA24 8 13 0.62 0.23 0.54 Eliminated Fair Reject
AA25 9 13 0.69 0.38 0.66 Eliminated Good Reject
Structural
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
SA1 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SA4 2 13 0.15 -0.69 0.15 Eliminated Poor Reject
SA6 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SA7 11 12 0.92 0.83 0.92 Appropriate Excellent Accept > 0.67
SA8 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
SA11 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SA16 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SA19 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54

Musical Meanings

Structure
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
ST1 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
ST3 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
ST4 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
ST5 9 13 0.69 0.38 0.66 Eliminated Good Reject
ST6 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
ST7 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
ST9 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
ST10 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
ST12 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
ST13 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
ST14 9 13 0.69 0.38 0.66 Eliminated Good Reject
ST15 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
ST16 9 12 0.75 0.50 0.74 Needs revision Good Reject
ST17 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
ST21 8 12 0.67 0.33 0.62 Eliminated Good Reject
ST22 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
Self
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
SE1 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SE2 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE3 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE4 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SE5 7 13 0.54 0.08 0.42 Eliminated Fair Reject
SE6 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
SE7 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE8 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE9 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE10 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE11 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SE12 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE13 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
SE14 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SE15 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
SE17 8 13 0.62 0.23 0.54 Eliminated Fair Reject
SE18 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SE21 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SE22 5 12 0.42 -0.17 0.28 Eliminated Poor Reject
Source
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
SO2 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO6 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO8 9 13 0.69 0.38 0.66 Eliminated Good Reject
SO9 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO10 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO11 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
SO12 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SO13 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO14 10 12 0.83 0.67 0.83 Appropriate Excellent Reject
SO15 8 13 0.62 0.23 0.54 Eliminated Fair Reject
SO16 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO17 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SO18 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SO19 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO20 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54
SO21 12 13 0.92 0.85 0.92 Appropriate Excellent Accept > 0.54
SO22 11 13 0.85 0.69 0.84 Appropriate Excellent Accept > 0.54
SO25 10 13 0.77 0.54 0.76 Needs revision Excellent Reject
SO26 13 13 1.00 1.00 1.00 Appropriate Excellent Accept > 0.54

Qualia, Reward and Exposure

Qualia
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
QU1 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
QU3 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
QU4 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
QU5 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
QU7 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
QU8 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
Preference
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
PO1 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
PO2 8 11 0.73 0.45 0.70 Needs revision Good Reject
PO3 5 11 0.45 -0.09 0.30 Eliminated Poor Reject
PO4 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
PO5 6 11 0.55 0.09 0.41 Eliminated Fair Reject
PO6 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
PO8 6 11 0.55 0.09 0.41 Eliminated Fair Reject
PO9 3 11 0.27 -0.45 0.21 Eliminated Poor Reject
Familiar
Item Sum N CVI.I CVI.R CVI.I.adj CVIFit KappaFit CVRFit
FA1 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
FA2 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64
FA4 8 11 0.73 0.45 0.70 Needs revision Good Reject
FA5 9 11 0.82 0.64 0.81 Appropriate Excellent Reject
FA6 10 11 0.91 0.82 0.91 Appropriate Excellent Accept > 0.64
FA10 11 11 1.00 1.00 1.00 Appropriate Excellent Accept > 0.64

See item_decisions for full details.

Back to top