Error using mne.read_evoked_fieldtrip

[Federica-2]

I have a .mat file with averages for different conditions that were computed in Fieldtrip with FT_TIMELOCKANALYSIS. I am trying to use mne.read_evoked_fieldtrip, but when I run this code I get an error: KeyError: 'data'

#Sampling frequency
sfreq = 1000 
#Definition of channel types and names.
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eog','eog']
ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']
#Creation of info object
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)

file_path = r"D:/EEGdata/toFIELDTRIP";
filename = file_path + '/averages_for_ft.mat'

myepochs = mne.read_evoked_fieldtrip(filename, info)

I have also tried to specify the 'data_name', selecting only one of the conditions included in the 'averages_for_ft.mat', but again, I get an error: TypeError: list indices must be integers or slices, not str

myepochs = mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')

I am using mne 0.20.dev0, and I am happy to send the 'averages_for_ft.mat' privately, if needed. Thanks.

[agramfort]

maybe @thht or @DiGyt can have a look? can you share a file to replicate the pb?

[Federica-2]

Sure, but I would prefer to share it privately. Where can I send it to? It's about 465000KB.

[Federica-2]

Hi there, where could I send you the file? Thanks.

[thht]

hi,
i am going to take a look.

@dede3de send me an email to thomas.hartmann@th-ht.de and i will provide you with a link to upload the file.

[thht]

so, there are two/three problems here:

1. your file contains a cell array for each of the conditions. i assume each entry refers to the data of one participant. cell array are not supported for reader function at the moment. the solution for your would be to build one file for each participant. i will work on a better error message.
2. the channels in the data are lowercase while the channels in your info structure are uppercase. this needs to match.
3. there is one extra bug when dealing with these missing channels and evoked data.

i think, 1 and 2 have you covered to read your data. i will open a PR for the bugs you found.

[Federica-2]

Thanks very much for looking at the file.
Re (1), Each cell array is a condition, and within each condition I have a structure for each of the participants. In each structure for each participant, there are:

• label of channels (cell type)
• frequency sample
• electrode locations, with coordinates and labels (structure, with double and cell type)
• average for each channel (rows) and each time point (column) (single type) E.g., 66 channels x 900 timepoints
• variance for each channel (rows) and each time point (column) (single type)
• time points (double type)
• cfg
• dimord

Your suggestion is to create one file for each participant. Which information would I need to include? Only the average for each channel and time point (i.e., a file with 66x900 for each participant)?

Re (2), I am using this code, which works fine:

#Sampling frequency
sfreq = 1000 
#Definition of channel types and names.
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eog','eog']
ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']
#Creation of info object
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)
epochs.info = info
montage =  mne.channels.make_standard_montage('standard_1020')
epochs.set_montage(montage, match_case=False)#https://github.com/mne-tools/mne-python/issues/7169

Re (3) Not sure which ones are the missing channels and evoked data.

Thank you.

[thht]

so, in your original file you have the following structure:

• condition_1
• condition_2

etc... (i use generic names here for your conditions). each of these fields is a cell-array containing one fieldtrip-structure per participant.

what you need to do is create a file for each participant. the general structure would be the same. i.e. there would be one variable per condition. but instead of containing a cell array with the data of all participants, the variable should only contain the fieldtrip-structure of one participant.

here is a small snippet i used to extract the data of the first participant:

%% init
restoredefaultpath
clear all global

%% set variables
f_name = '/home/th/owncloud/sbg/upload/mne_issue_7293/averages_for_ft.mat';

%% load data
data = load(f_name);

%% convert fields
all_fields = fieldnames(data);

for idx_field = 1:length(all_fields)
  cur_field = all_fields{idx_field};
  data.(cur_field) = data.(cur_field){1};
end %for

%% save new data
save('new_data.mat', '-struct', 'data');

the only thing you would need to do is to do this for all participants, i.e. add another loop around it.

concerning the channel names: they are in lower case in your mat files. they are in uppercase in the MNE info structure. they need to match (case sensitive) in order for the import to work.

[Federica-2]

Hi there, I am still having some issues with this, and I was hoping you could suggest a way forward.
I have checked that I have the latest version of mnedev with:

git checkout master
git fetch upstream
git merge upstream/master

I am still trying to use mne.read_evoked_fieldtrip to read a .mat file with averages for different conditions that were computed in Fieldtrip with FT_TIMELOCKANALYSIS.
As suggested by @thht , I have created a .mat file for each participant, with each variable being a condition. However, when I try to run the following code, an error appears:

import mne
print(mne.__version__)
import numpy as np

#SET UP INFO AND MONTAGE
#Sampling frequency
sfreq = 1000
#Definition of channel types and names.
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eog','eog']
ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']
#Create info object
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)

#IMPORT DATA
file_path = r"D:/EEGdata/EVOKED";
filename = file_path + '/ppt01_toMNE.mat'

evoked= mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')

Error:

evoked= mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')
<ipython-input-5-aa6c3580f59d>:1: RuntimeWarning: The following channels are present in the FieldTrip data but cannot be found in the provided info: {'po8', 'o2', 'ft7', 'f6', 'f8', 'f4', 'p3', 'c3', 'po7', 'cp4', 'af4', 't7', 'f7', 'oz', 'o1', 'fc4', 'c6', 'p5', 'tp7', 'p6', 'cp1', 'fcz', 'fp2', 'tp8', 'af8', 'fp1', 'cp3', 'fpz', 'f3', 'tp9', 'cpz', 'poz', 'cp2', 'c4', 'cz', 'fc3', 'po3', 'po4', 'fz', 't8', 'cp6', 'c5', 'p2', 'fc6', 'af3', 'pz', 'f1', 'ft10', 'p1', 'fc2', 'fc1', 'fc5', 'tp10', 'p4', 'f2', 'c1', 'c2', 'ft8', 'p8', 'af7', 'p7', 'ft9', 'f5', 'cp5'}.
These channels will be removed from the resulting data!
  evoked= mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')
Traceback (most recent call last):

  File "<ipython-input-5-aa6c3580f59d>", line 1, in <module>
    evoked= mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')

  File "c:\users\f\documents\mnedev_latest\mne-python\mne\io\fieldtrip\fieldtrip.py", line 166, in read_evoked_fieldtrip
    info = _create_info(ft_struct, info)  # create info structure

  File "c:\users\f\documents\mnedev_latest\mne-python\mne\io\fieldtrip\utils.py", line 53, in _create_info
    if ft_struct['trial'].ndim == 2:

KeyError: 'trial'

[drammock]

Quoting @thht from here:

2. the channels in the data are lowercase while the channels in your info structure are uppercase. this needs to match.

[thht]

exactly, as this warning tells you:

RuntimeWarning: The following channels are present in the FieldTrip data but cannot be found in the provided info: {'po8', 'o2', 'ft7', 'f6', 'f8', 'f4', 'p3', 'c3', 'po7', 'cp4', 'af4', 't7', 'f7', 'oz', 'o1', 'fc4', 'c6', 'p5', 'tp7', 'p6', 'cp1', 'fcz', 'fp2', 'tp8', 'af8', 'fp1', 'cp3', 'fpz', 'f3', 'tp9', 'cpz', 'poz', 'cp2', 'c4', 'cz', 'fc3', 'po3', 'po4', 'fz', 't8', 'cp6', 'c5', 'p2', 'fc6', 'af3', 'pz', 'f1', 'ft10', 'p1', 'fc2', 'fc1', 'fc5', 'tp10', 'p4', 'f2', 'c1', 'c2', 'ft8', 'p8', 'af7', 'p7', 'ft9', 'f5', 'cp5'}.
These channels will be removed from the resulting data!

you end up with a data structure without any data which leads to this error.

[Federica-2]

I thought this was being fixed by #7304, but I might have misunderstood then.

[thht]

#7304 introduced an error messages when the function found cell arrays and another issue when not all channels that are present in the data are also in the info object.

it is still the case that a channel needs to be present in both the data loaded from the mat file as well as the info field for its data to be loaded. and the check is case sensitive, so if channel names are in lowercase in the info and uppercase in the data (or vice versa), they will not match.

so, one thing you need to do is to either make the channel names in this line all lowercase:

ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']

or make all channel names upercase in the fieldtrip structures you save in matlab.

on another note: you do not seem to use the latest version of mne. i do not know what went wrong, but this line in your error message:

  File "c:\users\f\documents\mnedev_latest\mne-python\mne\io\fieldtrip\utils.py", line 53, in _create_info
    if ft_struct['trial'].ndim == 2:

points to line number 53. in the latest version, this line of code is in line 61...

you might want to try again to update your mne repository. pay attention to any error or warning messages you might get. another option is always to delete the repository and just clone it again, provided you do not have any uncommitted changes on your local drive....

[Federica-2]

Thanks @thht, git config --global url."https://".insteadOf git:// did the trick.