Data Synchronization

[changjo]

Hi Atom.
I have extracted the top-view and wide-view image frames and checked the first image but they are not synchronous. I have computed the homographies for wide-view and GNSS coordinates and converted them to the local pitch coordinates. To check them, I also computed the inverse homography matrix from the wide-view coordinates and applied it to the first frame of the wide-view local pitch coordinates and the first row (13:40:18.100000) of the GNSS local pitch coordinates and displayed them but they also are not matched.

Red dots are from wide-view and blue dots are from GNSS.

[AtomScott]

Hi, thanks for bringing this issue to our attention!
If possible can you share the snippet of code used so we can recreate the third image?

In our experiments we did synchronize the top-view and gnss data. There was quite a lot of error but I don't think it was as bad as the third image. I guess I can share the video, but I can't share the code just yet since it's still a mess. Also, we did this way back in January before we started annotating the videos so we infered blobs and pitch keypoints using older methods from OpenCV and not YOLOv5 etc. Apologies.

gps_overlay.mp4

• Blue circle - Detected blob for team A
• White circle - Detected blob for team B
• Blue small dot - GNSS coordinate that matched with the detected blob for team A
• White small dot - GNSS coordinate that matched with the detected blob for team B
• Green dots - Estimated pitch keypoints

I think synchronization is a good topic for the user guide so I will work on cleaning the code / making a more detailed tutorial after finishing #34 since that seems like a bigger problem!

[changjo]

Hi, thanks for the detailed explanations.
I haven't tried to project GNSS coordinates onto the top-view but wide-view. Your video shows they are synchronized well. I think I need to project them to the top-view. I also would like to know if the first row data of the first file of GNSS data matches the first frame of the wide-view and top-view video. I am attaching the snippet of the code I use. Thanks.

GNSS_FOUR_CORNERS_KEYS = ['(0,0)', '(0,68)', '(105,68)', '(105,0)']
WV_FOUR_CORNERS_KEYS = ['(0.0, 0.0)', '(0.0, 68.0)', '(105.0, 68.0)', '(105.0, 0.0)']


def get_homography(keypoints, four_corners_keys, unit='cm', inverse=False):
    original_coords = []
    pitch_coords = []
    for key in four_corners_keys:
        original_coords.append(keypoints[key])
        pitch_coords.append(list(map(lambda x: float(x) * 100 if unit == 'cm' else float(x), key.strip('()').split(','))))
        print(key, keypoints[key])

    original_coords = np.array(original_coords).reshape(-1, 1, 2)
    pitch_coords = np.array(pitch_coords).reshape(-1, 1, 2)

    if inverse:
        H, _ = cv2.findHomography(pitch_coords, original_coords)
    else:
        H, _ = cv2.findHomography(original_coords, pitch_coords)
    
    return H

# Convert GNSS cooridnates to local pitch coodinates
gnss_keypoints = load_json(GNSS_KEYPOINTS_FILENAME)
gnss_file_list = sorted(glob(os.path.join(DATA_ROOT, 'GNSS/*.csv')))

gnss_H = get_homography(gnss_keypoints, GNSS_FOUR_CORNERS_KEYS)

player_lon_lat_coords = np.array(df[['Lon', 'Lat']]).reshape(-1, 1, 2)
player_pitch_coords = cv2.perspectiveTransform(player_lon_lat_coords, gnss_H).reshape(-1, 2)
gnss_data_df['x'] = player_pitch_coords[:, 0]
gnss_data_df['y'] = player_pitch_coords[:, 1]


# Convert wide-view cooridnates to local pitch coodinates
wv_keypoints = load_json(WIDE_VIEW_KEYPOINTS_FILENAME)
wv_file_list = sorted(glob(os.path.join(DATA_ROOT, 'wide-view/*.csv')))

wv_H = get_homography(wv_keypoints, WV_FOUR_CORNERS_KEYS)
wv_H_inv = get_homography(wv_keypoints, WV_FOUR_CORNERS_KEYS, inverse=True)

df['image_x'] = df['bb_left'] + df['bb_width'] / 2
df['image_y'] = df['bb_top'] + df['bb_height']
player_image_coords = np.array(df[['image_x', 'image_y']]).reshape(-1, 2)
player_pitch_coords = cv2.perspectiveTransform(player_image_coord.reshape(-1, 1, 2), wv_H).reshape(-1, 2)
wv_data_df['x'] = player_pitch_coords[:, 0]
wv_data_df['y'] = player_pitch_coords[:, 1]


frame_num = 1
wv_coords_on_pitch = np.array(wv_data_df.loc[frame_num][['x', 'y']])
wv_coords_on_image = cv2.perspectiveTransform(wv_coords_on_pitch.reshape(-1, 1, 2), wv_H_inv).reshape(-1, 2)

first_time = '2022-01-01 13:40:18.100'
gnss_coords_on_pitch = np.array(gnss_data_df.loc[first_time][['x', 'y']])
gnss_coords_on_image = cv2.perspectiveTransform(gnss_coords_on_pitch.reshape(-1, 1, 2), wv_H_inv).reshape(-1, 2)

[changjo]

I projected the first row of the GNSS coordinates onto the first frame of the top-view. You may check it. Thanks.

• Blue dots: GNSS locations
• Green dots: keypoints.