[FEATURE] Pose Classification

[charlieforward9]

Full Implementation of Pose Classification from model embedding to user interfaces implemented exclusively through the Flutter client.

• Trial-based assessments (driven by classification)
• Classification breakdown in Analysis & PDF

[DrBubbles42]

The normalization function. We will need to implement the scale and the translation data on the incoming data from the keypoint model before it is passed to the balance model. Below are the stats that are used:

means of max x,y: 154.18105273776584 226.64141575183743
means of min x,y: 94.35607003542333 32.165283149850914
maxes x,y: 257.3079833984375 274.50079345703125
mins x,y: -4.650461196899414 -2.4850502014160156
Center x,y: [191.0224458 182.39294565]
dist x,y: [24.05671016 1.48454173]
dist absolute x,y: 24.126362310231567
centers (L_xy, Center_xy, R_xy): [[166.965735639622, 180.90840392019234], [191.02244580181596, 182.39294565113542], [215.0791559640099, 183.8774873820785]]
SDs (xmax, ymax, xmin, ymin, hipdist) : [15.63874639965528, 15.063585556528396, 16.839463809071564, 12.842026919066111, 14.672498078400517]
z: 1.9599639845400536

def normalize(data, y):
    xmax_mean, ymax_mean, xmin_mean, ymin_mean, center, xydist, dist, centers, SDs = get_stats(data)
    
    xnorm = []
    xnorm_y = []
    bads = []
    bads_y = []
    tmp = []
    
    index = 0
    for i in data:
        is_bad = False
        
        #find the center
        tmp = []
        icenter = [ (i[23][0] + i[24][0]) / 2, (i[23][1] + i[24][1]) / 2]
        idif = [(center[0] - icenter[0]), (center[1] - icenter[1])]
        
        #find max
        d = np.max(i, axis=0)
        xmax = d[0]
        ymax = d[1]
        d = np.min(i, axis=0)
        xmin = d[0]
        ymin = d[1]
        idistx = np.linalg.norm(i[23][0]-i[24][0])
        
        #check for bad data
        # if this below is_bad == false than the data point is bad and does not need to be presented to the balance model (Failure)
        # SD [xmax, ymax, xmin, ymin, hipdist]
        if xmax > xmax_mean + (z * SDs[0]) or xmax < xmax_mean - (z * SDs[0]):
            is_bad = True
            print("is bad xmax:", xmax_mean + (z * SDs[0]), xmax_mean - (z * SDs[0]))
            print(xmax)
        if ymax > ymax_mean + (z * SDs[1]) or ymax < ymax_mean - (z * SDs[1]):
            is_bad = True
            print("is bad ymax:", ymax_mean + (z * SDs[1]), ymax_mean - (z * SDs[1]))
            print(ymax)
        if xmin > xmin_mean + (z * SDs[2]) or xmin < xmin_mean - (z * SDs[2]):
            is_bad = True
            print("is bad xmin:", xmin_mean + (z * SDs[2]), xmin_mean - (z * SDs[2]))
            print(xmin)
        if ymin > ymin_mean + (z * SDs[3]) or ymin < ymin_mean - (z * SDs[3]):
            is_bad = True
            print("is bad ymin:", ymin_mean + (z * SDs[3]), ymin_mean - (z * SDs[3]))
            print(ymin)
        if idistx > xydist[0] + (z * SDs[4]) or idistx < xydist[0] - (SDs[4]):
            is_bad = True
            print("is bad hip dist: ",idistx, ">", xydist[0] + (z * SDs[4]),"::",idistx, "<", xydist[0] - (SDs[4]))
            print(xydist[0])
        
        ###### TODO scale data i is just the current data point being worked on
        scalex = xydist[0]*2/np.sqrt(((i[23][0]) - (i[24][0]))**2)
        scaley = ymax_mean / ymax
        
        ###### TODO translate the data 
        icenter = [ (scalex*i[23][0] + scalex*i[24][0]) / 2, (scaley*i[23][1] + scaley*i[24][1]) / 2]
        idif = [(center[0] - icenter[0]), (center[1] - icenter[1])]
        
        for j in range(0, len(i)):
            tmp.append([i[j][0]*scalex + idif[0], i[j][1]*scaley + idif[1], i[j][2], i[j][3], i[j][4]])

        if not is_bad:
            xnorm.append(np.array(tmp))
            xnorm_y.append(y[index])
            
        else:
            bads.append(np.array(tmp))
            bads_y.append(y[index])
            
        index += 1
    
    return xnorm, xnorm_y, bads, bads_y, centers

[DrBubbles42]

@charlieforward9 @Timothyrichpt I have fixed the data issue but the model does not work with this data and needs some tweaking. it is clear that the arms are crossed here. For now we can use the old model and it will do a very good job telling you if the arms are at the subjects side (1) or not (0). I have changed the model to output 0 as static and 1 as sway/fail. one thing i do notice is the head cluster looks very similar in both classes.

[charlieforward9]

@DrBubbles42 Should we explore Principal Component Analysis as a next step?

If I understand it correctly, you're already doing this by omitting the [3:5] indexes in the detection output. Would there be value in omitting the head (or part of it)

[DrBubbles42]

here is a quick comparison of the updated statistics which seem to be in the ball park of our previous data but with quite a bit a variance.

Old stats:
means of max x,y: 154.18105273776584 226.64141575183743
means of min x,y: 94.35607003542333 32.165283149850914
maxes x,y: 257.3079833984375 274.50079345703125
mins x,y: -4.650461196899414 -2.4850502014160156
Center x,y: [191.0224458 182.39294565]
dist x,y: [24.05671016 1.48454173]
dist absolute x,y: 24.126362310231567
centers (L_xy, Center_xy, R_xy): [[166.965735639622, 180.90840392019234], [191.02244580181596, 182.39294565113542], [215.0791559640099, 183.8774873820785]]
SDs (xmax, ymax, xmin, ymin, hipdist) : [15.63874639965528, 15.063585556528396, 16.839463809071564, 12.842026919066111, 14.672498078400517]
z: 1.9599639845400536

New stats:
means of max x,y: 146.11480420224242 213.86082010967382
means of min x,y: 106.89222010704887 56.81809859838155
maxes x,y: 182.31581115722656 238.57589721679688
mins x,y: 46.343658447265625 35.23603820800781
Center: [196.30995334 198.52594613]
dist x,y: [21.4193202 0.60478755]
dist: 21.432612077519533
centers: [[174.89063313779786, 197.92115858112288], [196.3099533425565, 198.52594613255457], [217.72927354731513, 199.13073368398625]]
SDs: [4.570367337482556, 10.555995191014471, 6.184699478898564, 9.23750454773484, 15.914418021459383]
z: 1.7506860712521695

[charlieforward9]

@Timothyrichpt What have you seen from the Classification Model color shading so far? Is it resembling a prediction of the patients pose?

[DrBubbles42]

@charlieforward9 for the updated balance_nn_82.12.tflite file here are the stats used in the normalization

means of max x,y: 148.9316624640058 218.3240010129572
means of min x,y: 102.51440336306322 48.208955628623634
maxes x,y: 257.3079833984375 274.50079345703125
mins x,y: -4.650461196899414 -2.4850502014160156
Center: [194.46347425 192.89205381]
dist x,y: [22.34033741 0.91201126]
dist: 22.373311160212694
centers: [[172.12313683600976, 191.98004254546555], [194.46347424701685, 192.89205381036996], [216.80381165802393, 193.80406507527437]]
SDs: [10.666145181421614, 13.742566405833044, 12.633746248003606, 15.850760792503975, 15.914418021459383]
z: 1.9599639845400536

after this is normalized I then scale each data point to be between 0 and 1 with 0 being the minimum of the data and 1 being the maximum of the data. this squeezed about 1% accuracy improvement out of the model. The following are the mins and maxes used for the normalization:

y values used for normalization min: 9.50367922e+01 max: 306.92673439
x values used for normalization min: 8.75818536e+01 max: 347.50192423