Hello, I'm Roberta Eilif from Aalto university.
I'm analyzing LinearRAG and I've noticed a few points I'd like to see highlighted.
1) LinearRAG is extremely sensitive to the provided split.
I noticed that the Contain Acc and Exactly Match (EM) results drop drastically when I use the original splits from the analyzed benchmarks. I investigated, and the split you use (and call chunks) seems quite biased. In many of them, the multihop information needed to answer the query is in the same split, which is wrong.
Why do you use a split different from the original dataset? You don't mention this approach in the paper.
2) LinearRAG's Top-k is "too good".
I noticed that you set top-k to k=1 in the retriever step. Where did this value 1 come from? How is it possible for a single chunk (pass) to have all the multihop information needed to answer the query? Is this because your splits already aggregate the necessary multihop information in a single pass?
I conducted some experiments with topk=5 and the original benchmark splits, and the results are below.
Hotpotqa
original split: EM = 34.0; F1 = 47.9: Contain_Acc=48.2
your split: EM = 44.5; F1=61.6; Contain_Acc=65.0
musique
original split: EM=16.4, F1=25.8, Contain_Acc = 20.9,
your split: EM=21.3, F1=33.9, Contain_Acc = 30.4,
2wiki:
original split: EM=34.0; F1=47.8, Contain_Acc=54.0
your split: EM=46.3; F1=61.8; Contain_Acc= 70.4
Hello, I'm Roberta Eilif from Aalto university.
I'm analyzing LinearRAG and I've noticed a few points I'd like to see highlighted.
1) LinearRAG is extremely sensitive to the provided split.
I noticed that the Contain Acc and Exactly Match (EM) results drop drastically when I use the original splits from the analyzed benchmarks. I investigated, and the split you use (and call chunks) seems quite biased. In many of them, the multihop information needed to answer the query is in the same split, which is wrong.
Why do you use a split different from the original dataset? You don't mention this approach in the paper.
2) LinearRAG's Top-k is "too good".
I noticed that you set top-k to k=1 in the retriever step. Where did this value 1 come from? How is it possible for a single chunk (pass) to have all the multihop information needed to answer the query? Is this because your splits already aggregate the necessary multihop information in a single pass?
I conducted some experiments with topk=5 and the original benchmark splits, and the results are below.
Hotpotqa
original split: EM = 34.0; F1 = 47.9: Contain_Acc=48.2
your split: EM = 44.5; F1=61.6; Contain_Acc=65.0
musique
original split: EM=16.4, F1=25.8, Contain_Acc = 20.9,
your split: EM=21.3, F1=33.9, Contain_Acc = 30.4,
2wiki:
original split: EM=34.0; F1=47.8, Contain_Acc=54.0
your split: EM=46.3; F1=61.8; Contain_Acc= 70.4