(Accelerated Model Architectures)

XMA is a repository comprising of fast kernels for model training.
We are planning on adding lots of experimental and fun model architectures with support for multiple accelerators like NVIDIA, AMD GPUs, Google TPUs and Amazon Trainiums.

Documentation: https://open-lm-engine.github.io/accelerated-model-architectures/

layers

functional	CUDA	MPS	Pallas	NKI	ROCm	Triton
GRU	❌	❌	❌	❌	❌	✅
LinearAttention	❌	❌	❌	❌	❌	✅
M2RNN	❌	❌	❌	❌	❌	✅
MoE	✅	❌	❌	❌	❌	✅
RNN	❌	❌	❌	❌	❌	✅

functional

functional	CUDA	MPS	Pallas	NKI	ROCm	Triton
continuous_count	✅	❌	❌	❌	❌	❌
cross_entropy	❌	❌	❌	❌	❌	✅
fused_linear_cross_entropy	❌	❌	❌	❌	❌	✅
fused_residual_add_rmsnorm	❌	❌	❌	❌	❌	✅
gru	❌	❌	❌	❌	❌	✅
linear_attention	❌	❌	❌	❌	❌	✅
m2rnn	❌	❌	❌	❌	❌	✅
p_norm	❌	❌	❌	❌	❌	✅
pack_sequence	✅	❌	❌	❌	❌	✅
rmsnorm	❌	❌	❌	❌	❌	✅
rnn	❌	❌	❌	❌	❌	✅
softmax	❌	❌	❌	❌	❌	✅
swiglu	✅	✅	❌	✅	❌	✅
swiglu_packed	✅	❌	❌	✅	❌	✅
unpack_sequence	✅	❌	❌	❌	❌	✅

Discord Server

Join the discord server if you are interested in LLM architecture or distributed training/inference research.