Show HN: 188M Hindi encoder, 28B tokens, 8K context, 1× RTX 4090https://github.com/kkkamur07/indic-modernBERT
I wanted to improve Hindi retrieval quality, particularly for longer documents ( as the current architectures don't really have a longer context length ), and was curious how far can I push a 4090 haha :) So I trained a Hindi-first ModernBERT from scratch: 188M parameters ~28.5B Hindi tokens 8,192-token context, one of the first Hindi encoders to support 8K context ( directly benefits the retrieval capacity, for which encoder models are used for ) 1× RTX 4090 (24GB) ~5 days of training After DPR fine-tuning, it achieves SOTA on the Hindi retrieval benchmarks I evaluated: mMARCO Hindi: 0.2825…