Husna Search.
TF-IDF search over ~5,000 Arabic articles, built with a custom preprocessing pipeline and a hand-rolled tokenizer.
Overview
~5,000 Arabic articles, one publisher
The dataset is 6,366 scraped rows from a single publisher; after dropping empty-content rows, 4,998 articles actually go into the index — mostly news (5,870 rows across the full scrape), with a smaller mix of blog posts and video entries.
Four steps before anything gets indexed
Every document — and every query, at search time — goes through the same pipeline before touching the index.
- Stopword removal
- NLTK Arabic list + a large custom list
- Cleaning
- maha: harakat, tatweel, links, emojis, punctuation
- Normalization
- maha normalize + custom lam-alef handling
- Light stemming
- Tashaphyne ArabicLightStemmer
A hand-rolled tokenizer feeding scikit-learn's TF-IDF
A custom word-level tokenizer builds a frequency-filtered vocabulary (min frequency 5, 313 tokens survive) with PAD/OOV tokens, which is then passed as the fixed vocabulary to scikit-learn's TfidfVectorizer. A query is preprocessed identically, vectorized, and scored against the full matrix with linear_kernel — equivalent to cosine similarity since TF-IDF vectors are L2-normalized — returning the top 5 matches with search latency shown in the UI.
Arabizi transliteration, adapted from DSAraby
The repo also bundles a small transliteration helper that maps Latin-script Arabic ("Arabizi", e.g. chokran likom kamliin) back to Arabic script — adapted from the DSAraby project rather than written from scratch.
Method
A one-time corpus preprocessing + indexing pass builds a fixed-vocabulary TF-IDF matrix; each query runs through the same preprocessing before being scored against it.
- 01Corpus
6,366 scraped articles, 4,998 usable
- 02Preprocess + tokenize
clean, normalize, stem, 313-token vocab
- 03TF-IDF + cosine sim.
top-5 ranked results
Stack
Reproduce it
$ git clone https://github.com/baselhusam/Search-Engine.git$ pip install -r requirements.txt$ streamlit run search_engine.py