Scrapers that turn university course catalogues into one clean dataset
A scraping system that reads UK university course catalogues and extracts every course with its level, duration, fees, English-test scores and entry requirements into one consistent dataset, with the major university rankings collected alongside.
- Context
- Freelance project, ongoing
- Role
- Scraping engineer
- When
- Aug 2026 to now
Results
| Measure | After |
|---|---|
| UK universities with scrapers so far | About 80 |
| Fields per course | Level, duration, fees, IELTS/PTE/Duolingo, A-level/HKDSE/IB |
| Reference site | 390 courses, core fields 100% filled |
| Rankings also collected | Guardian, Times, CUG, QS |
The problem
Every university publishes courses differently: different page structures, JavaScript-rendered catalogues, bot protection, and requirements written in free text. The client needed it all comparable, field by field, across hundreds of institutions.
What I did
One standard for every site
A single best-performing university is the quality baseline: every scraper has to produce the same required fields, and a status report shows which sites are complete, partial or blocked.
Parsing messy requirements
Entry requirements and English-test scores are written as free text, so dedicated extraction and tests pull out A-level, HKDSE, IB, IELTS, PTE and Duolingo values reliably.
Hard sites handled honestly
JavaScript-heavy catalogues use a headless browser, sitemaps and hidden APIs are used where they exist, and sites that block scraping are flagged rather than delivered with unreliable data.
Ready to load
Cleaned output goes to PostgreSQL or CSV, and the whole system runs in Docker so it can be scheduled and repeated.
Stack
- Python
- Selenium (headless Chrome)
- aiohttp and requests
- BeautifulSoup and lxml
- pandas
- PostgreSQL (peewee)
- Docker
- Pytest