Discover MIT Courses with AI and GitHub Copilot

What you’ll get

A way to explore a huge course catalog by turning it into data and letting an AI recommend courses you would never stumble on by clicking through pages. You “scrape” the catalog (pull its content into structured data) with a single paste into your browser, check the data is small enough for an AI to read, then ask the AI to surface courses that match your interests. The example uses MIT’s catalog, but the same trick works on any large list. It is intermediate: you paste one snippet into the browser console.

Tools you need

• A web browser (free): you run the scraper in its built-in developer console.
• OpenAI Tokenizer (free): counts the “tokens” (the chunks AI measures text in) so you know the data fits an AI’s limit.
• A long-context AI chat (freemium): ChatGPT, Gemini, or Claude, to read the catalog and recommend courses.
• GitHub Copilot (freemium, optional): if you would rather write a small script to filter the data.

Steps

1. Scrape the catalog: open the MIT Course Picker page, open your browser’s developer console (press F12, then the Console tab), paste this in, and run it. It pulls each course into a tidy list and copies it to your clipboard as JSON (a structured text format):

   const courses = [...document.querySelectorAll(".course-name")]
     .map((e) => e.closest(".course-lens"))
     .map((d) => ({
       title: d.querySelector(".course-name")?.textContent,
       description: d.querySelector(".course-description")?.textContent,
       semester: d.querySelector(".course-semester")?.textContent,
       instructor: d.querySelector(".course-instructor")?.textContent,
     }));
   copy(JSON.stringify(courses));

   You should see no error, and your clipboard now holds the whole catalog as JSON. (copy() is a console helper that puts its argument on your clipboard.)

2. Check the size: go to the OpenAI Tokenizer site and paste the JSON in. It shows a token count (the author’s catalog was around 343,000 tokens). This tells you whether it fits the AI you plan to use.

3. Make sure it fits your AI: modern models accept very large inputs (some over a million tokens), so a few hundred thousand is usually fine. If it is too big, scrape fewer fields or one department at a time.

4. Ask the AI for recommendations: paste the JSON into a long-context chat AI and ask it to play matchmaker:

   Here is a JSON list of courses: [paste the catalog].
   I am interested in [your topics, e.g. AI ethics and complex systems] and want courses
   that might reshape how I think, not just the obvious ones. Recommend 10, with a sentence
   each on why, and note any prerequisites.

   You should get a tailored shortlist you would never have found by browsing.

5. Optional, filter with code: if you prefer, have GitHub Copilot help you write a short script to pre-filter the JSON by keyword or department before handing it to the AI.

Original source

Based on a Hacker News post by low_tech_punk, who “vibe-coded” the MIT course catalog into data and used AI to explore it, getting past the limits of the normal browsing interface.

Notes & variations

• Do you even need to scrape? If the catalog offers a download or export, use that instead of scraping. Scraping is for sites that do not.
• Common pitfall: pasting enormous text into a chat box can hit interface limits. If it chokes, trim the data (fewer fields, one department) or use the model’s API.
• Tip for better results: pre-filter by department or keyword, and put your specific interests in the prompt, to get sharper recommendations.