Website Summariser Project

🌐 Let's Build Something Cool — A Website Summarizer!

Here's the plan: we'll scrape a website (pure Python, no AI involved in this part), feed that raw content into an OpenAI model, and get back a clean, concise summary. Pretty neat, right? 🎉

🛠️ Step 1 — Create the Scraper Function

Inside your level1 folder, create a new file called scraper.py and paste in the code below.

from bs4 import BeautifulSoup
import requests

# Standard headers to fetch a website
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
}

def fetch_website_contents(url):
    """
    Return the title and contents of the website at the given url;
    truncate to 2,000 characters as a sensible limit
    """
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")
    title = soup.title.string if soup.title else "No title found"
    if soup.body:
        for irrelevant in soup.body(["script", "style", "img", "input"]):
            irrelevant.decompose()
        text = soup.body.get_text(separator="\n", strip=True)
    else:
        text = ""
    return (title + "\n\n" + text)[:2_000]

def fetch_website_links(url):
    """
    Return the links on the webiste at the given url
    I realize this is inefficient as we're parsing twice! This is to keep the code in the lab simple.
    Feel free to use a class and optimize it!
    """
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")
    links = [link.get("href") for link in soup.find_all("a")]
    return [link for link in links if link]

from bs4 import BeautifulSoup
import requests

headers = {...}

"title + webpage text"

["<https://example.com>", "/about", ...]

Your folder structure should now look like this:

llm_engineering/
└── level1/
    ├── part1.ipynb
    └── scraper.py  ✅

📌 Note: This part has nothing to do with AI — it's purely about web scraping. Don't worry if it feels unfamiliar, it's not the focus of this course. Just set it up and we'll move on to the exciting AI part next!

📥 Step 2 — Import the Scraper File

▶️ Create a new code cell and run it:

from scraper import fetch_website_contents

🎭 Step 3 — Set Up the System & User Prompts

Let's try something a little different this time — a snarky, humorous summarizer! 😄

▶️ Create a new code cell and run it:

# Define our system prompt - you can experiment with this later!
# Try changing the last sentence to 'Respond in markdown in Spanish.' for fun 🌍

system_prompt = """
You are a snarky assistant that analyzes the contents of a website,
and provides a short, snarky, humorous summary, ignoring text that might be navigation related.
Respond in markdown. Do not wrap the markdown in a code block - respond just with the markdown.
"""

# Define our user prompt prefix
user_prompt_prefix = """
Here are the contents of a website.
Provide a short summary of this website.
If it includes news or announcements, then summarize these too.

"""

🧩 Step 4 — Build the Messages Template

▶️ Create a new code cell and run it:

def messages_for(website):
    return [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt_prefix + website}
    ]

Here, website params is raw website scraped data

🤖 Step 5 — Call the LLM

Now for the magic — let's send the scraped content to OpenAI and get back a summary!

▶️ Create a new code cell and run it:

# Calling the OpenAI API — you'll be doing this a lot from here on! 🚀

def summarize(url):
    website = fetch_website_contents(url) # fetching the raw scraped data
    response = openai.chat.completions.create(
        model = "gpt-4.1-mini",
        messages = messages_for(website) 
        # passing scaraped data along with the messages
    )
    return response.choices[0].message.content

Final Step — See It in Action!

▶️ Create a new code cell and run it:

summarize("<https://codeveda.dev>")

If everything is set up correctly, you should see a witty, snarky summary of the website printed below the cell.

👀 View the Raw Scraped Data to compare with LLM version

Curious about what the scraper actually pulls from the website before the AI processes it? Run this to peek behind the curtain! 🔍

▶️ Create a new code cell and run it:

scrapedData = fetch_website_contents("<https://codeveda.dev>")
print(scrapedData)

This prints the raw scraped content exactly as it's sent to the OpenAI model — no AI magic applied yet! 🕵️‍♂️

🎨 Bonus — Display the Summary in a Nicer Format

Why settle for plain text when you can render it beautifully? Let's display the summary with proper Markdown formatting! ✨

▶️ Create a new code cell and run it:

# Renders the summary as nicely formatted Markdown in the notebook output

def display_summary(url):
    summary = summarize(url)
    display(Markdown(summary))
    
display_summary("<https://codeveda.dev>")

Instead of raw text, you'll now see the summary rendered with headers, bold text, and proper formatting — much easier on the eyes! 👀🎉

🎓 That's a Wrap on Level 1 — Part 1!

And that's it for level1/part1.ipynb! 🎉 Hope you're enjoying the journey so far — because trust me, the best is yet to come! 🚀

Buckle up, because things are about to get even more exciting from here! 😄🔥