Overview

In this assignment, you will implement the decoding strategies that turn model scores into text, write prompts that return a fixed schema, retrieve documents the model was not trained on, and build a three-layer conversation memory.

The assignment includes four code files and five written responses, totaling 100 points.

• Part 1: Decoding (15 pts) — part1_decoding.py
• Part 2: Prompting and structured output (20 pts) — part2_prompting.py
• Part 3: Retrieval (20 pts) — part3_retrieval.py
• Part 4: Memory (20 pts) — part4_memory.py
• Short answer questions (25 pts) — separate PDF

Setup

You need Python 3.10 or newer.

pip install transformers torch google-genai fastembed numpy

Get an API key from Google AI Studio and set GEMINI_API_KEY in your environment. The free tier limits requests per minute and per day, so please budget multiple days to finish this homework. Complete the parts in order.

All functions you have to implement yourself are marked with # TODO. Each file also contains a test function to help you check your work. Parts 1 and 3 download a model the first time they run.

Part 1 — Decoding (15 pts)

part1_decoding.py

• 1.0: Load GPT-2 and return the scores for the next token.
• 1.1–1.4: Implement greedy decoding, temperature, top-k decoding, and top-p decoding.
• 1.5 Repetition penalty: Lower the score of every token already produced.
• 1.6 Generate: Compose the settings into a sampler and run it in a loop.
• 1.7 Evaluate: Report the share of n-grams that are distinct.

Run the file to print the next-token distribution and one generation per setting.

Part 2 — Prompting and Structured Output (20 pts)

part2_prompting.py

• 2.0 Client: Write a Gemini wrapper with retries and a disk cache.
• 2.1 Extract: Get the schema out of a ticket with no examples in the prompt.
• 2.2 Few-shot: Do the same with worked examples in the prompt.
• 2.3 Chain-of-thought: Have the model reason before it answers.
• 2.4 Self-consistency: Sample several reasoned answers and take the majority.
• 2.5 Validate: Check each extraction against the schema.

Run the file to score all four prompts on schema validity and per-field accuracy.

Part 3 — Retrieval (20 pts)

part3_retrieval.py

• 3.0 Chunk: Split thirty articles into overlapping windows.
• 3.1 Embed: Turn each chunk into a vector.
• 3.2 Search: Rank chunks by cosine similarity.
• 3.3 Baseline: Rank them again by shared words.
• 3.4 Measure: Report recall@k for both retrievers.
• 3.5 Answer: Answer from what you retrieved, cite it, and abstain when the corpus does not cover the question.

Run the file to compare the two retrievers and print both abstention rates.

Part 4 — Memory (20 pts)

part4_memory.py

• 4.0 Buffer: Keep the last n turns in a conversation and report what is removed.
• 4.1 Summary: Compact everything the buffer drops into a running summary.
• 4.2 Fact store: Pull durable facts out of each turn and search them with your Part 3 retriever.
• 4.3 Compose: Combine 4.0–4.2 to build a full memory context.
• 4.4 Replay: Replay a scripted conversation into your memory.

Run the file to see the context your memory builds at three buffer sizes.