Chapter 3 · Section 07 · On the clock

Test it,know its limits

Three real jobs · one impossible question · the limits that remain

Before you begin

Two chats,
two jobs

You built the advisor in one chat and started it in another, and those two keep their jobs for the whole test. The build chat — the one from the advisor guide's steps 1 to 3 — edits files and keeps the log. The advisor chat — the one you started in step 4 — answers questions and does nothing else. You carry the verdicts between them; that judgment is the exercise.

Check

Twelve minutes on the clock. The countdown on the screen at the front is the clock every milestone below refers to. Protect the last third — the rerun is the point.

Pace yourself against the clock

Time left on the room timer — not the time of day.

9:00 left You've tried all three jobs and graded the answers.
7:00 left You've asked the impossible question.
5:00 left You've told the build chat what went wrong.
3:00 left advisor-start.md is fixed and test-log.md has its first entry.
1:00 left You've restarted the advisor chat and asked the same question.
0:00 You've compared the two answers. What changed goes to the build chat — now, or after class. The point

If something's wrong

I closed the build chat, or it's lost the thread

Start a new chat in mba-advisor and treat it as the build chat. It can read every file you made, which is the whole reason the rules live in a file and not in a conversation.

I only have one chat open

Then it's the advisor chat. Start a second one for the build work when step 2 asks for it — don't ask the advisor to edit its own rules mid-conversation. It read them once, at the start, and a rule changed under it is a test you can't read.

Step 1 of 2

Now find out
what you actually built

In the advisor chat. Three real jobs, one impossible question. Different jobs stress different rules — and the impossible one is the most informative of the four. The questions below are examples: ask what you'd actually want to know.

One question from each row — examples; use your own words

Plan "Build a two-week onboarding plan around my goals and constraints."
Choose "Which three activities should I prioritize — and what should I drop?"
Navigate "I have a scheduling conflict between something required and something I've already committed to. What do the materials say, and whom do I contact?"

Grade every answer against the four:

Direct answer Personal fit Source trail Uncertainty
"Failure" is wider than the four. Anything that isn't what you want from your advisor counts — too long, too hedged, not encouraging enough, the wrong tone. If you'd want it different next time, it's a failure worth a rule.

The impossible question

"What's the deadline for the spring add/drop exception, and who signed off on that date?"

You're hoping for "I can't verify that from the supplied materials." That's a pass, not a failure. A good refusal names what's missing, points you to the right office or person, and does not invent.

Two more to try: when Persuasive Presentation II is due in 15.280 · how many points you'd need to bid to get your preferred Negotiations section.

Done when

If something's wrong

It answered the impossible question confidently

You've just produced the session's central exhibit: fluent, specific, plausible, with nothing underneath it. That's your failure for step 2 — the uncertainty rule. Then tell the room.

Every answer is good. Am I doing it wrong?

Push harder. Ask about something genuinely contested, or about a deadline that isn't in the packet, or ask it to choose between two things it has no basis to compare. An advisor that never says "I don't know" hasn't been asked anything hard enough.

I'm out of time

Do the impossible question. If you only run one test today, run that one — it's the one that tells you whether anything else it said can be trusted.

Step 2 of 2

Revise the rule,
not just the answer

A failure is information about your rules. Diagnose the pattern, then patch the pattern — and let the chats keep their jobs: the build chat edits and logs, the advisor chat answers.

Match the failure to the rule

Too generic Strengthen the profile requirement.
No source trail Strengthen the citation rule.
Overconfident Strengthen the uncertainty rule.
All summary Require a decision and a next action.
Not your style Add a rule about length, tone, or encouragement.

Then, in this order

  1. Tell the build chat what went wrong One line: the question, and what the answer missed. It fixes the matching rule in advisor-start.md and starts test-log.md — it knows exactly what it changed, so it writes the entry.
  2. Restart the advisor chat The running advisor read the rules when it started and won't see the edit. Close it, start a new chat in mba-advisor, and paste the restart prompt below — the start prompt with your question on the end, so it's one paste and no introduction. Grade the answer against the four.
  3. Close the loop in the build chat One line — what changed — and it finishes the entry. If the clock's gone, do this after class: the entry already holds the rule change, which is the part that matters.
Diagnose and revise · build chat

I tested the advisor. The question: [paste it]. What went wrong with the answer: [it summarised instead of deciding / it ignored my profile / it cited nothing / it sounded sure about something the materials don't cover / it was too long / the tone isn't what I want]. Strengthen (or add) the matching rule in advisor-start.md so this doesn't happen again — change nothing else, and show me the new wording. Then create test-log.md with one entry: the question, what the answer missed, and the rule you changed. Leave a line for what happens when I ask again.

Restart the advisor · advisor chat

Act as my personal Sloan MBA advisor for this session. First read source-map.md, my-profile.md, and the rules in advisor-start.md. Use the files in mba-materials as the source of truth for factual claims about Sloan. If any of those three files is missing, tell me which one before you start. Help me make decisions, not just find passages. For every recommendation: answer directly, explain why it fits my profile, cite the source file and section, and label judgment and uncertainty. If the materials do not support an answer, say so. Skip the introduction. Answer this question: [paste the same question, word for word]

Close the loop · build chat

I asked the advisor the same question in a new chat. This time: [better — it cited two sources and gave a decision / the same failure / worse]. Finish the entry in test-log.md with that.

Done when

If something's wrong

The rerun is no better

Also information. Ask the build chat to show you the rule it wrote — a rule that's too vague to act on is the usual cause. Sharpen it once more; if there's no clock left, log "no change" and say why. A logged failure is worth more than an unlogged fix.

I asked the advisor to fix its own rules

It can, but you've lost the test: an advisor that edits its rules mid-conversation is answering with the old ones in its head. Keep the roles — build chat edits, advisor chat answers — and restart the advisor after any edit.

I'm out of time

Do the diagnose-and-revise prompt and stop there. One entry in test-log.md with the rule you changed is enough to tick it; the rerun can happen after class.

It works. And you know its limits.

You built an agent that works across a folder of evidence, grounded it in official sources, taught it your priorities, wrote its rules down where you can inspect them, and then went looking for where it breaks. The last part is the part most people skip.

Bring both folders back next session. They're the starting point for publishing, connecting, and making this reusable.

Two questions left open How does the page become visible to recruiters?
How does the advisor become reusable, connected, and current?

Build quickly. Inspect carefully.
Increase agency deliberately.

All guides