AI Docs Generator
I swapped two parameters and asked whether the documentation would notice. It did.
- My role
- Solo · full-stack + research design & evaluation
- Type
- Undergraduate thesis
- Scope
- Generation tool · GPT-4 integration · Likert evaluation
- Stack
- GPT-4 · Next.js · Flask · PostgreSQL · GitHub API
A full-stack research product (my undergraduate thesis, published) that reads source from GitHub and generates structured documentation with GPT-4, returned as reviewable Markdown. Twelve developers shaped the requirements before I wrote any code; five rated the result on a Likert scale and every rating came back positive. I had no way to know whether they were right, so afterwards I wrote one.
The question was 'useful,' not 'more.'
AI can clearly generate text about code. The thesis asked something narrower and harder: could it produce a draft developers found useful enough to actually start from?
Full-stack, end to end.
A Next.js dashboard and file workspace, a Flask REST API wiring the GitHub and OpenAI APIs, and PostgreSQL behind JWT auth — built solo. Generation is stateless: nothing on that path is persisted, and the only table in the database is users. Your GitHub token lives in the browser session and never reaches it.
Twelve shaped it, five judged it.
The requirements came from twelve developers before I wrote code — most of them past five years' experience. Five then rated the output on completeness, readability, relevance, and how much it helped with documentation time. Twenty ratings, every one of them positive, not a single negative.
The result is about my instrument.
Twenty out of twenty is not a finding about a tool. My respondents answered under their own names, on a scale with no neutral option, and nobody ever documented a file by hand to compare against. I set out to ask whether the output was accurate and built something that measured whether people liked it. So I stopped asking people.
So I measured the code instead.
A harness that parses each file with Python's AST and checks every name the documentation asserts against what the source actually contains. Across nineteen files of this app's own backend it found all seventy-one public definitions documented and nothing invented: it flagged four names, and all four turned out to be my harness's fault rather than the model's. That is not the output the five developers rated — different model, different code, a year later — so it settles nothing about their verdict. It is simply the first number in this project that can be checked at all.
The harness was wrong before the model was.
My first run accused the model of inventing 58 names and omitting four in ten public functions. Both were my bugs. It read "no `__all__` declarations are defined" as a claim that `__all__` exists, scored a Markdown table header as a dependency, and couldn't see a function documented as `ErrorHandler.handle_bad_request(self, error)` because it only looked for bare names. Every headline I nearly published was an artifact of my own parser — which is the same failure the study had, wearing a lab coat.
Then I broke the code to see if the docs noticed.
Grounding couldn't fail — the whole file is in the prompt, so of course the model doesn't invent identifiers. The failure that matters is subtler: does it read the code, or describe what the function is called? So I swapped two parameters in a signature and left the name, docstring and body untouched. `__init__(blueprint_name, url_prefix)` became `__init__(url_prefix, blueprint_name)`. Regenerated: the documentation reported the swapped order. Five for five. It reads the code.
What didn't work is the useful part.
I first tried detecting this by diffing the documentation of mutated code against the original. It can't work, and finding out why was worth more than the result. Even at temperature 0 — the setting everyone assumes is deterministic — this model paraphrases itself: two runs over identical code produced documents only 49% alike. Against that churn, inverting an `if` moved the prose no more than renaming a local variable did. My cosmetic control is what caught it. You cannot measure generated documentation by comparing it to more generated documentation; you have to compare it to the code.
Five developers told me the documentation was complete, and every instrument I built afterwards agreed with them. The answer never changed — but it went from something I was told to something I could check, and each instrument that failed on the way taught me more than the one that finally worked.