Adia Mahardika
All work

Thesis · LLM evaluation

Does AI-written documentation actually read the code?

I swapped two parameters and asked whether the documentation would notice. It did.

Undergraduate thesis · sole author3 min read8 sections2 figures

In 30 seconds

Question
Developers rated the GPT-4 documentation highly. Was the model reading the code, or guessing from the function's name?
Method
A controlled perturbation: swap the order of two inputs, leave the name and body untouched, and regenerate the documentation.
Finding
It reported the new order in 5 of 5 tests. A separate evaluation harness checked 19 files and found no invented names.

At a glance

My role
Solo · full-stack + research design & evaluation
Type
Undergraduate thesis
Scope
Generation tool · GPT-4 integration · Likert evaluation
Stack
GPT-4 · Next.js · Flask · PostgreSQL · GitHub API

A full-stack research product (my undergraduate thesis, published) that reads source from GitHub and generates structured documentation with GPT-4, returned as reviewable Markdown. Twelve developers shaped the requirements before I wrote any code; five rated the result on a Likert scale and every rating came back positive. I had no way to know whether they were right, so afterwards I wrote one.

AI Docs dashboard listing connected GitHub repositories
Figure 1. AI Docs dashboard listing connected GitHub repositories

1The question was 'useful,' not 'more.'

AI can clearly generate text about code.

Read moreShow less

The thesis asked something narrower and harder: could it produce a draft developers found useful enough to actually start from?

2Full-stack, end to end.

A Next.js dashboard and file workspace, a Flask REST API wiring the GitHub and OpenAI APIs, and PostgreSQL behind JWT auth — built solo.

Read more and see the screenShow less

Generation is stateless: nothing on that path is persisted, and the only table in the database is users. Your GitHub token lives in the browser session and never reaches it.

AI Docs file selection and source viewer
Figure 2. AI Docs file selection and source viewer

3Twelve shaped it, five judged it.

The requirements came from twelve developers before I wrote code — most of them past five years' experience.

Read moreShow less

Five then rated the output on completeness, readability, relevance, and how much it helped with documentation time. Twenty ratings, every one of them positive, not a single negative.

4The result is about my instrument.

Twenty out of twenty is not a finding about a tool.

Read moreShow less

My respondents answered under their own names, on a scale with no neutral option, and nobody ever documented a file by hand to compare against. I set out to ask whether the output was accurate and built something that measured whether people liked it. So I stopped asking people.

5So I measured the code instead.

A harness that parses each file with Python's AST and checks every name the documentation asserts against what the source actually contains.

Read moreShow less

Across nineteen files of this app's own backend it found all seventy-one public definitions documented and nothing invented: it flagged four names, and all four turned out to be my harness's fault rather than the model's. That is not the output the five developers rated — different model, different code, a year later — so it settles nothing about their verdict. It is simply the first number in this project that can be checked at all.

6The harness was wrong before the model was.

My first run accused the model of inventing 58 names and omitting four in ten public functions.

Read moreShow less

Both were my bugs. It read "no `__all__` declarations are defined" as a claim that `__all__` exists, scored a Markdown table header as a dependency, and couldn't see a function documented as `ErrorHandler.handle_bad_request(self, error)` because it only looked for bare names. Every headline I nearly published was an artifact of my own parser — which is the same failure the study had, wearing a lab coat.

7Then I broke the code to see if the docs noticed.

Grounding couldn't fail — the whole file is in the prompt, so of course the model doesn't invent identifiers.

Read moreShow less

The failure that matters is subtler: does it read the code, or describe what the function is called? So I swapped two parameters in a signature and left the name, docstring and body untouched. `__init__(blueprint_name, url_prefix)` became `__init__(url_prefix, blueprint_name)`. Regenerated: the documentation reported the swapped order. Five for five. It reads the code.

8What didn't work is the useful part.

I first tried detecting this by diffing the documentation of mutated code against the original.

Read moreShow less

It can't work, and finding out why was worth more than the result. Even at temperature 0 — the setting everyone assumes is deterministic — this model paraphrases itself: two runs over identical code produced documents only 49% alike. Against that churn, inverting an `if` moved the prose no more than renaming a local variable did. My cosmetic control is what caught it. You cannot measure generated documentation by comparing it to more generated documentation; you have to compare it to the code.

  • Next.js
  • React
  • TypeScript
  • Flask
  • Python
  • REST API
  • PostgreSQL
  • OpenAI / GPT-4
  • GitHub API
  • AST verification