--- name: open-ended-coding description: "Builds a codebook from open-ended survey answers, applies it to every row of a CSV with a definition for every code, counts themes with honest denominators, pulls verbatim quotes with row ids, and checks coder agreement (Cohen's kappa, Krippendorff's alpha) on a double-coded sample, human or LLM. Use for \"code these responses\", \"themes from open-ended answers\", \"thematic analysis of survey comments\", \"categorize or tag free-text feedback\", \"analyze NPS comments or verbatims\", \"inter-rater reliability\" or \"does the LLM coding agree with mine\"." license: MIT metadata: version: "0.2.0" status: "beta" author: "QuestionPunk" source: "questionpunk/research-skills" requires-account: "false" plugin: "feedback-analysis" --- # Open-ended coding Turns a column of free-text survey answers into a codebook, a coded file, a frequency table with stated denominators, verbatim quotes with row ids, and a coder-agreement check with confidence intervals. You do the coding: reading, defining codes, assigning them to every row. The script does the arithmetic that goes wrong in conversation: [`scripts/open_ended_coding.py`](scripts/open_ended_coding.py) (Python 3.9+, standard library only). Run it; do not redo the math yourself. Run it by its path inside this skill's directory: the commands below write `scripts/open_ended_coding.py`, meaning `python3 /scripts/open_ended_coding.py` from wherever the data is. References: - [`references/sources.md`](references/sources.md): formulas, thresholds, citations, what is convention. - [`references/formats-and-briefs.md`](references/formats-and-briefs.md): file formats, the second coder's brief, adjudication, the reporting checklist, every script option. - [`references/llm-coding.md`](references/llm-coding.md): what studies of LLM coders found. ## When to use Use when the user has open-ended answers, one row per respondent, and asks: - "Code these responses", "categorize or tag this free-text feedback", "build a code frame". - "Themes from the open-ended answers", "thematic analysis of survey comments". - "Analyze the NPS comments / verbatims", "how many people mentioned price". - "Inter-rater reliability", "what is the kappa between these coders". - "Does the LLM coding agree with mine", "check the model's tags against my sample". Do not use it for: - The NPS score, its interval or trend: `nps-analysis` (it sends the comments here). - Removing speeders, duplicates and gibberish rows first: `survey-data-cleaning`. - Writing the question or the interview guide: `survey-from-decision`, `ai-interview-guide`. - Significance tests between segments, or cross-tabs of closed questions. `counts` cuts code shares by a segment with an interval per cell; it does not test the differences. - Sentiment scoring or topic models. This skill can count a sentiment column you filled; it does not score text. - Weighted estimates. `counts` is unweighted; it takes no survey weight column. ### Which kind of analysis is this? | The user wants | Do | |---|---| | Counts: "how many said what", comparable across waves or segments, many short answers | **Codebook coding** (this skill): fixed definitions, every row coded, agreement measured. | | Meaning: an interpretive account of a few long interviews, themes developed by one analyst | **Reflexive thematic analysis** (Braun and Clarke). Not this skill: no fixed codebook, no kappa. Say so; offer codebook coding only if counts are what they need. | Never call this skill's output "Braun and Clarke thematic analysis": they reject agreement statistics for their method. A skill named in this section may not be installed. If it is missing, say so, name the method the request needs, and do not stretch this skill to cover it. ## What to ask for Start from what the user gave you. Ask one question only if the file or the question wording is missing: "Which column holds the answers, what exactly were respondents asked, and what decision should the coding inform?" The question says what is on-topic; the decision says which distinctions the codes must keep. | Input | Default if not given | |---|---| | CSV, one row per respondent, with the answer column | Required. Pasted text: save it as CSV with `id` = line number and say so. | | Id column | `id`. None in the file: add `id` = row number and say so. | | Question wording | Required to judge what is on-topic. Use the column header if it is the full question. | | Decision the coding supports | Ask once; if unknown, say the codebook follows the respondents' own distinctions. | | Existing codebook | None: build one from a sample. If given, convert it to the fixed format, keep its codes. | | Codes per response | Every code that applies, each counted once per response. One code only if asked. | | Nets (parent codes) | None. Add a `parent` column to the codebook if the user wants roll-ups. | | Segment column | None: overall counts only. | | Second coder | A person codes the sample blind if one is available; otherwise a fresh-context pass, labelled as such. | | Double-coded sample | 20% of answered rows, at least 30 (all rows if fewer): `--fraction 0.2 --at-least 30`. | | Small-base threshold | 30 (`--min-base`): below it, counts only. | | Seeds | 1 for the codebook sample, 2 for the pilot, 3 for the double-coded sample (a new seed per draw), 1 for the bootstrap. Report them. | ## Steps The order is fixed: sample, draft codebook, pilot, freeze a version, code everything, double-code, adjudicate, count. Do not skip or reorder. One exception: when the user brings files that are already coded and asks only for agreement, start at step 8 (then step 9 if they want the disputes settled) and report only the summary line, `### Agreement` and `### Caveats`. 1. **Look at the file.** Read the header and some rows; count rows and blanks. Say so if duplicates or test rows are visible. Rows are respondents: never drop blank rows, they are a category. Files are read as UTF-8 (a BOM is fine); a semicolon, tab or pipe between columns is detected when the header has no comma, and `--csv-delimiter` sets it. `--delimiter` (alias `--code-sep`) is the separator between codes inside the `codes` cell, not between columns. 2. **Sample** for the codebook: 100 answered rows, or all of them if fewer, in a seeded random order. ```bash python3 scripts/open_ended_coding.py sample responses.csv --text-col answer --n 100 --all-if-fewer --shuffle --seed 1 --out sample_codebook.csv ``` 3. **Draft the codebook.** Read the sample top to bottom (the shuffled order) and code it as you go, in the respondents' terms: save it as `sample_codebook_coded.csv` with a `codes` column added, every row you have read coded. Then check the stopping rule (base of 50 answers, runs of 25 moving one answer at a time, stop at 5% new codes or fewer): ```bash python3 scripts/open_ended_coding.py saturation sample_codebook_coded.csv ``` It refuses rows with no code: an unread row is not evidence. Under 75 rows, pass a smaller `--base` and `--run` and say which. If the rule is not met, draw 50 more (`--exclude sample_codebook_coded.csv --shuffle`), append them to the end of the coded sample, code them and rerun. Merge codes you cannot tell apart; split a code holding two ideas a reader would act on differently. Write one row per code: | Column | Content | |---|---| | `code` | Short upper-case name, no spaces: `PRICE` | | `label` | What the reader sees in the table | | `definition` | One or two sentences: what the respondent is saying | | `include` | Cases that count, especially borderline ones | | `exclude` | Near misses, and which code takes them instead | | `example_quote` | One answer from the data, word for word | A code without `include` and `exclude` is not finished. Reserved codes: `OTHER` (a real answer that fits no code), `UNCODABLE` (typed, no usable content), `NO_ANSWER` (blank). 4. **Pilot.** Code 30 fresh answers with the draft and have the second coder (the one step 7 uses) do the same, then compare informally. Fix definitions that were read two ways. ```bash python3 scripts/open_ended_coding.py sample responses.csv --text-col answer --exclude sample_codebook_coded.csv --n 30 --all-if-fewer --seed 2 --out pilot.csv ``` If it reports no eligible answers, the codebook sample used them all: skip the pilot, say so, and let the double-coded sample in step 7 be the first test. 5. **Freeze version 1.** From here every change is a new version with a line in the change log, and **any change to a definition means recoding every row it could affect** and re-testing agreement. 6. **Code every row** in batches small enough that you read each answer in full; with thousands of rows keep going batch by batch, and never swap reading for keyword rules. For each batch write `id,codes` (`;` between codes) to a batch file, blank answers included as `id,NO_ANSWER`; never retype the answers. An LLM coder may add a `rationale` column (the words that triggered each code, see [`references/llm-coding.md`](references/llm-coding.md)); `merge` ignores it. Assign every code whose definition is met; `exclude` decides borderline cases; never invent a code mid-pass (use `OTHER`); `UNCODABLE` and `NO_ANSWER` stand alone. Answers in another language are coded in that language. Counts come from the fully coded file, never from the sample. Merge the batches by id, then validate against the original file until clean: ```bash python3 scripts/open_ended_coding.py merge responses.csv --adjudicated batch_01.csv --adjudicated batch_02.csv --final-col codes --out coded.csv python3 scripts/open_ended_coding.py validate coded.csv --codebook codebook.csv --text-col answer --source responses.csv ``` `--source` confirms that every respondent is in the coded file with the same answer text. Fix the rows it lists; do not edit the codebook to make errors disappear. 7. **Double-code a blind sample** of answers that neither built nor piloted the codebook (the output holds only id and answer; `--stratify-col` spreads the draw over segments): ```bash python3 scripts/open_ended_coding.py sample coded.csv --text-col answer --exclude sample_codebook_coded.csv --exclude pilot.csv --fraction 0.2 --at-least 30 --seed 3 --out sample_second.csv ``` The share is of the answers not excluded; report the sample's share of all answered rows. If it draws fewer than 30 (a small file), rerun without the `--exclude` options and say in the report that the agreement sample overlaps the rows used to build the codebook. Never reuse an earlier seed on the same file: the draw would repeat the codebook sample. `--top-up-min 5 --top-up-out audit_rare.csv` also writes rows for rare codes to a separate audit file. They are chosen from your codes, so they are coded blind too but kept out of kappa and alpha: read them only for disagreements. - **A person is available:** stop here and hand them `sample_second.csv`, `codebook.csv` and the [brief](references/formats-and-briefs.md#brief-for-a-blind-second-coder); they code it without seeing your codes, and you continue when they return it. This is the only check that says whether a model's reading matches a human's. - **No person:** give the [brief](references/formats-and-briefs.md#brief-for-a-blind-second-coder), the codebook and the sample to a fresh context that has seen neither this conversation nor your codes: a subagent, or where the client has none, a new chat or session given only those three files. If neither is possible, say the second pass was not blind. **An LLM coding twice is not double coding**: it shows whether the definitions can be applied consistently, nothing more. Label it "same-model second pass" everywhere and recommend a human check before a decision that rests on a close count. 8. **Compute agreement** and write out the disputed rows: ```bash python3 scripts/open_ended_coding.py agreement coded.csv second_coded.csv --sample sample_second.csv \ --codebook codebook.csv --text-col answer --disagreements disagreements.csv \ --coder-info "claude-run1,llm,MODEL" --coder-info "anna,human" --codebook-version v1 ``` With a person's coding as the reference, pass it as `--gold human.csv` (listed first in `--coder-info`) to add precision, recall and F1 per code. Three or more files give alpha per code and a pairwise kappa table. Read the result per code, with its interval: - kappa under 0.60 or alpha under 0.667: revise the code (tighten, merge or split), new version, recode, test again on a fresh sample (`--exclude sample_second.csv`); - alpha from 0.667 to 0.800, or an interval that crosses a band (marked `*`): report as tentative; - "not tested": neither coder used the code in the sample. Say so; do not report it as agreement; - "on every row": every coder gave the code to every answer, so kappa and alpha are undefined. Report the percent agreement and say the sample could not test the code; - "high agreement, low kappa": a rare code. Judge it on positive agreement and the disputed rows. Agreement is kappa or alpha with percent agreement beside it. Chi-square, Pearson's r and Cronbach's alpha are not agreement measures; do not report them as inter-rater reliability. 9. **Adjudicate.** For each row in `disagreements.csv`, decide the final codes by the codebook (a person decides when one is involved) and fill `final_codes`. Then: ```bash python3 scripts/open_ended_coding.py merge coded.csv --adjudicated disagreements.csv --out coded_final.csv ``` Agreement figures stay as measured before adjudication. 10. **Count**, and read the codebook health section it prints: ```bash python3 scripts/open_ended_coding.py counts coded_final.csv --codebook codebook.csv --text-col answer --segment-col plan ``` Use the numbers as printed. `--` means the base is under the threshold: report "7 of 17" and do not compare segments flagged `SMALL BASE`. If health flags `OTHER`, a code on most answers, or two codes that mostly occur together, fix the codebook (new version, recode) before reporting. Report nets from the nets table, never by adding code counts. 11. **Pick quotes.** For each top code choose 2–3 answers that show its range. Copy them word for word with the row id into `quotes.csv` (`id,code,quote`) and check them. Trim with "..." only at the ends (the check ignores them there); never join two answers. ```bash python3 scripts/open_ended_coding.py validate coded_final.csv --codebook codebook.csv --text-col answer --source responses.csv --quotes quotes.csv ``` After the check, replace names, emails and phone numbers with `[name]`, `[email]`, `[phone]` in the quotes and in the codebook's example quotes you show, and say quotes were redacted. Exit code 2 means bad input: fix the cause the message names. `--json` works on every subcommand. ## Output Always this shape, in this order (an agreement-only request: the summary line, `### Agreement` and `### Caveats`). ```markdown ## Open-ended coding: "" **: 7 of 17 who answered.** (With 30 or more who answered: " of (%, 95% interval to )".) Data: , answer column ``. 20 respondents; 17 answered, 3 blank (No answer); 2 of the answers uncodable. Method: codebook v built from a random sample of answers (seed ), applied to every row. ### Codebook (v) | Code | Label | Definition | Include | Exclude | Example quote | ### Frequencies | Code | n | % of all respondents (N) | % of those who answered (n) | 95% interval | | ... / Other / Uncodable | Responses can carry several codes, so shares add up to more than 100%. ### By (only if asked) | Code | n (%, 95% interval) of | n (%, 95% interval) of | Note | ### Quotes **