Your satisfaction score comes from a poll in the community channel, where fifteen regulars gave the API a 4.6. A third of the keys issued last month have gone quiet, so the developers with the most to say were never asked. The score is accurate for the people who answered, but the budget meeting will read it as the whole developer base.
What you get
You will end up with a sample drawn from keys that made calls, an eight-question survey, and a report that scores each segment and names the metric behind each low score. This is for you if your only satisfaction score is self-selected.
Short answer
Draw the sample from API keys that made a call in the last 30 days, stratified by SDK language and by tenure. A Node script that anyone can re-run makes the same draw every time. Ask eight questions: one satisfaction score, one effort score for each of install, auth, first call, errors, and docs, one open question, and one consent box. Report each segment separately and attach a baseline metric to every low score.
You will need
Node 22 or later, a list of active API keys with the SDK language and the first and last call
date of each, and contact emails for those keys. The scorecard from
choose developer experience metrics for your API
supplies the baseline metrics the report points at. Verified 2026-10-21 against Node 22.22.0. The
language of each key comes from the User-Agent header your SDKs send, which
RFC 9110 defines as information
about the software originating the request. The survey is the satisfaction dimension of
SPACE,
whose authors argue that productivity cannot be measured by a single metric or dimension.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Google Forms or Typeform email survey | Any list of addresses, including the developers who stopped calling, with no code in the portal | A return under a tenth of the list, skewed to the people who felt strongly either way | You need an answer this week from the people using the API today |
| In-portal prompt after login | A dashboard developers visit anyway, and a question short enough to answer before the page they came for | Only people who still log in see it, so the developers who left are absent from the result | The developers you most need to hear from no longer open the portal |
| Structured interviews scored against the same questions | Five people, thirty minutes each, when the scores need a reason behind them | Two days of calendar time for five data points, and an interviewer who hears what they expected | You need a number per segment rather than five explanations |
An email survey reaches everyone on the list, including the developers who stopped calling, and the planning assumption is a return under a tenth of it.1 The portal prompt returns several times that share, from the people who still log in, the ones least likely to have left. Interviews explain the scores and cannot be averaged: five conversations are five data points,2 read against one discussion guide. Run the first two together and score the interviews against the same eight questions.
Draw the sample from keys that made calls
The frame is every key that made a call in the 30 days up to the day you draw, read from gateway logs rather than from the sign-up table. A key that never called has no experience to rate, and a key that went quiet months ago belongs in a different survey. Each key in the frame falls in a stratum: its SDK language, and its tenure counted from its first call.
Three functions in survey.mjs decide who is in the frame and where each key sits:
/** A key is in the frame when it made a call in the `days` days up to `asOf`. */
export const isActive = (key, asOf, days = 30) => daysBetween(key.last_call, asOf) <= days
export function tenure(key, asOf) {
const days = daysBetween(key.first_call, asOf)
if (days <= 7) return TENURES[0]
if (days > 182) return TENURES[2]
return TENURES[1]
}
export const stratumOf = (key, asOf) => `${key.language} / ${tenure(key, asOf)}`draw.mjs allocates 40 keys across the strata in proportion, by largest remainder. It then sorts
each stratum by a SHA-256 hash from node:crypto over a
seed and the key id, and takes the allocated number from the top. A sort is not a lottery: keep
the seed and a colleague gets the same 40 keys from the same list.
node draw.mjsactive keys in the 30 days to 2026-09-30: 69 of 80
11 keys made no call in that window and are outside the frame
stratum keys drawn
go / first week 9 5
go / one week to six months 3 2
go / over six months 3 2
python / first week 4 2
python / one week to six months 10 6
python / over six months 7 4
ruby / first week 3 2
ruby / one week to six months 3 2
ruby / over six months 2 1
typescript / first week 9 5
typescript / one week to six months 11 6
typescript / over six months 5 3
total 69 40
first five drawn: key_045 key_029 key_046 key_066 key_059Eleven keys stay outside the frame. Ask those one question by email, about why they stopped, and keep the answers out of the score.
Ask eight questions and refuse the ninth
Eight questions fit on one screen and take under three minutes. The composition is fixed: one satisfaction score, five effort scores, one open question, and one consent box. The NN/g guide to qualitative surveys says that respondents tire of answering many open-ended questions, which is why there is one.
The first three entries of questions.json carry their scale, so a report can never read a 5 as
good on one scale and bad on the other:
{ "id": "sat", "kind": "satisfaction", "scale": "1 very dissatisfied to 5 very satisfied",
"text": "Rate your overall satisfaction with the Acme API." },
{ "id": "effort-install", "kind": "effort", "stage": "install", "scale": "1 no effort to 5 very high effort",
"text": "Rate the effort it took to install the SDK, or to make a first request without one." },
{ "id": "effort-auth", "kind": "effort", "stage": "auth", "scale": "1 no effort to 5 very high effort",
"text": "Rate the effort it took to get a key and make an authenticated request." },The effort scale descends from the Customer Effort Score and runs the other way from satisfaction: 5 is the most effort, which is the bad end.3 The consent box is the eighth question rather than an afterthought: the informed consent an interview needs starts there, with who is asking, why, what is kept, and how to withdraw.
A guard in survey.mjs checks the composition rather than the count, so a second satisfaction
question dressed as a recommendation score is refused:
node nine.mjseight questions: accepted
nine questions: 9 questions, the survey has eight; 2 satisfaction questions, wanted 1Eight is a rule to defend in review, because every stakeholder has a ninth.
Report by segment, and point each low score at a metric
A single satisfaction number is the thing the survey exists to replace. report.mjs prints
satisfaction by language and by tenure, with the count beside each mean, and refuses to print a
cell with fewer than three answers, because a mean of two people names them. Any stage with a
mean effort of 3.0 or worse is paired with the baseline metric that says whether the complaint
shows in the traffic.
The pairing is a table in score.mjs, taken from the scorecard on the metrics page:
export const BASELINE = {
install: 'install friction: cold install time and dependency count per language',
auth: 'share of keys issued this quarter that never make a successful call',
'first-call': 'time to first successful call, median and 90th percentile',
errors: '4xx share per endpoint per distinct caller',
docs: 'operations carrying an example, and operations documenting their failures',
}node report.mjsresponses: 18 of 40 sampled keys (45%), portal 13, email 5
satisfaction, 1 to 5, by language and tenure
language first week one week to six months over six months all
go n<3 n<3 n<3 2.8 (4)
python n<3 3.0 (4) 4.5 (4) 3.8 (8)
typescript n<3 2.7 (3) n<3 2.8 (6)
all 2.3 (3) 2.9 (8) 4.0 (7) 3.2 (18)
effort, 1 to 5, by stage
stage all first week over six months
install 2.6 3.7 (3) 2.3 (7)
auth 2.4 3.3 (3) 2.4 (7)
first-call 1.9 2.0 (3) 1.7 (7)
errors 3.5 4.0 (3) 3.4 (7)
docs 3.2 3.7 (3) 3.3 (7)
stages scoring 3.0 or worse, and the baseline metric to read next
errors 3.5 4xx share per endpoint per distinct caller
docs 3.2 operations carrying an example, and operations documenting their failures
16 of 18 answered the open question; 8 agreed to a follow-up interviewThe whole sample scores 3.2, first-week developers 2.3, and developers past six months 4.0, so the average describes nobody. The errors stage scores worst, at 3.5, and the metric to read next is the 4xx share per endpoint per caller.
Check it worked
Thirteen tests pin the rules, with Node’s own node:test runner.
The draw is identical on a second run with the same seed and differs with a
different one. The community channel alone scores at least half a point higher than the
stratified sample, with none of the quiet developers in it. The last three feed the scorer a
skipped answer, the tenure bands a date that does not parse, and the draw a Danish locale.
node --test survey.test.mjs1..13
# tests 13
# suites 0
# pass 13
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 203.727108If # pass 13 drops to twelve after an edit to questions.json, the composition changed.
When it goes wrong
A 4.6 arrives in the same month as a rise in churn. The poll ran in the community channel, so it sampled the people who chose to stay. Draw from keys instead, and if the channel poll must appear in the report, label it as the channel’s own line.4 The fixture shows the gap. The same question went to the channel and to the stratified sample, and the last column is the share of each group that made no call for two weeks.
node pitfall.mjswho answered n satisfaction quiet 14+ days
community channel members 15 4.6 0%
stratified sample 18 3.2 39%
of which, quiet 14+ days 7 3.0
of which, called this fortnight 11 3.4The channel scores 4.6 with nobody quiet, because a developer who has drifted away is not in the channel to be asked. The seven sampled people who have gone quiet score 3.0, and that 3.0 is the number the 4.6 hides.
Most answers came from the portal prompt and every segment looks fine, because the developers who stopped logging in never saw the prompt. Compare the tenure mix of the respondents with that of the frame, and email any stratum the portal missed.
A combined DX score of 4.0 appears on a slide. Someone averaged the satisfaction item with the five effort items, whose 5 is the bad end, so more effort pushed the score up. Report the two scales as separate rows, each with its direction printed beside it.
When not to do this
Do not run the survey before the baseline exists. Each low score on the report points at a metric, and with no metric behind it a 3.5 for errors is an opinion with a decimal point.
Do not run it more than once a quarter, or send it to the same keys twice in a row. A developer who is asked every month stops answering, and the ones who keep answering are the engaged minority the sample was built to dilute.
Do not report a cell with two people in it, however much a stakeholder wants the Ruby number. Two answers identify two people, and the consent box promised a follow-up call rather than a leak. Participant privacy is part of the method a developer survey borrows from user research, along with the questions.
Do not treat the open question as data. Sixteen answers in prose are sixteen anecdotes, and the one you remember is the one written in anger.
Related how-tos
Last verified
Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The keys and the answers are a constructed fixture of 80 keys and 33 responses, not a real program’s quarter. No email was sent, no prompt was shown, and no interview was held. Google Forms, Typeform, and the interview guide are cited from their documentation and were not used.
Footnotes
-
Pew Research Center reports the response rate to its telephone polls at 7% in 2017 and 6% in 2018. It adds that its own studies from 1997, 2003, 2012, and 2016 found little relationship between response rates and accuracy. That comforts a pollster with a weighted national sample more than a developer program with four hundred addresses, where 6% is twenty-four people. ↩︎ Back to text
-
The number five comes from usability testing rather than from interviewing. Nielsen’s Why You Only Need to Test with 5 Users says that a first study with five participants finds 85% of the usability problems. An interview is a different method, as the same group’s guide to user interviews explains, and the figure has traveled to it regardless. ↩︎ Back to text
-
The effort item descends from the Customer Effort Score, which Dixon, Freeman, and Toman introduced in the Harvard Business Review in 2010 as Stop Trying to Delight Your Customers. It was one question about the customer’s own effort, scored from 1 to 5 with 5 the most effort. The authors reported that it predicted loyalty better than satisfaction measures or the Net Promoter Score. ↩︎ Back to text
-
Stack Overflow’s survey prints its own sampling caveat. The methodology page of the 2025 edition reports 49,009 responses from 177 countries, recruited primarily through channels the company owns, from onsite messaging to newsletter subscribers. It then says that highly engaged users were more likely to notice the prompts. That sentence is the community-channel pitfall, written down by a team with 49,009 answers and the candor to say who gave them. ↩︎ Back to text