# How to run a developer satisfaction survey for your API > Draw the sample from keys that made calls, ask eight questions, and report every segment with the baseline metric behind each low score. Source: https://voxgig.com/howto/run-a-developer-satisfaction-survey-for-your-api - Audience: devrel - Level: beginner - Verified: 2026-10-21 - Published: 2026-10-19 ## Short answer Draw the sample from API keys that made a call in the last 30 days, stratified by SDK language and by tenure. A Node script that anyone can re-run makes the same draw every time. Ask eight questions: one satisfaction score, one effort score for each of install, auth, first call, errors, and docs, one open question, and one consent box. Report each segment separately and attach a baseline metric to every low score. --- ## You will need Node 22 or later, a list of active API keys with the SDK language and the first and last call date of each, and contact emails for those keys. The scorecard from [choose developer experience metrics for your API](/howto/choose-developer-experience-metrics-for-your-api) supplies the baseline metrics the report points at. Verified 2026-10-21 against Node 22.22.0. The language of each key comes from the `User-Agent` header your SDKs send, which [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-user-agent) defines as information about the software originating the request. The survey is the satisfaction dimension of [SPACE](https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/), whose authors argue that productivity cannot be measured by a single metric or dimension. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Google Forms](https://developers.google.com/workspace/forms/api/guides/retrieve-forms-responses) or [Typeform](https://www.typeform.com/developers/responses/) email survey | Any list of addresses, including the developers who stopped calling, with no code in the portal | A return under a tenth of the list, skewed to the people who felt strongly either way | You need an answer this week from the people using the API today | | [In-portal prompt after login](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/dialog) | A dashboard developers visit anyway, and a question short enough to answer before the page they came for | Only people who still log in see it, so the developers who left are absent from the result | The developers you most need to hear from no longer open the portal | | [Structured interviews scored against the same questions](https://www.gov.uk/service-manual/user-research/using-in-depth-interviews) | Five people, thirty minutes each, when the scores need a reason behind them | Two days of calendar time for five data points, and an interviewer who hears what they expected | You need a number per segment rather than five explanations | An email survey reaches everyone on the list, including the developers who stopped calling, and the planning assumption is a return under a tenth of it.[^1] The portal prompt returns several times that share, from the people who still log in, the ones least likely to have left. Interviews explain the scores and cannot be averaged: five conversations are five data points,[^2] read against one discussion guide. Run the first two together and score the interviews against the same eight questions. ## Draw the sample from keys that made calls The frame is every key that made a call in the 30 days up to the day you draw, read from gateway logs rather than from the sign-up table. A key that never called has no experience to rate, and a key that went quiet months ago belongs in a different survey. Each key in the frame falls in a stratum: its SDK language, and its tenure counted from its first call. Three functions in `survey.mjs` decide who is in the frame and where each key sits: ```js title="survey.mjs" /** A key is in the frame when it made a call in the `days` days up to `asOf`. */ export const isActive = (key, asOf, days = 30) => daysBetween(key.last_call, asOf) <= days export function tenure(key, asOf) { const days = daysBetween(key.first_call, asOf) if (days <= 7) return TENURES[0] if (days > 182) return TENURES[2] return TENURES[1] } export const stratumOf = (key, asOf) => `${key.language} / ${tenure(key, asOf)}` ``` `draw.mjs` allocates 40 keys across the strata in proportion, by largest remainder. It then sorts each stratum by a SHA-256 hash from [`node:crypto`](https://nodejs.org/api/crypto.html) over a seed and the key id, and takes the allocated number from the top. A sort is not a lottery: keep the seed and a colleague gets the same 40 keys from the same list. ```bash node draw.mjs ``` ```text output active keys in the 30 days to 2026-09-30: 69 of 80 11 keys made no call in that window and are outside the frame stratum keys drawn go / first week 9 5 go / one week to six months 3 2 go / over six months 3 2 python / first week 4 2 python / one week to six months 10 6 python / over six months 7 4 ruby / first week 3 2 ruby / one week to six months 3 2 ruby / over six months 2 1 typescript / first week 9 5 typescript / one week to six months 11 6 typescript / over six months 5 3 total 69 40 first five drawn: key_045 key_029 key_046 key_066 key_059 ``` Eleven keys stay outside the frame. Ask those one question by email, about why they stopped, and keep the answers out of the score. ## Ask eight questions and refuse the ninth Eight questions fit on one screen and take under three minutes. The composition is fixed: one satisfaction score, five effort scores, one open question, and one consent box. The [NN/g guide to qualitative surveys](https://www.nngroup.com/articles/qualitative-surveys/) says that respondents tire of answering many open-ended questions, which is why there is one. The first three entries of `questions.json` carry their scale, so a report can never read a 5 as good on one scale and bad on the other: ```json title="questions.json" { "id": "sat", "kind": "satisfaction", "scale": "1 very dissatisfied to 5 very satisfied", "text": "Rate your overall satisfaction with the Acme API." }, { "id": "effort-install", "kind": "effort", "stage": "install", "scale": "1 no effort to 5 very high effort", "text": "Rate the effort it took to install the SDK, or to make a first request without one." }, { "id": "effort-auth", "kind": "effort", "stage": "auth", "scale": "1 no effort to 5 very high effort", "text": "Rate the effort it took to get a key and make an authenticated request." }, ``` The effort scale descends from the Customer Effort Score and runs the other way from satisfaction: 5 is the most effort, which is the bad end.[^3] The consent box is the eighth question rather than an afterthought: the [informed consent](https://www.gov.uk/service-manual/user-research/getting-users-consent-for-research) an interview needs starts there, with who is asking, why, what is kept, and how to withdraw. A guard in `survey.mjs` checks the composition rather than the count, so a second satisfaction question dressed as a recommendation score is refused: ```bash node nine.mjs ``` ```text output eight questions: accepted nine questions: 9 questions, the survey has eight; 2 satisfaction questions, wanted 1 ``` Eight is a rule to defend in review, because every stakeholder has a ninth. ## Report by segment, and point each low score at a metric A single satisfaction number is the thing the survey exists to replace. `report.mjs` prints satisfaction by language and by tenure, with the count beside each mean, and refuses to print a cell with fewer than three answers, because a mean of two people names them. Any stage with a mean effort of 3.0 or worse is paired with the baseline metric that says whether the complaint shows in the traffic. The pairing is a table in `score.mjs`, taken from the scorecard on the metrics page: ```js title="score.mjs" export const BASELINE = { install: 'install friction: cold install time and dependency count per language', auth: 'share of keys issued this quarter that never make a successful call', 'first-call': 'time to first successful call, median and 90th percentile', errors: '4xx share per endpoint per distinct caller', docs: 'operations carrying an example, and operations documenting their failures', } ``` ```bash node report.mjs ``` ```text output responses: 18 of 40 sampled keys (45%), portal 13, email 5 satisfaction, 1 to 5, by language and tenure language first week one week to six months over six months all go n<3 n<3 n<3 2.8 (4) python n<3 3.0 (4) 4.5 (4) 3.8 (8) typescript n<3 2.7 (3) n<3 2.8 (6) all 2.3 (3) 2.9 (8) 4.0 (7) 3.2 (18) effort, 1 to 5, by stage stage all first week over six months install 2.6 3.7 (3) 2.3 (7) auth 2.4 3.3 (3) 2.4 (7) first-call 1.9 2.0 (3) 1.7 (7) errors 3.5 4.0 (3) 3.4 (7) docs 3.2 3.7 (3) 3.3 (7) stages scoring 3.0 or worse, and the baseline metric to read next errors 3.5 4xx share per endpoint per distinct caller docs 3.2 operations carrying an example, and operations documenting their failures 16 of 18 answered the open question; 8 agreed to a follow-up interview ``` The whole sample scores 3.2, first-week developers 2.3, and developers past six months 4.0, so the average describes nobody. The errors stage scores worst, at 3.5, and the metric to read next is the 4xx share per endpoint per caller. ## Check it worked Thirteen tests pin the rules, with Node's own [`node:test`](https://nodejs.org/api/test.html) runner. The draw is identical on a second run with the same seed and differs with a different one. The community channel alone scores at least half a point higher than the stratified sample, with none of the quiet developers in it. The last three feed the scorer a skipped answer, the tenure bands a date that does not parse, and the draw a Danish locale. ```bash node --test survey.test.mjs ``` ```text output 1..13 # tests 13 # suites 0 # pass 13 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 203.727108 ``` If `# pass 13` drops to twelve after an edit to `questions.json`, the composition changed. ## When it goes wrong A 4.6 arrives in the same month as a rise in churn. The poll ran in the community channel, so it sampled the people who chose to stay. Draw from keys instead, and if the channel poll must appear in the report, label it as the channel's own line.[^4] The fixture shows the gap. The same question went to the channel and to the stratified sample, and the last column is the share of each group that made no call for two weeks. ```bash node pitfall.mjs ``` ```text output who answered n satisfaction quiet 14+ days community channel members 15 4.6 0% stratified sample 18 3.2 39% of which, quiet 14+ days 7 3.0 of which, called this fortnight 11 3.4 ``` The channel scores 4.6 with nobody quiet, because a developer who has drifted away is not in the channel to be asked. The seven sampled people who have gone quiet score 3.0, and that 3.0 is the number the 4.6 hides. Most answers came from the portal prompt and every segment looks fine, because the developers who stopped logging in never saw the prompt. Compare the tenure mix of the respondents with that of the frame, and email any stratum the portal missed. A combined DX score of 4.0 appears on a slide. Someone averaged the satisfaction item with the five effort items, whose 5 is the bad end, so more effort pushed the score up. Report the two scales as separate rows, each with its direction printed beside it. ## When not to do this Do not run the survey before the baseline exists. Each low score on the report points at a metric, and with no metric behind it a 3.5 for errors is an opinion with a decimal point. Do not run it more than once a quarter, or send it to the same keys twice in a row. A developer who is asked every month stops answering, and the ones who keep answering are the engaged minority the sample was built to dilute. Do not report a cell with two people in it, however much a stakeholder wants the Ruby number. Two answers identify two people, and the consent box promised a follow-up call rather than a leak. [Participant privacy](https://www.gov.uk/service-manual/user-research/managing-user-research-data-participant-privacy) is part of the method a developer survey borrows from user research, along with the questions. Do not treat the open question as data. Sixteen answers in prose are sixteen anecdotes, and the one you remember is the one written in anger. ## Related how-tos - [Audit a DevRel program in one week](/howto/audit-a-devrel-programme-in-one-week) - [Choose a developer community platform for year one](/howto/choose-a-developer-community-platform) - [Choose developer experience metrics for your API](/howto/choose-developer-experience-metrics-for-your-api) - [Score your API's error messages with a rubric](/howto/score-api-error-messages-with-a-rubric) ## Last verified Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The keys and the answers are a constructed fixture of 80 keys and 33 responses, not a real program's quarter. No email was sent, no prompt was shown, and no interview was held. Google Forms, Typeform, and the interview guide are cited from their documentation and were not used. [^1]: [Pew Research Center](https://www.pewresearch.org/short-reads/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/) reports the response rate to its telephone polls at 7% in 2017 and 6% in 2018. It adds that its own studies from 1997, 2003, 2012, and 2016 found little relationship between response rates and accuracy. That comforts a pollster with a weighted national sample more than a developer program with four hundred addresses, where 6% is twenty-four people. [^2]: The number five comes from usability testing rather than from interviewing. Nielsen's [Why You Only Need to Test with 5 Users](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/) says that a first study with five participants finds 85% of the usability problems. An interview is a different method, as the same group's [guide to user interviews](https://www.nngroup.com/articles/user-interviews/) explains, and the figure has traveled to it regardless. [^3]: The effort item descends from the Customer Effort Score, which Dixon, Freeman, and Toman introduced in the Harvard Business Review in 2010 as [Stop Trying to Delight Your Customers](https://hbr.org/2010/07/stop-trying-to-delight-your-customers). It was one question about the customer's own effort, scored from 1 to 5 with 5 the most effort. The authors reported that it predicted loyalty better than satisfaction measures or the Net Promoter Score. [^4]: Stack Overflow's survey prints its own sampling caveat. The [methodology page of the 2025 edition](https://survey.stackoverflow.co/2025/methodology) reports 49,009 responses from 177 countries, recruited primarily through channels the company owns, from onsite messaging to newsletter subscribers. It then says that highly engaged users were more likely to notice the prompts. That sentence is the community-channel pitfall, written down by a team with 49,009 answers and the candor to say who gave them.