# How to load API data into Observable Framework with a data loader > Write a data loader that fetches a paginated API under Node at build time with a key from the environment, and read its JSON on the page with FileAttachment. Source: https://voxgig.com/howto/observable-framework-data-loader-for-an-api - Audience: data-analyst - Level: intermediate - Languages: javascript - Verified: 2026-10-21 - Published: 2026-10-17 ## Short answer Put a Node script at src/data/orders.json.js that reads the API URL and key from process.env, follows the pagination, logs to stderr, and writes one JSON document to stdout. Reference it from the page as FileAttachment("data/orders.json").json(). The build runs the loader, hashes the output into dist/_file, and ships the page without the key. A fetch in the page itself is always fresh, but every visitor receives the key. --- ## You will need Node 20 or later, [Observable Framework](https://github.com/observablehq/framework) 1.13, and an API key readable from an environment variable. Verified 2026-10-21 against Node 22.22.0 and @observablehq/framework 1.13.4. The sample pins that version in its `package.json`, so `npm ci` installs the one dependency, and it builds against a fake orders API on 127.0.0.1 that wants a bearer key and pages its results. The key is read with [`process.env`](https://nodejs.org/api/process.html#processenv), which the loader inherits from whatever shell or CI job runs the build. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Client-side fetch inside the page](https://developer.mozilla.org/en-US/docs/Web/API/Window/fetch) | Public data, or data the visitor may see with credentials of their own | A key in the HTML for anyone to copy, a CORS policy the API must grant your origin, and a request per visitor per load | The API needs a key the visitor must not hold, or it sends no CORS headers | | [Observable Framework data loader](https://github.com/observablehq/framework/blob/main/docs/data-loaders.md) | Dashboards, where a snapshot computed at build time is fast and the key stays in the build environment | Data as old as the last build, a build to schedule whenever it should refresh, and a cache that reuses old output until it is cleared | Numbers that must be live to the minute, or data that differs per visitor | | [Observable Notebooks 2.0 HTML notebook](https://github.com/observablehq/notebook-kit) | Personal exploration, where a cell does `await fetch(...)` while you look at the data in a local preview | Code that runs in the browser, so a key in a cell is a key in the file you may later share, with no build step to hide it in | Anything you publish, where the Framework loader keeps the key out of the page | Freshness and secrecy pull in opposite directions. The loader runs where the key lives and produces a file, so the browser gets data without a credential, at the price of data that is as old as the last build. A `fetch` in the page is as fresh as the last reload, and whatever it needs to authenticate ships with the page. The notebook suits finding out what the data looks like,[^1] and stops suiting the moment the file is shared with a key in it. ## Write the loader where the key lives A data loader is an ordinary program whose stdout becomes a file. This one reads the API URL and the key from the environment, follows the pagination, and writes the orders as one JSON array. ```js title="src/data/orders.json.js" const base = process.env.API_URL; const token = process.env.API_TOKEN; if (!base || !token) { console.error("orders.json.js: API_URL and API_TOKEN must be set"); process.exit(1); } const orders = []; let next = "/orders?page=1"; while (next) { const response = await fetch(base + next, { headers: { authorization: `Bearer ${token}` } }); if (!response.ok) { console.error(`orders.json.js: ${next} answered ${response.status}`); process.exit(1); } const page = await response.json(); orders.push(...page.data); console.error(`orders.json.js: ${next} gave ${page.data.length} orders`); next = page.next; } process.stdout.write(JSON.stringify(orders)); ``` Two rules shape it. Every log line goes to `console.error`, which is stderr, because Framework's [data loader documentation](https://github.com/observablehq/framework/blob/main/docs/data-loaders.md) says that anything logged to stdout is included in the output file and sent to the client. And every failure ends in `process.exit(1)`. The same page says a loader that fails must return a non-zero exit code, and that empty output is not by itself treated as an error. A loader that swallows a 401 and exits 0 publishes an empty dashboard with a green build beside it. ```bash node demo.mjs ``` ```text output the loader on its own exit 0, stdout 527 bytes, 1 line stdout parses as JSON: 7 orders stderr: orders.json.js: /orders?page=1 gave 3 orders stderr: orders.json.js: /orders?page=2 gave 3 orders stderr: orders.json.js: /orders?page=3 gave 1 orders ``` Run on its own, the loader is a script like any other: three requests, one line of JSON on stdout, and the progress on stderr where the build log will show it. ## Reference the file and let the build run the loader The page asks for a file that does not exist in the source tree. ```js title="src/index.md" const orders = FileAttachment("data/orders.json").json(); ``` Framework resolves the path relative to the page, finds no `data/orders.json`, and looks for a file of the same name with a second extension, which is how `orders.json.js` becomes the loader for `orders.json`.[^2] The argument has to be a string literal, because the build finds loaders by reading the page rather than running it,[^3] and only files a page references this way are built. The config states the source root, which stops the build warning that `root` is missing, and names the output directory beside it.[^4] ```js title="observablehq.config.js" export default { root: "src", output: "dist", title: "Orders", }; ``` The demo clears the cache and the output, sets `API_URL` and `API_TOKEN`, and runs `observable build` as a child process with the build's own log sent to stderr. ```bash node demo.mjs ``` ```text output observable build pages: index.html, live.html data file: dist/_file/data/orders.79f00478.json, 7 orders index.html references _file/data/orders.79f00478.json: true cache written: true requests the API saw during the build: 3 ``` The loader ran once, under the build, against the API: three requests, the whole of the build's traffic to the API. Its output landed in `dist/_file/data/` under a name that carries a content hash, so a changed snapshot gets a changed URL and a visitor's cache cannot serve the old one. The same output was written to `src/.observablehq/cache/`. A second build would read it from there without running the loader, which the pitfall below turns on. ## Prove the key is in neither the page nor the file A second page in the sample does what the first draft did and fetches from the page itself, with a placeholder where a key would be. ```js title="src/live.md" const token = "sk_live_PLACEHOLDER_do_not_do_this"; const orders = await fetch("https://api.example.com/orders", { headers: { authorization: `Bearer ${token}` }, }).then((response) => response.json()); ``` Framework compiles a page's code blocks into the page's HTML, so a literal in a block is a literal in `dist/live.html`. The loader's key, by contrast, exists only in the process that ran the build. ```bash node demo.mjs ``` ```text output where the token is "secret-orders-token" in dist/index.html: 0, in the data file: 0 "secret-orders-token" in all 7 files under dist/: 0 "sk_live_PLACEHOLDER" in dist/live.html: 1 ``` The count is the verification. The key the build used appears nowhere in the output. The key the page used appears exactly where every visitor can read it. Pointed at the fake API, the second page would also meet [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) in a browser, because the fake sends no `Access-Control-Allow-Origin` header, and most internal APIs send none either. The loader runs under Node, where CORS does not apply. ## Check it worked The fourth test runs a clean build and asserts on the output tree rather than on the loader alone. ```bash node --test loader.test.mjs ``` ```text output 1..4 # tests 4 # suites 0 # pass 4 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 3388.53926 ``` `# pass 4` is the line to look for. The fourth test checks that `dist/_file/data/` holds an `orders..json` with all seven orders and that the cache was written. It then checks that the build's token is in none of the files under `dist` and the placeholder is in `live.html` once. ## When it goes wrong The dashboard shows nothing and the build was green. A `console.log` in the loader went to stdout, which is the file. ```bash node pitfall.mjs ``` ```text output console.error for progress (src/data/orders.json.js) stdout starts: "[{\"id\":\"ord_1001\",\"placed\":\"2026-03-02\"," JSON.parse(stdout): 7 orders console.log for progress (chatty-loader.mjs) stdout starts: "fetched /orders?page=1: 3 orders\nfetched" JSON.parse(stdout): SyntaxError: Unexpected token 'e' ``` The chatty loader exits 0, so the build caches and serves its output, and the page's `.json()` call fails in the browser instead. Log to stderr, and test the loader's stdout with `JSON.parse` before the build does. The dashboard shows last week's numbers after a build. Framework's cache under `src/.observablehq/cache` is read during a build whenever the output is present, with no check of modification times. A build on a machine that kept the cache never runs the loader. Delete the cache before a build that must refresh, as `build.mjs` does, or keep the cache out of CI. The build fails on a file the project never mentions. Inside a larger repository, the bundler walks up from the project looking for a `tsconfig.json`. This sample's first build died on the site's own, which extends a package the sample does not install. A `tsconfig.json` of the project's own, even a small one, stops the walk there. ## When not to do this Do not use a loader for data that must be live or that differs per visitor. The file is a snapshot taken at build time, the same for everyone. A dashboard that promises the current minute needs a `fetch` at load time against an API designed to be called from a browser, with its own short-lived credentials. Do not put a key in a page block because the API allowed your origin. CORS decides which pages may call the API; it does nothing to hide what those pages contain, and the key is in the HTML of every visitor who loads the page. Do not let the notebook become the publication. The HTML notebook format is made for editing and sharing, and a cell that fetched with a key during exploration carries that key into every copy. Move the fetch into a loader before the notebook leaves your machine. Do not let a loader exit 0 on a failure it noticed. An empty or partial file with a zero exit code is cached and served, and the build passes. Framework warns about an empty file in the build log but cannot tell a partial file from a complete one. ## Related how-tos - [Call a keyed API from a browser without shipping the key](/howto/keep-an-api-key-out-of-browser-code) - [Loop over every page of a REST collection](/howto/loop-over-every-page-of-a-collection) - [Add a bearer token to fetch without a client library](/howto/bearer-token-fetch-wrapper) ## Last verified Verified 2026-10-21 against Node 22.22.0 and @observablehq/framework 1.13.4. Every output block is what the command preceding it printed, against the fake orders API in the code directory. The build resolves the versions of Framework's own `npm:` imports over the network, and that is the only traffic that leaves the machine. The Notebook Kit row is read from its documentation and was not run, because its cells execute in a browser. [^1]: The notebook file format is HTML with one `script` element per cell. The [Notebook Kit documentation](https://github.com/observablehq/notebook-kit/blob/main/docs/kit.html) asks that each cell's source use four spaces of indentation, which are trimmed when the notebook is parsed, and adds that this is likely to become contextual. To an HTML parser the indentation inside a `script` element is text like any other, so the four spaces are a rule the format lays over HTML. The same page explains the choice of HTML: editors and diffs already handle it. [^2]: The routing is by second extension, and the [list](https://github.com/observablehq/framework/blob/main/docs/data-loaders.md) runs from `.js` and `.ts` through Python, R, Rust, Go, Java, Julia, PHP, and shell to `.exe` for an arbitrary executable. The documentation adds that the first extension does not affect the generated snapshot and that the loader is solely responsible for producing the expected output. The name `orders.json.js` is a promise about the output that only its author keeps. [^3]: The [files documentation](https://github.com/observablehq/framework/blob/main/docs/files.md) says `FileAttachment` can only be passed a static string literal, and that constructing a dynamic path is invalid syntax. The reason is given in the next sentence: static analysis is used to invoke data loaders at build time. A function that refuses a variable as its argument is unusual, and here it is the mechanism by which a build knows what to build before any code has run. [^4]: The [configuration reference](https://github.com/observablehq/framework/blob/main/docs/config.md) gives `root` a default of `src` and notes that before version 1.7.0 the default was `docs`. A default that moved once is why the build warns when the option is unstated, and why a project that states it will read the same way after the next move.