How to ship a DataFrame-first Python client for your analysts#

Every notebook in the analytics team opens with the same forty lines: a requests loop over your list endpoint, a json_normalize, and a block of astype calls that each notebook copies a little differently. One notebook sums an id column. Another reads score as a float because one page carried a null, and your API gets blamed for both.

What you get

You will end up with a generated Python package where ad.applicants() returns every page as one typed frame, plus a test that pins its dtypes to your OpenAPI schema. This is for you if you own an API whose analysts keep rewriting its client in notebooks.

Short answer

Add sdkgen’s py-data target beside the py target, add the paging feature, and regenerate to get a Python package that returns typed pandas frames. Its data() function returns a client with one method per entity: ad.applicants() drains every page into a frame typed from the model. Dates stay strings until you pass parse_dates. A helper over openapi-python-client reaches the same frame in about thirty lines you own.

You will need

Node 22 or later for the generator, Python 3.12 or later, and the OpenAPI 3 document for your API. The document on this page, api.json, describes a recruiting API with applicants and jobs behind a cursor-paginated envelope of data, next_cursor, and has_more. Verified 2026-10-20 against Node 22.22.0, Python 3.13.16, @voxgig/sdkgen 4.35.0, @voxgig/create-sdkgen 0.30.8, pandas 3.0.6, openapi-python-client 0.29.1, httpx 0.28.1, and attrs 26.1.0. Both routes run against mock_api.py, a local stand-in for the API, so nothing here calls a network service. dlt is described from its documentation and was not run.

Approaches compared

Voxgig maintains sdkgen. This page compares it with openapi-python-client and dlt.

The hand-written helper costs you the pagination loop and the dtype table, and every schema change lands on whoever maintains them. dlt puts a pipeline and a DuckDB file between the API and the notebook. That is the right shape for a nightly load and the wrong one for a cell that wants the current table. The py-data target costs a second generated package and a pandas-only output. Its flattening and typing rules come from the model rather than from you, which is the saving and also the loss of control.

Add the data target beside the Python target

The data package wraps the Python SDK rather than replacing it, so the two targets are generated from one model. generate.sh is the whole build, run once with network access; its output is committed under generated/ so the rest of the page runs offline.

npx --yes @voxgig/create-sdkgen@0.30.8 hirewire --def api.json --target py --target py-data
(cd hirewire-sdk/.sdk && npm run add-feature paging && npm run generate)

Two flags name two targets, and the scaffold records them in the model, which is the other way to add one: list it beside py and regenerate.

@"./py.aontu"
@"./py-data.aontu"

The order of the two steps matters. The paging feature is not in a fresh model until you add it, and the frame methods drain whatever paging state that feature records. A model without it yields one page per call. The target also refuses to build alone: without py in the same model the generate step stops with py-data requires the py target in the same SDK.

The entity model is where the column types come from. For applicant, score is $INTEGER,¹ and applied_at is a $STRING whose format the model records as fo: "date-time":

    applied_at: {
      fo: "date-time"
      h: "Applied At"
      n: "applied_at"
      r: true
      t: "`$STRING`"
    }

The data target maps $INTEGER to pandas Int64, $BOOLEAN to boolean, and every $STRING to string, format included, so date parsing is left to the caller. The generated hirewire_data package holds the methods, a README.md with a column table per entity, an AGENTS.md, and notebooks/quickstart.ipynb, all committed here.

Pull every page into one frame

frames_demo.py builds the client the way a notebook would, with the credential made explicit and the base URL pointed at the stand-in. The paging feature is switched on through the options the SDK constructor takes.

ad = data(token="test-key", base_url=mock_api.base_url(api),
          feature={"paging": {"active": True, "limit": 2}})

df = ad.applicants(quiet=True)

PYTHONPATH=vendor:generated/py:generated/py-data python3 frames_demo.py

rows: 5, from 3 requests
  GET /v1/applicants?limit=2&page=1
  GET /v1/applicants?cursor=c2&limit=2
  GET /v1/applicants?cursor=c4&limit=2

       id      stage  score  remote                 applied_at location.city
0  app_01  interview     82    True       2026-03-02T07:30:00Z         Lagos
1  app_02  screening   <NA>   False  2026-03-02T08:00:00+02:00       Utrecht
2  app_03    applied     67    True       2026-03-01T22:15:00Z        Taipei
3  app_04      offer     91   False       2026-02-27T16:45:00Z         Haifa
4  app_05   rejected   <NA>    True       2026-02-26T09:05:00Z       Nairobi

dtypes
applied_at           string
email                string
id                   string
job_id               string
name                 string
remote              boolean
score                 Int64
stage                string
location.city           str
location.country        str

applied_at with parse_dates: datetime64[us, UTC]

Five rows arrived in three requests of two. The first request carries page=1, because the paging feature assumes page numbers until a response hands it a cursor; the API ignores the parameter and answers the first page. Each later request carries the next_cursor the previous response returned, until has_more is false.

Read the dtypes block against the schema. score is Int64,² so the two nulls print as <NA> and the column stays integral. remote is boolean, and every declared string is string. The nested location object was flattened one level into location.city and location.country, the two columns pandas typed on its own, since the model declares location as one object field. applied_at is a string column until the last line, where parse_dates turns it into a UTC datetime.

Write the same frame by hand over the generated client

The comparison client was generated from the same api.json into generated/opc. It gives you a Client, one module per operation, and attrs models with to_dict(). The frame is yours.

def applicants_frame(client: Client, page_size: int = 2) -> pd.DataFrame:
    """Every applicant as one row. The loop, the unwrapping and the typing are yours."""
    rows: list[dict] = []
    cursor = UNSET
    seen: set[str] = set()
    while True:
        page = list_applicants.sync(client=client, cursor=cursor, limit=page_size)
        if page is None:
            raise RuntimeError("list_applicants answered with an undocumented status")
        # to_dict() drops fields the API did not send, so a column can be missing
        # from one page and present in the next.
        rows.extend(applicant.to_dict() for applicant in page.data)
        if not page.has_more or page.next_cursor in (None, UNSET):
            break
        if page.next_cursor in seen:
            raise RuntimeError(f"pagination repeated cursor {page.next_cursor!r}")
        seen.add(page.next_cursor)
        cursor = page.next_cursor
    return pd.json_normalize(rows, sep=".")


def typed(df: pd.DataFrame) -> pd.DataFrame:
    """The dtype work py-data reads off the model, done here by hand and by name."""
    out = df.convert_dtypes()
    # Without format="ISO8601" pandas takes the format from the first value and
    # raises on any value that carries fractional seconds when that one did not.
    out["applied_at"] = pd.to_datetime(out["applied_at"], utc=True, format="ISO8601")
    return out

PYTHONPATH=vendor:generated/opc python3 to_dataframe.py

rows: 5, from 3 requests

                 json_normalize        after typed()
id                          str               string
job_id                      str               string
name                        str               string
stage                       str               string
applied_at                  str  datetime64[us, UTC]
email                       str               string
score                   float64                Int64
remote                     bool              boolean
location.city               str               string
location.country            str               string

applied_at as to_dict() wrote it: 2026-03-02T07:30:00+00:00

The left column is what json_normalize infers on its own: score has become float64 because one page carried a null, and the dates are text. The right column is after convert_dtypes and one to_datetime call, which matches the generated method with parse_dates on every declared column. The difference is who names the columns: typed() knows applied_at by hand, and a renamed field in the schema breaks it without a test to say so. The last line shows the models round-tripping the timestamp, so Z on the wire comes back as +00:00.³

The helper also stops at a repeated cursor, which would otherwise keep the cell running forever.

Check it worked

test_frames.py runs both routes against the mock API. The test that guards the whole point compares the frame’s dtypes with a table built from the OpenAPI schema. A second one asserts that a filter matching nothing still returns the declared columns with the declared dtypes.

    def test_declared_columns_carry_the_schema_dtypes(self):
        df = self.ad.applicants(quiet=True)
        self.assertEqual({c: str(df[c].dtype) for c in SCHEMA_DTYPES}, SCHEMA_DTYPES)

PYTHONPATH=vendor:generated/py:generated/py-data:generated/opc python3 test_frames.py

test_the_helper_lands_on_the_same_rows_and_dtypes (__main__.HandWrittenHelper.test_the_helper_lands_on_the_same_rows_and_dtypes) ... ok
test_the_helper_parses_timestamps_with_and_without_fractions (__main__.HandWrittenHelper.test_the_helper_parses_timestamps_with_and_without_fractions) ... ok
test_the_helper_refuses_a_cursor_that_repeats (__main__.HandWrittenHelper.test_the_helper_refuses_a_cursor_that_repeats) ... ok
test_a_filter_that_matches_nothing_keeps_the_declared_columns (__main__.PyDataFrames.test_a_filter_that_matches_nothing_keeps_the_declared_columns) ... ok
test_a_null_integer_stays_an_integer_column (__main__.PyDataFrames.test_a_null_integer_stays_an_integer_column) ... ok
test_dates_are_strings_until_parse_dates (__main__.PyDataFrames.test_dates_are_strings_until_parse_dates) ... ok
test_declared_columns_carry_the_schema_dtypes (__main__.PyDataFrames.test_declared_columns_carry_the_schema_dtypes) ... ok
test_every_page_is_drained_into_one_frame (__main__.PyDataFrames.test_every_page_is_drained_into_one_frame) ... ok
test_one_level_of_nesting_becomes_dotted_columns (__main__.PyDataFrames.test_one_level_of_nesting_becomes_dotted_columns) ... ok

Nine tests print ok and the runner ends with OK. The paging test also counts requests at the mock, because three requests for five rows is what proves the drain.

When it goes wrong

A date column sorts wrongly and .dt raises. The method typed applied_at as string, because the data target never infers dates from a string format. pitfall.py shows the cost with two applicants who applied the same morning in different offsets.

PYTHONPATH=vendor:generated/py:generated/py-data python3 pitfall.py

1. applied_at stays a string column until you ask for dates
   dtype: string
   latest by string order: Bram de Vries  2026-03-02T08:00:00+02:00
   latest by parsed order: Ada Okafor  2026-03-02 07:30:00+00:00
   .dt on the string column: Can only use .dt accessor with datetimelike values
   parse_dates when one value has milliseconds: ['2026-03-02 07:30:00+00:00', 'NaT']

2. without the paging feature, one page comes back and nothing complains
   rows: 2, from 1 request
   the API saw: ['/v1/applicants']

Sorted as text, the +02:00 timestamp lands after the Z one, so string order names the second applicant as the most recent when the first applied ninety minutes later. Pass parse_dates=["applied_at"] only when every value has one precision, since it turns any other format into NaT, as the first block’s last line shows. Otherwise call to_datetime with utc=True and format="ISO8601", as typed() does.

The frame has exactly one page of rows. The client was built without the paging feature, so the drain made one call and returned it, which the package treats as an API with one page. Pass feature={"paging": {"active": True}} to data(), and check that the model carries the feature at all.

The generate step fails on py-data with a message about the py target. The data package imports the SDK that target writes, so add it with npm run add-target py and regenerate.

When not to do this

Do not ship sdkgen’s data target to analysts who work in Polars. pandas is the only frame library the generated package knows, declared as a hard dependency with the reason written into the target model. Those analysts are better served by the openapi-python-client helper with polars.from_dicts in place of json_normalize.

Do not generate a frame package for an API whose entities have no list operation. The methods come from list and load; an API of actions gets an empty class and an ad.frame() escape hatch.

Do not read the generated dtypes as validation. A column that will not convert to its declared type is left as the API sent it, by design, so a frame can look typed and carry one object column. The dtype test on this page is how you notice.

Do not adopt a second generated package to save one helper for one entity. The helper here is thirty lines; the package is a release, a PyPI distribution, and a sibling dependency to keep in step with the SDK. The target pays off when several entities need frames and the API keeps changing.

Do not reach for dlt to fill a notebook cell. Its rest_api source and pipeline.dataset() are built for loads that run on a schedule and keep history, and the frame arrives from the DuckDB file rather than from the API.

Related how-tos

• Describe pagination so generators can follow it

• Loop over every page of a REST collection

• Generate Python docstrings and type hints from an API spec

• Generate your first SDK with sdkgen

Last verified

Verified 2026-10-20 against Node 22.22.0, Python 3.13.16, @voxgig/sdkgen 4.35.0, @voxgig/create-sdkgen 0.30.8, pandas 3.0.6, numpy 2.5.4, requests 2.34.2, openapi-python-client 0.29.1, httpx 0.28.1, and attrs 26.1.0. Every output block is what the command preceding it printed. The scaffold and both generators ran once, in generate.sh, and their output is committed under generated/; the page’s commands read those files and run against the local stand-in API. The generated quickstart notebook was read and not executed, and dlt is described from its documentation.