Accessing Airtable's Pagination Using Python's Requests Library

[aaron_altamura]

Hey Guys,

How can I access Airtable’s pagination by including a given offset in either the url or query?

My attempt below at including in the url has failed with the error “KeyError: ‘offset’”.

I’ve been able to access the 1st offset per the function at_page(). For simplicity’s sake, I took the result from the function and hardcoded it to the variable offset_01.

Thank you!

import requests
import pandas as pd

offset_01 = 'itrCG3VDp2c34x1j1/recKTJGYMICe13iA8'
url = 'https://api.airtable.com/v0/PRIVATETABLEKEY/accounts' + \
    '&offset=' + offset_01
headers = {'Authorization': 'Bearer PRIVATEAPIKEY'}


# Calling API
response = requests.get(
    url,
    headers=headers,
)
json_response = response.json()


# Finding AT page offset

def at_page(at_json):
    df_initial = pd.DataFrame(at_json)
    at_offset = df_initial['offset'][0]

    return at_offset

offset = at_page(json_response)

PS also have an open stackoverflow:

stackoverflow.com

Accessing Airtable's Pagination Using Python's Requests Library

python-3.x, api, python-requests, airtable

asked by empty__cup on 06:54PM - 08 Jan 20 UTC

[Bill_French]

aaron_altamura:

How can I access Airtable’s pagination by including a given offset in either the url or query?

Any reason you’re not using a library for this?

I think this is sort’a what you want -

def get_iter(self, **options):
“”"
Record Retriever Iterator

    Returns iterator with lists in batches according to pageSize.
    To get all records at once use :any:`get_all`

    >>> for page in airtable.get_iter():
    ...     for record in page:
    ...         print(record)
    [{'fields': ... }, ...]

    Keyword Args:
        max_records (``int``, optional): The maximum total number of
            records that will be returned. See :any:`MaxRecordsParam`
        view (``str``, optional): The name or ID of a view.
            See :any:`ViewParam`.
        page_size (``int``, optional ): The number of records returned
            in each request. Must be less than or equal to 100.
            Default is 100. See :any:`PageSizeParam`.
        fields (``str``, ``list``, optional): Name of field or fields to
            be retrieved. Default is all fields. See :any:`FieldsParam`.
        sort (``list``, optional): List of fields to sort by.
            Default order is ascending. See :any:`SortParam`.
        formula (``str``, optional): Airtable formula.
            See :any:`FormulaParam`.

    Returns:
        iterator (``list``): List of Records, grouped by pageSize

    """
    offset = None
    while True:
        data = self._get(self.url_table, offset=offset, **options)
        records = data.get("records", [])
        time.sleep(self.API_LIMIT)
        yield records
        offset = data.get("offset")
        if not offset:
            break

[aaron_altamura]

Thank you so much! Will review this and properly reply shortly.

I chose not to use the wrapper as I’m concerned it’ll lose support and get stale. It’s not supported by any legit orgs. There’s also a lot of stale issues open, which can speak to a small community as it is.

Requests won’t have that issue.

[Bill_French]

aaron_altamura:

as I’m concerned it’ll lose support and get stale

Yep - I fear this as well, but the code itself might provide you with some insights that you can apply to your own code.

I think there may be another one out there too with more support.

[aaron_altamura · Accepted solution]

Hi Bill,

Funny enough, your discussion with @Bastiaan_Bijl helped me find how to query within the requests library:

Offset value invalid; output does not match API documentation API

Hello, I am trying to build a query in Power Query/Power BI to load an Airtable table into the data model. I have found some working code which uses the offset parameter returned by the API to deal with the pagination. However, the offset value returned does not match the API documentation and gives an error when I try to parse it. According to the API docs, the offset should look like this: "offset": "reco1XxXxXxXxXxXxXx" However, what I get is this: "offset":"itrZXxXxXxXxXxXxXxXx/rec5XxX…

It’s the params={‘offset’: ‘itrXXXXXX/recXXXXXX’} query that allowed me to access the next 100 records.

# Calling API
response = requests.get(
    url,
    headers=headers,
    params={'offset': 'itrXXXXXX/recXXXXXX'}
)
json_response = response.json()

Thank you for your contributions and looking forward to continue to support the community as well :slightly_smiling_face: