title: 'GitHub Data Connector Deployment Guide' sidebar_label: 'Deployment Guide' description: 'Operating guide for the GitHub data connector in production: PATs, rate limits, pagination, and observability.' sidebar_position: 10 pagination_prev: null pagination_next: null tags:
Production operating guide for the GitHub data connector covering authentication, GitHub API rate limits, and operational tuning.
The GitHub connector uses the GitHub REST and GraphQL APIs with a personal access token (PAT) or GitHub App installation token.
| Parameter | Description |
|---|---|
github_token | PAT or installation token. Use ${secrets:...} to resolve from a secret store. |
Tokens must be sourced from a secret store (opens in a new tab) in production. Scope the PAT to the minimum required permissions:
repo scope.repo (private) or public_repo (public).read:org.For long-running deployments, prefer GitHub App tokens (installation tokens) over user PATs — they have higher rate limits (15,000/hr vs 5,000/hr per authenticated user) and are not tied to a specific user account.
GitHub's REST API rate limits:
| Auth mode | Limit |
|---|---|
| Unauthenticated | 60 requests/hr per IP |
| Authenticated (PAT) | 5,000 requests/hr |
| GitHub App installation | 15,000 requests/hr |
| Enterprise Server (typical) | Configurable |
The connector respects GitHub's Retry-After and X-RateLimit-Reset headers and backs off accordingly. It stops issuing requests while the primary limit's remaining budget is at or below 10% of the limit, so other users of the same token keep a reserve, and resumes at the next reset window. GitHub meters the REST quota per token, so every dataset that uses the same token or GitHub App installation shares one REST rate limiter.
GraphQL is paced separately. GitHub's GraphQL secondary rate limit (opens in a new tab) is 2,000 points per minute, and a non-mutation GraphQL request costs 1 point regardless of how many nodes or nested connections it asks for. The connector paces itself to 90% of that — a sustained 1,800 requests per minute — and every GraphQL-backed table on one authentication context shares a single limiter, so adding datasets divides that budget rather than multiplying it. Because every query costs the same 1 point, the shared limiter is first-come-first-served across tables: one large scan cannot starve the others. A secondary-limit 403 is retried on the same page using the response's retry-after.
GraphQL CPU time is not estimated locally: HTTP duration is not GitHub's CPU accounting, and budgeting against it would serialize scans GitHub would still accept.
Page width is chosen per table, not fixed at GitHub's 100-item maximum. GitHub enforces a per-request compute budget (GraphQL API resource limits announcement (opens in a new tab)) that is separate from — and reached long before — the 500,000-node ceiling (opens in a new tab), and a page wide enough to exceed it is rejected outright with Resource limits for this query exceeded, every node in the page returned as null. Tables whose rows expand into many nested connections therefore request narrower pages: pulls is requested 25 at a time in both comment modes, while issues and milestones still use 100.
Nested connections are paged too. reviews, review_threads and release_assets exist only underneath a parent in GitHub's GraphQL schema, and GitHub caps a nested connection page at 100 nodes and will not page it from inside the parent query. The connector fetches the remainder with follow-up requests addressed at the parent, so a pull request with more than 100 reviews is read completely rather than truncated — a truncated nested connection would drop whole rows, and a short COUNT(*) gives no sign it is short. A follow-up page that fails is retried on its own, without fetching the parent page again; a gateway error retries it with a smaller page. When a nested connection cannot be continued, the scan fails and names the parent rather than returning a partial set. This happens when a follow-up page repeats the previous cursor, when a page reports more nodes but returns no cursor, or when the last page leaves the connection with fewer nodes than its totalCount.
Datasets backed by high-volume endpoints (e.g., repos.commits on a monorepo) may require many hours to initially hydrate. Use incremental acceleration with a since filter where possible.
:::warning[pulls scan ceilings differ by query mode]
In github_query_mode: auto, the connector fails with Maximum pagination iterations (1000) exceeded after 1,000 pagination iterations (1,001 total page fetches counting the initial page), rather than silently truncating. At 25 rows per page, that bounds a pulls dataset at 25 x 1001 = 25,025 pull requests per refresh. In github_query_mode: search, GitHub Search has its own 1,000-result limit (opens in a new tab), which is the effective cap for a single query.
The page was wider in earlier releases, putting the arithmetic ceiling at 100,100 — but on a repository large enough for that to bind, the wider page was rejected by the compute budget and returned no rows at all. The number of pull requests actually reachable went up, not down. :::
Tables read through the GitHub GraphQL API, such as pulls, issues, commits, and stargazers, use the GraphQL connector's retry policy. Each page, and each GraphQL request sent when the dataset loads to check the repository or organization and infer its schema, is retried up to 5 times with Fibonacci backoff, honoring Retry-After. An empty HTTP 200, which the GitHub GraphQL API returns intermittently, is retried on a new connection, and a 502, a 504, or a GitHub backend-timeout error is retried with a smaller page. See Retry Behavior (opens in a new tab) in the GraphQL deployment guide for the full list of retried responses.
The files table and the workflow tables read through the GitHub REST API, as do the ref lookups for commits. A REST request (opens in a new tab) retries connection errors and timeouts with Fibonacci backoff, up to 5 times for the files tree request and 3 times for other requests. An HTTP error status from the REST API, including 5xx and 429, is not retried.
Permanent errors (401 Unauthorized, 404 Not Found, 422 Validation Failed) surface immediately.
github.com; lower for GitHub Enterprise Server on the same network.The GitHub connector does not register connector-specific dataset-level instruments in the current release. Monitor via:
query_duration_ms, query_returned_rows, query_failures) from runtime.metrics./settings/tokens for token-level quota tracking.See Component Metrics (opens in a new tab) for general configuration.
GitHub API calls participate in task history (opens in a new tab) through the HTTP client's span. Each page fetch is a child of the enclosing sql_query or acceleration_refresh task.
| Symptom | Likely cause | Resolution |
|---|---|---|
401 Bad credentials | PAT expired / revoked / wrong value. | Rotate the PAT; update the secret store. |
403 rate limit exceeded | Primary hourly rate limit hit. | Increase refresh interval; switch to GitHub App auth for higher quota; use incremental refresh with since. |
403 Secondary rate limit | Burst of concurrent requests tripped abuse detection. | Reduce concurrent refresh; connector will back off automatically. |
404 Not Found on a private repo | Token lacks repo scope. | Regenerate PAT with repo scope. |
| Very slow initial hydration | Large dataset + strict rate limit. | Run first refresh off-peak; use since/updated_since for incremental refreshes. |