How to Scale Data Infrastructure from a Few Agents to Many

Most agent projects start small. A team connects three agents to a production database, the queries return in milliseconds, and the architecture looks finished.

The same architecture fails at three hundred agents. Databases run out of connections. Warehouse bills grow with every agent added. The same scan runs hundreds of times an hour, because no agent knows what another agent already read.

The failure is not gradual. Each limit holds until the fleet crosses it, then the whole tier degrades at the same time.

This guide covers what changes between a few agents and many, and which limit breaks first. It then covers how to size the data tier and hold source load flat as the fleet grows.

What Changes Between a Few Agents and Many?

Query volume grows faster than the fleet

One agent task is not one query. A tool-calling loop lists tables, reads a schema, runs two or three exploratory queries, then runs the query it needs. The count varies by task and by toolset. This guide uses twenty queries per task as an illustrative figure, and the sizing section below replaces it with a measurement.

Total query volume is the product of three numbers: agent count, tasks per agent, and queries per task. Adding agents multiplies the first term. Giving agents harder work multiplies the third.

Agent traffic has no daily curve

Human traffic rises in the morning and falls at night. Capacity plans depend on that shape. Agent traffic does not have it.

A scheduled fleet starts every agent in the same second. An event-driven fleet starts one agent per event, so the arrival rate follows the event source. Both produce bursts that a replica sized for average load cannot absorb.

Agents in a fleet read the same data

Agents in one fleet usually work on one domain. They read the same product table, the same account records, and the same recent events.

Ten agents that answer questions about orders run the same scan ten times. Each scan is correct. Nine of them are repeat work.

Each agent becomes a client of every source

A direct connection design creates one client per agent per source. Two hundred agents across six sources produce 1,200 connection pools.

Each pool holds idle connections, its own credential, and its own retry behavior. That count grows as a product, not a sum.

What Breaks First?

The limits below are ordered by ceiling. Each one has less headroom than the one after it, so a growing fleet reaches them in roughly this sequence.

1. Connection limits on the source

PostgreSQL and MySQL cap concurrent connections, and each connection reserves memory. A default PostgreSQL install allows 100 connections, and applications already hold most of them.

An agent fleet that opens its own connections competes with production traffic for that budget. The database refuses new connections before it runs out of CPU.

2. Source latency under concurrent scans

An operational database answers point reads quickly. Agents send analytical queries instead: aggregates, joins across large tables, and scans without a selective filter.

These queries hold their connections longer and compete for the same buffer pool. Application latency rises at the same time, so the fleet degrades the system it depends on.

3. API rate limits

Many agent datasets arrive through a REST or GraphQL service with a request quota. That quota was sized for application traffic.

A fleet exhausts it quickly, and the service returns 429 responses. Agents then retry, which adds load to a service that is already refusing requests.

4. Per-query billing

Warehouses bill per query, per scanned byte, or per second of warehouse time. Agent fan-out multiplies every one of those units.

This limit has no error message. The cost signal arrives one month late, so teams find the problem on a bill rather than on a dashboard.

5. Attribution and control

A shared service account hides which agent ran which query. The database records one client for the whole fleet.

Without per-agent identity, a team cannot cap one agent, bill one team, or explain one incident. This limit stays invisible until an incident needs it.

How Do You Size the Data Tier?

Start with a measurement, not an estimate. Instrument one agent and count its queries per task across a full day of real work.

The capacity formula

peak queries per second =
    agents x tasks per agent per hour x queries per task / 3600 x burst factor

Work an example with 500 agents, 6 tasks per agent per hour, 20 queries per task, and a burst factor of 10. The fleet runs 60,000 queries per hour, which averages 16.7 queries per second and peaks near 167.

Translate the rate into concurrency

Multiply the peak rate by the average query duration. At 300 milliseconds per query, 167 queries per second need about 50 concurrent connections.

Compare that number against the connection budget of the source. Fifty concurrent analytical queries against a production PostgreSQL instance is a capacity problem, not a tuning problem.

Size for the burst, not the average

The burst factor dominates the result, and teams usually guess it. Measure it instead: record query arrival times for a week, then compare the busiest second against the mean.

Which Scaling Patterns Apply?

Five patterns cover most fleets. Teams usually combine two of them.

Direct connections

Each agent opens its own connections to each source. It is the fastest design to build and the first to break. Use it for a pilot, not for a fleet.

Connection pooler

A pooler such as PgBouncer multiplexes many clients onto few server connections. It solves the connection count and nothing else.

Query volume, repeated scans, and per-query billing all keep growing with the fleet. A pooler delays the first failure; it does not change the growth curve.

Read replicas

A replica moves agent reads off the primary database. The application keeps its capacity, and the agents get their own.

Replicas scale in fixed steps, not continuously. Each one is a full copy of the database with its own cost, and a query that spans two sources still needs a separate answer.

Shared data plane with acceleration

One query tier sits between the fleet and the sources. It holds one connection pool per source and a materialized copy of the working set.

Agents query the tier. The tier reads each source on a refresh schedule. Source load then tracks that schedule instead of the agent count, which is the property that makes fleet size stop mattering.

Per-agent runtime

Each agent gets its own runtime process, usually as a sidecar. Isolation is strong, and one agent cannot affect another.

The cost is operational: many processes to deploy, patch, and observe. Run per-agent runtimes over a shared accelerated tier, so the isolation does not recreate the load problem.

How Does a Shared Data Plane Hold Source Load Flat?

The property comes from where each read lands. Agent queries reach a local copy. The refresh job reaches the source.

Adding 100 agents adds queries against the local copy. It adds no queries against the source database. Source load then depends on the refresh interval and the dataset size, both of which the team sets.

Freshness becomes an explicit setting rather than a side effect. A dataset that must be current within seconds refreshes from a change stream, and change data capture keeps it current without a full reload. A reference table refreshes hourly.

What a shared plane does not solve

A shared tier is a shared failure domain. One agent with an unbounded scan can consume its capacity, and one bad refresh serves stale data to every agent at the same time.

Both need explicit controls: per-agent quotas, a limit on concurrent queries, and a freshness check per dataset. The isolation that the direct design gave for free must now be built. The per-agent isolation guide covers that boundary model.

How Do You Move from a Few Agents to Many?

Change one thing per step, and measure the result before the next step.

1. Measure query fan-out. Count queries per task for one agent over a full day.
2. Record the burst factor. Compare the busiest second against the average arrival rate.
3. Put a shared query tier in front of the sources. Keep every source authoritative.
4. Give each agent its own identity in that tier. Remove shared service accounts.
5. Accelerate the highest-volume dataset. Set its refresh interval from a freshness requirement.
6. Add per-agent quotas and query logs. Attribute every query to one agent.
7. Raise the agent count in steps. Watch source connections and latency after each step.

Step 3 is the one that changes the growth curve. The earlier steps show where the fleet stands. The later steps stop one agent from consuming the tier.

How Do You Hold Cost Flat as Agents Multiply?

Agent data cost has three parts: reads against the source, data transfer, and the runtime the fleet uses. Fleet growth multiplies the first two.

Convert a per-query cost into a fixed cost

Warehouse billing scales with query count and scanned bytes. Acceleration replaces many billed source queries with one scheduled refresh.

The refresh cost depends on data volume and refresh interval. It does not depend on how many agents read the result.

Cap the fan-out per task

A cheaper tier does not fix an agent that runs 200 queries to answer one question. Give agents fewer, wider tables and precomputed aggregates, so one query returns what three used to.

Every round trip removed cuts cost and latency together. This work belongs in the tool and schema design, not in the data tier.

Bill by agent

Per-agent attribution turns cost into something a team can act on. Without it, one expensive agent looks like a platform cost increase that nobody owns.

Advanced Topics

Query deduplication across agents

Agents in a fleet ask similar questions in the same minutes. A result cache in front of the query tier collapses those repeats into one execution.

Key the cache on the query plan rather than the raw SQL string. A plan-based key matches queries that are semantically equivalent but written differently. Filter values and the authorization scope stay part of the key, so two agents reading different tenants never share an entry. The cache invalidation guide covers the expiry rules this needs.

Admission control and noisy neighbors

A shared tier needs a limit on concurrent queries per agent and a maximum query duration. Without both, one agent with an unbounded scan consumes the tier for everyone.

Queue the excess rather than refusing it, and return a clear error when the queue fills. An explicit error lets an agent retry with a narrower query. A timeout gives it nothing to act on.

Cold start for a new agent

A new agent joins a fleet and reads an accelerated dataset that already exists. Its first query does not wait for a load from the source.

This is the operational difference between a shared working set and a per-agent cache. A per-agent cache repeats the initial load for every new agent, which makes startup the slowest part of scaling out.

Per-agent attribution

Attribution needs an identity that reaches the query tier, not one shared connection string. Short-lived credentials per agent carry that identity into the query log.

The log then answers three questions: which agent ran the query, which datasets it touched, and what the query cost. Attribution added after an incident cannot answer those questions for the incident.

Refresh policy per dataset

A fleet has mixed freshness needs. One policy for every dataset either wastes refresh capacity or serves stale data.

Set each interval from the decision the data supports. Order status for a support agent needs seconds. The product catalog it reads in the same task needs hours.

Scaling Agent Data Infrastructure with Spice

Spice runs as the shared data plane described above. It connects to 40+ data sources including PostgreSQL, MySQL, MongoDB, DynamoDB, Snowflake, Databricks, S3, Iceberg, and Delta Lake, and exposes them through one SQL endpoint.

SQL federation and acceleration route each query to the system that holds the data, so agents reach every source through one connection and one dialect. The client count stops growing with the product of agents and sources.

Data lake acceleration materializes the working set in memory or on disk with Cayenne, the Spice columnar accelerator. Each dataset carries its own refresh policy, so agent reads land on the local copy and source load tracks the refresh schedule. Real-time change data capture keeps that copy current with PostgreSQL, MySQL, MongoDB, and DynamoDB.

The MCP server gateway exposes datasets and tools to agents through one governed endpoint, so a new agent joins the fleet without a new integration. Secure AI agents covers the per-agent credentials, scoped access, and audit trail that attribution and quotas depend on.

Spice runs from a single binary or container next to the agents, in a data center, in any cloud, or on Kubernetes. Distributed query adds multi-node execution for the point where one node cannot hold the working set or finish the scan.