How job pacing works

A background job can write millions of items. Sent as fast as possible, those writes would take the table’s whole capacity, and every other writer on the table would start to see throttling. So each job paces itself. It aims at a ceiling below the table’s capacity, slows down the moment DynamoDB throttles it, and shares that ceiling with any other job on the same table.

The ceiling is 70% of the table’s write capacity

A job writes at most 70% of the table’s write capacity per second. Dynomatic takes that capacity from what DescribeTable reports:

• On a provisioned table, it is the provisioned write capacity.
• On an on-demand table, it is the table’s warm write throughput. When the table sets a maximum on-demand write throughput, the lower of the two applies.
• When the table reports neither, Dynomatic assumes 4,000 writes a second, the warm throughput of a new on-demand table.

A provisioned table with 1,000 WCU therefore gets a ceiling of 700 WCU a second. The other 30% stays free for the writers that are not the job. A running job reads the table’s capacity again from time to time, every minute for a commit, a bulk edit, or an import, so a capacity change moves the ceiling too.

The Background jobs panel shows the ceiling on each running row as N of M WCU/s, where N is the current write rate. The cost line in the Bulk edit dialog uses the same ceiling for its time estimate.

A throttle halves the job’s speed

The ceiling is a limit, not a target the job holds no matter what. DynamoDB can throttle a job below its ceiling, for example when its writes land on one hot partition. The job therefore also limits how many requests it has in flight at once.

That limit starts small and grows while requests succeed. When DynamoDB throttles a request, the limit halves, and the throttled item waits a random time before it goes again. The longest possible wait starts at 50 milliseconds and doubles with each retry of the same item, up to 5 seconds. The limit also stops growing while DynamoDB answers much more slowly than it did when the job started.

The result is a job that settles just under whatever rate the table can take at that moment. It speeds up again once the throttling stops.

Jobs on one table share one budget

Two jobs that write to the same table do not each get their own ceiling. They join one shared budget, at the lower of their two ceilings. Together they never write faster than one job could alone. A commit job, a bulk edit, an import, and an embedding backfill on the same table all split the same budget.

Sharing is the safe choice. With a ceiling per job, a second job would double the load on the table, and a third would push it past its capacity.

Reads stay under 70% of read capacity

A bulk edit of every row a filter matches has to find those rows first, so its job runs the tab’s Query or Scan itself and reads only the key attributes. Those reads also keep to 70% of the read capacity they draw on.

• A Query or Scan on the table uses the table’s read capacity.
• A Query on a global secondary index uses that index’s own read capacity.

The capacity comes from the same places as the write capacity. When nothing is reported, Dynomatic assumes 12,000 reads a second. Jobs that read the same table or index share one read budget, at the lowest rate any of them allows.

Repeated failures pause or stop a job

Throttles slow a job down. Other failures can stop it. The first two checks below apply to the jobs the panel calls Bulk write, which are commits, bulk edits, and imports from File or editor.

• If more than one in five of the job’s last 500 results failed for a reason other than throttling, the job pauses as Paused: too many errors.
• If DynamoDB rejects each of the first 50 items as invalid, the job fails with DynamoDB rejected the first 50 items.
• If credentials expire or access is denied, the job pauses with that reason.

A pause keeps the job’s place, so you fix the cause and resume instead of starting again.

A job keeps its place across quits and crashes

A job records the result of each item as it goes. When you quit, Dynomatic pauses each running job before it exits. After a crash, a job that was running reads Interrupted. Either way, Resume continues from the last place the job recorded.

Nothing resumes on its own after a restart, by design. While the app was closed, credentials may have expired, or the workspace may now point at another account or another table with the same name. So Resume first resolves the workspace and its credentials again, then checks that the table is the same table and that its keys have not changed. A job that fails one of those checks stays where it was, and the panel says why.