DBX Performance Guide

One of the key benefits of using the Dropbox API is to be able to programmatically execute actions in bulk. If you issue calls in bulk or at high scale, you’ll need to consider possible performance implications of doing so.

These considerations include API rate limits, optimal/efficient use of the API, and absolute scale. This guide will help you keep your app running smoothly with the Dropbox API.

The Dropbox API enforces rate limits on the number of API calls issued over a period of time on a per-authorization basis

• For user-links, rate limits apply per user who has linked your app. Multiple apps that a user may have linked don’t contribute to each other’s rate limiting.
• For team-links, rate limits apply per team when calling Business Endpoints. If your team-linked app is calling User Endpoints on behalf of team members with Dropbox-API-Select-User, the rate limits apply per team member.

If your application experiences a rate limit, the API call will return an HTTP 429 error, returning the reason of too_many_requests (in JSON, or plaintext per the Content-Type HTTP header).

Rate limited responses always include a Retry-After header that provides the limit in seconds that your app would need to wait before retrying to ensure a successful response.

Rate limited requests themselves also count towards rate limits, and thus rapid retry loops without pause or respecting this header will be counter-productive.

While Dropbox does not publish exact rate limits, these limits are not designed to inhibit normal applications. Your application should handle 429 errors as described. If your application receives excessive rate limit responses, it may suggest inefficient call patterns that should be optimized.

Data Transport Limit

Also, note that certain Dropbox Business plans are also subject to a special kind of rate limiting for upload calls. For more information, see the Data transport limit page.

Too Many Writes

When sending files to a destination, your application may receive a similar HTTP 429 error with a different reason code: too_many_write_operations. This suggests that there are too many writes occurring on the destination, which may be due to your application, but may also be due to other users or applications operating on the space at the same time.

If your application is uploading singular files as users interact with them, encountering this will be uncommon. The exception returns a Retry-After of zero, indicating that your application should simply retry the upload.

However, if your application is attempting to write a number of files the same space in parallel threads, you application may cause frequent 429 too_many_write_operations, which can be solved with correct batching.

Namespaces

To optimize for this scenario, we first must understand what constitutes a unique space. In Dropbox, these are referred to as namespaces.

Each user’s private Dropbox folder maps to a root namespace. Shared folders are also mapped to their own namespaces, as are team folders in Dropbox Business and Dropbox Enterprise. Namespaces are also where we set ownership and access permissions. A user has the same access type for all of the files and folders in a namespace.

Namespace Locks

Write operations on a file within a namespace first acquire a lock on that namespace and release the lock when the operation has completed in order to ensure consistency. While this process is generally quick, enough parallel threads writing to the namespace may create lock contention, resulting in the 429 too_many_write_operations exception.

Batch Upload

Dropbox enables applications to write in parallel to the same namespace to avoid this scenario using files/upload_session endpoints.

Each file upload to Dropbox consists of the following stages: appending the byte contents of the file to an upload buffer on the Dropbox server, obtaining a namespace lock, and then committing those bytes as a file into a target namespace. The /files/upload endpoint does this atomically, whereas upload session decouples these steps.

Thus, by using upload sessions your app may send files to dropbox in parallel, but only issue one commit at the end, eliminating contention. The general approach to do so is as follows:

1. Group files that are being uploaded into one or more batches. A batch can contain up to 1,000 entries. For each file, call /files/upload_session/start, or for each batch, call /files/upload_session/start_batch to start an upload session for each file. If uploading large files, optionally set "session_type": "concurrent" to be able to upload different pieces of each single file in parallel as well.
   
2. For each file in a batch, in parallel, call /files/upload_session/append_v2 as needed to upload the full contents of the file over multiple requests. Be sure to specify "close": true on the last append (or on /files/upload_session/start, if the full contents are uploaded in one call).
3. Call /files/upload_session/finish_batch once for the entire batch to commit the uploaded files. Note that /files/upload_session/finish_batch may complete asynchronously and return an async_job_id that can be used with /files/upload_session/finish_batch/check.

While your batches can span across multiple namespaces, it is more efficient (and reduces contention risk) to keep them to a singular namespace.

Large Files

The /files/upload endpoint is designed to work with files that are under 150 MBs. Files over 150 MBs in size should be uploaded in chunks using the methodology described above using the /files/upload_session/start[_batch], /files/upload_session/append_v2, and /files/upload_session/finish[_batch] calls.

• Consider uploading chunks in multiples of 4 MBs.
• Consider sending data only with /files/upload_session/append_v2 calls so that /files/upload_session/start and /files/upload_session/finish requests are smaller, faster, and easier to retry.

Code sample

Check out this code sample that takes advantage of these features to upload file data as quickly as possible.

The API allows you to add and manage content rapidly and at scale that users would not be able to perform in the Dropbox desktop, mobile, or web client.

If you find yourself automating the creation of complex shared folder structures at scale in background or automated workflows, you should consult our large deployments document for scale guidelines.

When building team-linked apps, using a singular account as a “service account” to own large amounts of content is generally not recommended. Instead, use the Dropbox-API-Select-User to upload content to the logical owning team member.

Read Batch Operators

One of the more common causes of API rate limits—as well as extra HTTP round trip time—is traversing a folder’s metadata with /files/list_folder[/continue], then performing operations on each entry.

Many of these operations, such as /files/get_thumbnail, have equivalent batch operators (/files/get_thumbnail_batch) that enable you to pass an array of ids, rather than getting one at a time. Doing so can dramatically lower number of API calls and improve your performance.

Previews and Thumbnails

User files on Dropbox may be large. For display preview purposes, downloading the file from Dropbox may be unnecessary. When possible, try to leverage /files/get_thumbnail and /files/get_preview for this purpose to enhance performance.

Caching

Dropbox file metadata from /files/list_folder[/continue] includes a variety of fields, such as last modification time (server_modified), revision id (rev), and content hash (content_hash), that can help you determine whether it’s necessary to [re]upload or download the content to or from Dropbox. For more information, visit our File Access Guide.

Polling

Inefficient API calling resulting in rate limiting may be due to your application's desire to stay up to date, in near-realtime with excessive polling of Dropbox endpoints. For best practices, see our Detecting Changes guide.

Peering

If you’re building a large-scale SaaS application and are looking to further optimize at the network level to Dropbox, peering is available.