How I got past Cloud Build’s 600-trigger limit

I co-architected an internal developer platform (IDP) on GCP and Kubernetes that 150 engineers across 10+ teams deploy with. Its CI was trigger-based: every deployment registered its own Cloud Build trigger. Then our dev GCP project hit Cloud Build’s limit of 600 triggers. A new deployment needs a new trigger, so a project at the limit can’t take another deployment. The usual fix for a quota problem is a quota increase, and this limit doesn’t have one. I had to change how builds start.

The constraint

Cloud Build’s quotas page lists “Number of build triggers” at 600 under its resource limits, and it describes system limits as “fixed values that can’t be changed.” The limit applies per GCP project.

The limit counts every trigger in the project, including the ones a platform didn’t create. Each environment on our platform has its own GCP project, so its deployments shared those 600 slots with whatever else was there. Our dev project is the one that hit the limit, and it has only around 300 deployments today. The rest of the count was manually created triggers, made for other reasons or from before the platform existed. Some deprecated triggers were already disabled, and that slack bought us time. It didn’t change the maths. Every new deployment still cost one trigger from the same 600, so the count kept climbing toward a number that would never move.

What I tried first

My first idea was a quota increase. Then I found out that 600 is the maximum, with no increase path. I looked at two other options before designing something new.

Sharding triggers across projects. The limit is per project, so more projects would mean more triggers. It would have taken too much migration, and each new project would only have added another 600. It also did nothing for the approval gating we needed, with rules per environment and PR summaries in the approver’s inbox.

GitHub Actions. Actions brings a different kind of cost and a different setup, and it’s one more tool for the team to learn. We already knew Cloud Build. We also wanted CI that developers never have to think about: they push, and the rest happens in the background.

One queue in front of the build endpoint. My first design for the new pipeline had a single hop. The webhook published to Pub/Sub, and a push subscription called the build endpoint. Monorepos were the problem. One push to one branch can affect many deployments, and if a single endpoint starts all of them, one failure retries the whole batch. So I added a service in the middle to split each push by deployment.

What worked

A Cloud Build trigger does two jobs. It matches a push to a repo and branch, and it starts a build with the right configuration. Our platform already knew every deployment, its repo and branch, and its build settings, so it could do both jobs itself. Cloud Build doesn’t need a trigger to run a build: its API accepts one directly. The 600 limit counts triggers, so a platform that starts its own builds bypasses it.

I designed a pipeline that receives every push, works out which deployments it affects, and starts their builds through the API.

The path of a push

1. Webhook. A GitHub App installed on each repo sends every push event to a FastAPI service, which publishes it to a Pub/Sub topic. GitHub wants a response within 10 seconds, and this service’s work doesn’t grow with the number of deployments a push affects.
2. Expansion. A push subscription delivers the event to the expander, which finds every deployment watching that repo and branch.
3. One message per deployment. The expander publishes one message per matching deployment to a second topic. Message ordering is on, with the deployment ID as the ordering key, so messages for one deployment arrive one at a time and in order, while different deployments run in parallel.
4. Approval. A second push subscription delivers each message to the build service. Each environment sets its own rules. With approvals disabled, the build starts straight away. With approvals on, the commit is marked as pending, and approvers get an email saying who committed where, with a summary from a separate code-review product. Gates can also require a PR and a minimum review score.
5. Build. The build service starts the build with a direct Cloud Build API call (builds.create), using that deployment’s settings, such as its machine type. No trigger is involved.
6. Retries. The build service rejects a message with a 4xx when a check fails, or when that deployment already has a build in progress or a release pending. Pub/Sub treats that as a failed delivery and retries about once a minute. max_delivery_attempts is 45, which gives a message about 45 minutes. Our builds typically take 20 to 25 minutes at most, so that’s roughly twice the time an in-progress build needs. After that, the message moves to a dead-letter queue, where someone investigates it and retries it by hand.

Ordering and the in-progress check together mean two pushes to the same deployment never race. In simplified pseudo-code, the two endpoints that do the work look like this:

# Simplified pseudo-code

def expander(event):
    for deployment in deployments_watching(event.repo, event.branch):
        build_requests.publish(
            {"deployment": deployment.id, "commit": event.commit},
            ordering_key=deployment.id,
        )

def build_endpoint(message):
    deployment = get_deployment(message["deployment"])

    if deployment.is_busy() or not checks_pass(message):
        return 4xx  # Pub/Sub retries this message

    if deployment.environment.needs_approval:
        mark_pending_and_email_approvers(message)
    else:
        cloud_build.create_build(deployment.build_config(message))
    return 200

The decisions behind it

• Expand first, then order. Retries happen per deployment. If one deployment from a push fails a check, only that one retries. The cost is that later pushes to that deployment wait behind it until it’s acknowledged or dead-lettered.
• Rejection is how a push waits. When a deployment is busy, the build service rejects the message, and Pub/Sub holds it and tries again. The 45 counts message deliveries, not builds.
• Cloud Build owns the build queue. One push can start more builds than the default pool’s 10 to 30 concurrent builds. Once builds.create accepts a build, queueing is Cloud Build’s job, so the platform works the same whether a push starts 5 builds or 50.
• Approval lives in the pipeline. Every push now passes through the platform before a build starts, so each environment’s rules apply at that point.
• Pub/Sub over Kafka. We were already on GCP, and Pub/Sub is managed, with queueing and retries built in. Kafka would have been another system to run, for a workload with one consumer.
• Push subscriptions with ordering keys. Google warns that push may not suit topics where messages are frequently published with the same ordering key. A deployment gets one message per push to its branch, which is low volume per key.
• Manual dead-letter triage. It’s simple to run, and it isn’t automatic yet. A message that lands there waits until someone looks at it.

If you build this

These are all in Google’s docs, and they’re easy to miss:

• Ordering needs two settings. Enable message ordering on the subscription, and publish related messages with the same ordering key from the same region.
• A push endpoint acknowledges a message only with 102, 200, 201, 202 or 204. Any other status is a negative acknowledgement.
• Redelivering one message also redelivers later messages with the same key, even acknowledged ones, so your endpoint has to handle duplicates.
• max_delivery_attempts must be between 5 and 100, and it’s approximate. Pub/Sub only counts attempts once the dead-letter topic and its IAM permissions are set up correctly.
• Attach a subscription to the dead-letter topic. Messages published to a topic with no subscription are lost.
• Order might not be preserved in the dead-letter topic, so don’t assume it when you retry from there.
• Cloud Build can expire a queued build. If a build waits longer than its queueTtl, its status becomes EXPIRED.

Results

Before: one trigger per deployment, and a dev project at the 600 limit. After:

• No trigger per deployment. Adding a deployment no longer adds a trigger, so the 600 limit no longer caps how many deployments a GCP project can hold.
• Scale. The platform now has 1,000+ distinct deployments across 15+ environments, added over 10+ months. That total spans many GCP projects, so it isn’t a direct comparison with the limit. None of them needs a trigger.
• Speed. With approvals disabled, a build starts less than 2 seconds after the push.
• Durability. Every push is a durable, auditable event.
• Order. Pushes to the same deployment are handled in order, and a failure stays with the deployment that caused it.

If every new deployment costs one cloud resource, that resource’s limit is your platform’s limit.

What I’d do differently

Not much in the design. Once the system was working, there wasn’t a lot I’d change.

I’d want to have known about the 600 limit earlier. We chose one trigger per deployment without it in mind, and found out when dev ran into it. If you create one cloud resource per deployment, per tenant or per anything else that grows, read that resource’s limits and check what already counts against them before you commit to the design.