fix(deliverymq): Add index to attempts_default partition

[samadalishah]

Summary

Adds a PostgreSQL composite index on attempts_default to optimize retry execution lookups for the latest attempt by event, tenant, and destination.

Context

Retry execution fetches the latest prior attempt from logstore before publishing the next delivery task.
deliverymq-retry task → retry scheduler → fetch latest attempt → execute delivery

The query filters by:

• event_id
• tenant_id
• destination_id

and orders by:

• time DESC
• id DESC

with LIMIT 1.

Under high I/O load, PostgreSQL can choose an existing tenant/destination/time index and apply event_id as a heap filter. In large default partitions, that can require many page reads for a single retry lookup.

During an incident, slow Postgres retry lookups contributed to retry tasks running longer than the Redis retry visibility timeout RETRY_VISIBILITY_TIMEOUT_SECONDS (default 30 secs). That allowed the same retry task to become visible again and increased duplicate retry pressure.

Changes

Adds this index:

CREATE INDEX CONCURRENTLY IF NOT EXISTS attempts_default_event_tenant_destination_time_id_idx
ON attempts_default (event_id, tenant_id, destination_id, time DESC, id DESC);

Operational notes

This index is created only on attempts_default, which matches the current production table layout.
CREATE INDEX CONCURRENTLY is used to avoid blocking writes while the index is built. Building the index can still be I/O intensive on large deployments, so it should be monitored during rollout.
Longer term, monthly partitions would make retention and table maintenance safer, since old attempts could be removed by dropping partitions instead of batched deletes.

[samadalishah]

For more context:
We recently had an incident where a bunch of tasks went into deliverymq-retry-dlq and threw these logs:
task exceeded max receive count, moved to dead-letter queue

Our default partition already has 30 million rows. Looking into the attempts table, we noticed a particular event was attempted more than the allowed maximum (attempt_max: 6, but attempt_number: 11). This was happening for several events.

The root cause was that the Postgres DB was overloaded due to storage I/O saturation. Because the lookup query took more than 30 seconds (and up to 40+ seconds in some instances) and the Redis RETRY_VISIBILITY_TIMEOUT_SECONDS default is 30 seconds, the same task became visible multiple times. This created a feedback loop, causing even more load on the DB.

As a quick fix, we scaled up the DB instance to alleviate the bottleneck. However, we found that adding this index would be a more permanent help.

We have already applied this index to our database, but we opened this PR to see what the general stance is for this case. Also, are there any plans to add data cleanup as a feature to Outpost?

[alexluong]

Hi @samadalishah, thanks for opening the PR and sharing more context.

I'm trying to fully understand the issue. This statement strikes me as a bit odd.

Under high I/O load, PostgreSQL can choose an existing tenant/destination/time index and apply event_id as a heap filter. In large default partitions, that can require many page reads for a single retry lookup.

I don't fully follow why this is the case. I'm not sure why this is the behavior, especially "under high I/O load" since I don't fully see how that's related.

I'm personally leaning more towards the query itself using ANY even tho there's only 1 item.

Do you have the plan of the query when it used the wrong index by chance?