GraphQL Mutation Failures Due to Database Hostname Resolution Error

[SankaraSubramanian048]

During normal processing of GraphQL operations through Hasura GraphQL Engine, multiple requests failed with database connection errors. The errors occurred while executing GraphQL mutations that perform multiple insert operations into the backend database.

The investigation identified that the system experienced a temporary DNS resolution failure when attempting to connect to the database host. Because the database hostname could not be resolved during that period, the application was unable to establish a connection to the database.

As a result, GraphQL requests failed and the application returned service errors.

Observed Error

The key error message observed in logs was:

postgres-error: connection error
could not translate host name "" to address: Temporary failure in name resolution

This indicates that the system could not resolve the database hostname to an IP address at the time of the request.
The failure occurred before the database connection was established, meaning the GraphQL mutation was not executed.
Additionally, during investigation we noticed references to a transaction statement similar to:
BEGIN ISOLATION LEVEL REPEATABLE READ

Any guidance or insight would be greatly appreciated.

[manimovassagh]

This is a classic transient DNS resolution issue, not really a Hasura-specific bug. A few things to check:

1. DNS caching / TTL: If your database hostname has a short TTL and your DNS resolver hiccups, you get exactly this. Consider using a local DNS cache (like dnsmasq or systemd-resolved) on the host running Hasura to buffer against short outages.

2. Connection pooling: If you are using PgBouncer or a similar connection pooler between Hasura and Postgres, the pooler should maintain existing connections and only need DNS for new ones. This significantly reduces the blast radius of a DNS blip.

3. Hasura connection retries: Check your HASURA_GRAPHQL_PG_CONNECTIONS and related pool settings. Hasura will attempt to establish new connections when the pool is exhausted or connections are dropped. If DNS fails at that exact moment, you get this error. Increasing the pool idle timeout can help keep existing connections alive longer.

4. REPEATABLE READ isolation: The BEGIN ISOLATION LEVEL REPEATABLE READ you saw is expected -- Hasura uses this for mutations that touch multiple tables to ensure consistency. This is not related to the DNS failure; it just happens to be the statement that was about to execute when the connection failed.

5. Infrastructure-level fixes:

   • If running on Kubernetes, make sure your ndots and DNS policy are configured properly (the default ndots:5 can cause excessive DNS lookups).
   • If on AWS, check if you are hitting the DNS query rate limit on your VPC resolver (1024 packets per second per interface).
   • Consider using the IP address directly in HASURA_GRAPHQL_DATABASE_URL as a short-term workaround to confirm DNS is the issue.

The core fix is to make your DNS resolution more resilient rather than changing anything in Hasura itself.