Problem

You’re trying to connect to a custom endpoint within the same VPC/VNet as your workspace. The endpoint is hosted on non-standard ports (ports other than HTTP-80 or HTTPS-443). You receive a Connection Refused error during connection attempts. 

You notice the issue when working on a standard (formerly shared) access mode cluster on Databricks Runtime 11.3 LTS or below. 

Cause

Databricks Runtimes 11.3 LTS and below restrict outbound access to certain ports by default on standard access mode clusters, even within the same VPC/VNet. 

Solution

Databricks recommends using Databricks Runtime 12.2 LTS or above with standard access mode. 

If you prefer to continue using Databricks Runtime 11.3 LTS or below, use the following cluster-scoped init script to allow access to the custom endpoint. This script continuously ensures that outbound traffic to the specified port and CIDR is allowed from the cluster nodes.

#!/bin/bash cat << 'EOF' > /tmp/set_rules.sh #!/bin/bash set -x sleep_interval=30s port="<your-target-port>"  ## Change this to your target port cidr="<your-workspace-VPC-or-VNET-CIDR>"  ## Replace with your workspace VPC/VNet CIDR while true; do     rules=$(iptables -L | grep -i "$port")     if [[ "$rules" != *"dpt:$port"* && $(getent group spark-users) ]]; then         echo "Changing rules at $(date)"          iptables -I OUTPUT 2 -d $cidr -j ACCEPT -p tcp --dport $port     fi     sleep ${sleep_interval} done EOF chmod a+x /tmp/set_rules.sh /tmp/set_rules.sh >> /tmp/set_rules.log & disown