Vfrdtyky

I am unable to connect to any service I expose from my GKE cluster despite being able to see the pod up and running. Any ideas what I am doing wrong?

I have a GKE cluster which has private nodes (public master). I have it configured to assign the pods ips from 10.0.x.x and the services ips from 10.2.x.x

A terraform for the cluster is shown below

resource "google_container_cluster" "playground" {

  provider    = "google-beta"

  name        = "playground"

  description = "Playground cluster"

  project     = "${module.playground_project.project_id}"

  zone        = "europe-west4-a"



  min_master_version = "1.11.2-gke.9"



  master_auth {

    username = "admin"

    password = "xxx"

  }

  lifecycle {

    ignore_changes = ["initial_node_count", "node_config", "node_pool", "network", "subnetwork"]

  }



  network = "${google_compute_network.playground.self_link}"



  subnetwork = "${google_compute_subnetwork.playground-gke.self_link}"



  private_cluster_config {

    enable_private_endpoint = false

    enable_private_nodes    = true

    master_ipv4_cidr_block  = "172.30.16.0/28"

  }







  master_authorized_networks_config {

    cidr_blocks = [

      {

        cidr_block   = "${var.my_ip}"

      },

    ]

  }

  node_pool {

    name = "default-pool" # Default empty node pool

  }

  ip_allocation_policy {

    # create_subnetwork = true  # subnetwork_name   = "gke-playground"

    cluster_secondary_range_name  = "subnet-play-gke-pods" # 10.0.0.0/15

    services_secondary_range_name = "subnet-play-gke-services" #10.2.0.0/15

  }

}



resource "google_container_node_pool" "np" {

  provider = "google-beta"

  name     = "node-pool-1"

  project  = "${module.playground_project.project_id}"

  zone     = "europe-west4-a"

  cluster  = "playground"



  depends_on = ["google_container_cluster.playground"]



  management {

    auto_upgrade = true

    auto_repair  = true

  }



  lifecycle {

    ignore_changes = ["node_count"]

  }



  # Enable this or autoscaling, not both

  # node_count = 1



  autoscaling {

    min_node_count = 1

    max_node_count = 3

  }

  initial_node_count = 1

  node_config {

    # preemptible  = true

    machine_type = "n1-standard-1"

    disk_size_gb = "20"

    disk_type    = "pd-standard"



    # metadata

    # labels

    # tags

    tags = ["gke"]



    labels = [

      {

        environment = "playground"

      },

    ]



    oauth_scopes = [

      "https://www.googleapis.com/auth/compute",

      "https://www.googleapis.com/auth/devstorage.read_only",

      "https://www.googleapis.com/auth/logging.write",

      "https://www.googleapis.com/auth/monitoring",

    ]

  }

}

I have setup a bastion within the VPC that can reach the nodes (verified by sshing to the nodes)

I am able to deploy an application like so:

local $ kubectl run hello --image=gcr.io/google-samples/hello-app:1.0 --port=8080

deployment.apps "hello" created



local $ kubectl get deployment hello

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

hello     1         1         1            1           1m



local $ kubectl expose deployment hello --target-port=8080 --type=NodePort

service "hello" exposed



local $ kubectl describe service hello

Name:                     hello

Namespace:                default

Labels:                   run=hello

Annotations:              <none>

Selector:                 run=hello

Type:                     NodePort

IP:                       10.2.109.113

Port:                     <unset>  8080/TCP

TargetPort:               8080/TCP

NodePort:                 <unset>  32420/TCP

Endpoints:                10.0.2.13:8080

Session Affinity:         None

External Traffic Policy:  Cluster

Events:                   <none>

Now if I connect to my bastion I can query the pod directly on 10.0.2.13:8080, which is great

bastion $ curl 10.0.2.13:8080

Hello, world!

Version: 1.0.0

Hostname: hello-68669bb559-x7zpb

But if I try to connect to the service url on 10.2.109.113 my connection times out:

bastion $ curl -vvvv --connect-timeout 10 10.2.109.113:32420

* Rebuilt URL to: 10.2.109.113:32420/

*   Trying 10.2.109.113...

* TCP_NODELAY set

* Connection timed out after 10001 milliseconds

* Curl_http_done: called premature == 1

* stopped the pause stream!

* Closing connection 0

curl: (28) Connection timed out after 10001 milliseconds

A dump from gcloud with my firewall rules is below to give an idea of the firewall rules

local $gcloud compute firewall-rules list

NAME                                     NETWORK   DIRECTION  PRIORITY  ALLOW                         DENY  DISABLED

default-allow-icmp                       default   INGRESS    65534     icmp                                False

default-allow-internal                   default   INGRESS    65534     tcp:0-65535,udp:0-65535,icmp        False

default-allow-rdp                        default   INGRESS    65534     tcp:3389                            False

default-allow-ssh                        default   INGRESS    65534     tcp:22                              False

egress-from-bastion-to-me-over-ssh   vpc-play  EGRESS     1000      tcp:22                              False

gke-playground-f9a5cbc4-all              vpc-play  INGRESS    1000      sctp,tcp,udp,icmp,esp,ah            False

gke-playground-f9a5cbc4-master           vpc-play  INGRESS    1000      tcp:10250,tcp:443                   False

gke-playground-f9a5cbc4-vms              vpc-play  INGRESS    1000      icmp,tcp:1-65535,udp:1-65535        False

ingress-from-bastion-to-gke-over-all     vpc-play  INGRESS    1000      all                                 False

ingress-from-me-to-bastion-over-ssh  vpc-play  INGRESS    1000      tcp:22                              False

k8s-fw-l7--fff685c495e2595e              vpc-play  INGRESS    1000      tcp:30000-32767                     False

nat-europe-west4-a                       vpc-play  INGRESS    1000      all                                 False

nat-gateway-europe-west4-a-vm-ssh        vpc-play  INGRESS    1000      tcp:22                              False

service IPs are not rout-able outside the cluster. If you want to test the service, try curl [node_IP]:32420
This will hit the nodeport on the node which will reach your service endpoint