Load Balancer at Scaleway
Server applications failures and lack of scalability can cause real problems for a server. When server applications fail, a node is unavailable to answer requests...
At Golem.ai, our day-to-day decisions and choices justify the trust our customers place in us.
Security is at the heart of each and every one of them.
That's why we distinguish between public and private traffic at the very heart of our applications and our Kubernetes Hosting Platform operated by Scaleway, our French Cloud provider.
Thus, Public traffic, which by definition enables our customers to consume our applications, is strictly controlled by our powerful Web Application Firewall (WAF) and AntiDDoS, while private traffic is dedicated to the administration of our applications by our Teams, and is only accessible via a ZeroTrust VPN Tunnel connected to the Private LoadBalancer.
This gives us a 360° view of network activity upstream and downstream of the platform.
In terms of watertightness between our different STAGING, PREPRODUCTION, PRODUCTION and MONITORING environments, each has its own virtual private network (VLAN Layer 2 within a VPC).
This prevents private flows from being opened up to the outside world, and isolates flows between environments if necessary.
The operations described below correspond to the needs of Golem.ai.
Adapt to your needs !
To use a Private LoadBalancer between our application and the Golem.ai Private Network, we'll need to perform the following steps in order.
Click Load Balancers in the Network section of the Scaleway console side menu. If you have not already created a Load Balancer, the product creation page is displayed. Otherwise, your list of existing Load Balancers displays. Then, choose "Private Load Balancer". More info here, and below...
Informations1. LB Name Fill in the name of the loadbalancer, respecting the nomenclature. private-lb-private-“env”-“domain without extension” eg . private-lb-prod-test for test.golem.ai2. LB Zone Select zone where application is localised (PARIS 1 by default)3. LB Model Select LB-S model. 200 Mbps of bandwidth is more than enough.4. LB Type Select Private Load Balancer 5. Finalise Click on "Create Load Balancer"
To use Private LB, we no longer need to use Ingress.
Traffic is sent directly to the application's service in a private flow !
Scaleway Private LB ⇒ SVC K8s ⇒ Application
In our Service K8s object, we use the following annotations:
service.beta.kubernetes.io/scw-loadbalancer-externally-managed : true service.beta.kubernetes.io/scw-loadbalancer-id : zone/lb_idservice.beta.kubernetes.io/scw-loadbalancer-zone : zone
Please just note what CCM will or won’t manage in this case:
• Won’t create/delete the LB.
• Ignores the global configurations (such as size, private mode, IPs).
• Won’t detach private networks attached to the LB.
• won’t manage extra frontends and backends not starting with the service id.
• Will refuse to manage a LB with a name starting with the cluster id.
E.g.:
---apiVersion: v1kind: Servicemetadata: name: test-lb-private namespace: ops-tools annotations: service.beta.kubernetes.io/scw-loadbalancer-externally-managed: "true" service.beta.kubernetes.io/scw-loadbalancer-id: "fr-par-1/0d1b714f-2767-4937-a9e5-4399cfd45338" service.beta.kubernetes.io/scw-loadbalancer-zone: fr-par-1 labels: app.kubernetes.io/name: test-lb-private app.kubernetes.io/instance: test-lb-privatespec: ports: - name: https port: 443 <= IMPORTANT: 443 port for use of HTTPS certificate in Private LB protocol: TCP targetPort: 80 - name: test port: 81 protocol: TCP targetPort: 81 selector: app.kubernetes.io/name: test-lb-private app.kubernetes.io/instance: test-lb-private type: LoadBalancer <= IMPORTANT: LoadBalancer type for link
Apply.
Change the name of the frontend using port 443 to the domain you want to use it on.
the “golem.ai” extension is mandatory.
In the following, we'll show you how to use a different domain extension to access the application from a different environment.
If we are accessing the application from a different environment(s) than the one it was created in, we must add the private networks:
For HTTPS access to the application from its environment or from other environments, we need to add an A entry.
Private @IP for test.golem.ai : 172.16.0.53 (See the loadbalancer's private network @IP)
Scaleway's Private Loadbalancer doesn’t support automatic management of let's encrypt certificates by CertManager.
So we’ve to manage their entire lifecycle and deploy them ourselves.
To avoid this, we've set up a mechanism using the APIs of Scaleway, CloudFlare and Certbot, as described below.
Requirements :
⇒ Available on the Golem.ai Github Repository : https://github.com/golem-ai/privatelb-scaleway
To use this script, you'll need certbot and the certbot-dns-cloudflare plugin installed on the runtime environment.
Two files :
Important fields are here :
On certmanager-privatelbscw.sh
##Certbot##lets_server=https://acme-v02.api.letsencrypt.org/directoryauth_email=XXXXXXXXXcloudflare_config=cloudflare.ini##Scaleway API Token##scw_token=XXXXXXXXX##URI : Application name without domain extension##hostname=("test")##Domain extension : golem.ai , toto.ai , tata.ai , titi.ai##default_envs=("golem.ai")##Multi Private Network : To access resources from an environment other than the one from which the SVC K8s was created##multipns=("no")##LBs###test:0d1b714f-2767-4937-a9e5-4399cfd45338private_lbs=("0d1b714f-2767-4937-a9e5-4399cfd45338")zones=("fr-par-1")
lets_server : Fill in the Let's encrypt environment with which you wish to generate the certificate (production or staging).
auth_email : Fill in the email address we wish to use for certificate generation
scw_token : Secret token for Scaleway API authentication
domains : Fill in the domain we created earlier (test)
multipns : Would we like to access our applications from a non-production environments ? (no / yes)
private_lbs : Our Private LB ID
Zone : Area from which we have created our Private LB
On cloudflare.ini
# Cloudflare API token used by Certbotdns_cloudflare_api_token = XXXXXXXXX
dns_cloudflare_api_token : Fill with Cloudflare API token
Execution :
The information entered in the example will be used to access the test application via the url test.golem.ai from the production environment.
Once completed, save and run the script!
./certmanager-privatelbscw.sh xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 20 13:34:06 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 25 10:56:11 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 25 10:47:04 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 27 11:42:58 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 27 11:48:00 2024 GMT----Could not read certificate from <stdin>40D7FB7F107F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:Unable to load certificatetest.golem.ai----Saving debug log to /var/log/letsencrypt/letsencrypt.logRequesting a certificate for test.golem.aiSuccessfully received certificate.Certificate is saved at: /etc/letsencrypt/live/test.golem.ai/fullchain.pemKey is saved at: /etc/letsencrypt/live/test.golem.ai/privkey.pemThis certificate expires on 2024-08-28.These files will be updated when the certificate renews.Certbot has set up a scheduled task to automatically renew this certificate in the background.- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 4887 100 1178 100 3709 1593 5016 --:--:-- --:--:-- --:--:-- 6612 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 4442 0 4442 0 0 10888 0 --:--:-- --:--:-- --:--:-- 10887b33097cf-4e49-456b-990b-1bc8a64caef3fd7c47b7-9214-4945-9bae-67f18bfe0070443{"id":"fd7c47b7-9214-4945-9bae-67f18bfe0070", "name":"test.golem.ai", "inbound_port":443, "backend":{"id":"b33097cf-4e49-456b-990b-1bc8a64caef3", "name":"915d146e-fef6-4873-92f9-58cbea07ac4e_tcp_30715", "forward_protocol":"tcp", "forward_port":30715, "forward_port_algorithm":"roundrobin", "sticky_sessions":"none", "sticky_sessions_cookie_name":"", "health_check":{"port":30715, "check_delay":5000, "check_timeout":5000, "check_max_retries":5, "check_send_proxy":false, "transient_check_delay":null, "tcp_config":{}}, "pool":["172.16.0.54", "172.16.0.7", "172.16.0.25", "172.16.0.9", "172.16.0.21", "172.16.0.18", "172.16.0.15", "172.16.0.31", "172.16.0.14", "172.16.0.17", "172.16.0.22", "172.16.0.27"], "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "send_proxy_v2":false, "timeout_server":10000, "timeout_connect":600000, "timeout_tunnel":600000, "on_marked_down_action":"on_marked_down_action_none", "proxy_protocol":"proxy_protocol_none", "created_at":"2024-05-30T11:39:17.384840Z", "updated_at":"2024-05-30T11:39:17.384840Z", "failover_host":null, "ssl_bridging":false, "ignore_ssl_server_verify":null, "redispatch_attempt_count":0, "max_retries":3, "max_connections":null, "timeout_queue":null}, "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[{"id":"9bf4b0ab-8fcd-405a-9c9c-9e246e03b057", "status":"pending", "ip_address":"", "created_at":"2024-05-30T10:23:15.724826Z", "updated_at":"2024-05-30T12:59:23.088049484Z", "region":"fr-par", "zone":"fr-par-1"}], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "timeout_client":600000, "certificate":{"id":"aba65e0d-7391-4685-b8c3-f86b88c2ade5", "type":"custom", "status":"ready", "common_name":"test.golem.ai", "subject_alternative_name":[], "fingerprint":"", "not_valid_before":"2024-05-30T11:59:18Z", "not_valid_after":"2024-08-28T11:59:17Z", "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "name":"test.golem.ai-2024-05-30", "created_at":"2024-05-30T12:59:21.483814Z", "updated_at":"2024-05-30T12:59:23.049179683Z", "status_details":null}, "certificate_ids":["aba65e0d-7391-4685-b8c3-f86b88c2ade5"], "created_at":"2024-05-30T11:39:17.735956Z", "updated_at":"2024-05-30T12:59:23.045618828Z", "enable_http3":true}
The certificate has been created and is associated with the frontend on which we want to access the application on port 443.
Renewal only within 30 days of expiry.
Access to the application :
Once again, test.golem.ai is only accessible from our ZeroTrust Golem.ai Private Network.
Outside this network, the application doesn’t exist (172.16.0.53 / non-routable IP address).
nslookup test.golem.aiServer: 127.0.0.53Address: 127.0.0.53#53Non-authoritative answer:Name: test.golem.aiAddress: 172.16.0.53
Server applications failures and lack of scalability can cause real problems for a server. When server applications fail, a node is unavailable to answer requests...
Networking is one of the most complex aspects of Kubernetes. If you wish to run a publicly accessible service in Kubernetes successfully you need to understand how to leverage the tools.
This article will guide you through the best practices to deploy and distribute the workload on a multi-cloud Kubernetes environment on Scaleway's Kosmos.