Scaleway Blog

Scaleway Private LB for Golem.ai Security Assets

Deploy
Kevin Baude
5 min read

At Golem.ai, our day-to-day decisions and choices justify the trust our customers place in us.

Security is at the heart of each and every one of them.

That's why we distinguish between public and private traffic at the very heart of our applications and our Kubernetes Hosting Platform operated by Scaleway, our French Cloud provider.

Thus, Public traffic, which by definition enables our customers to consume our applications, is strictly controlled by our powerful Web Application Firewall (WAF) and AntiDDoS, while private traffic is dedicated to the administration of our applications by our Teams, and is only accessible via a ZeroTrust VPN Tunnel connected to the Private LoadBalancer.

This gives us a 360° view of network activity upstream and downstream of the platform.

In terms of watertightness between our different STAGING, PREPRODUCTION, PRODUCTION and MONITORING environments, each has its own virtual private network (VLAN Layer 2 within a VPC).

This prevents private flows from being opened up to the outside world, and isolates flows between environments if necessary.

Endpoint : Public / Private Flow

Operation

The operations described below correspond to the needs of Golem.ai.
Adapt to your needs !

To use a Private LoadBalancer between our application and the Golem.ai Private Network, we'll need to perform the following steps in order.

1 . Create a new Private LB with WebUI , CLI or Terraform

Click Load Balancers in the Network section of the Scaleway console side menu. If you have not already created a Load Balancer, the product creation page is displayed. Otherwise, your list of existing Load Balancers displays. Then, choose "Private Load Balancer". More info here, and below…

Informations

1. LB Name

 Fill in the name of the loadbalancer, respecting the nomenclature.
 private-lb-private-“env”-“domain without extension”
 eg . private-lb-prod-test for test.golem.ai

2. LB Zone

 Select zone where application is localised (PARIS 1 by default)

3. LB Model

 Select LB-S model.
 200 Mbps of bandwidth is more than enough.

4. LB Type

 Select Private Load Balancer

5. Finalise

 Click on "Create Load Balancer"

2 . Linking your application to PrivateLB

To use Private LB, we no longer need to use Ingress.

Traffic is sent directly to the application's service in a private flow !

Scaleway Private LB ⇒ SVC K8s ⇒ Application

In our Service K8s object, we use the following annotations:

service.beta.kubernetes.io/scw-loadbalancer-externally-managed : true 
service.beta.kubernetes.io/scw-loadbalancer-id : zone/lb_id
service.beta.kubernetes.io/scw-loadbalancer-zone : zone

Please just note what CCM will or won’t manage in this case:
• Won’t create/delete the LB.
• Ignores the global configurations (such as size, private mode, IPs).
• Won’t detach private networks attached to the LB.
• won’t manage extra frontends and backends not starting with the service id.
• Will refuse to manage a LB with a name starting with the cluster id.

E.g.:

 ---
apiVersion: v1
kind: Service
metadata:
  name: test-lb-private
  namespace: ops-tools
  annotations:
    service.beta.kubernetes.io/scw-loadbalancer-externally-managed: "true"
    service.beta.kubernetes.io/scw-loadbalancer-id: "fr-par-1/0d1b714f-2767-4937-a9e5-4399cfd45338"
    service.beta.kubernetes.io/scw-loadbalancer-zone: fr-par-1
  labels:
    app.kubernetes.io/name: test-lb-private
    app.kubernetes.io/instance: test-lb-private
spec:
  ports:
    - name: https
      port: 443 <= IMPORTANT: 443 port for use of HTTPS certificate in Private LB
      protocol: TCP
      targetPort: 80
    - name: test
      port: 81
      protocol: TCP
      targetPort: 81
  selector:
    app.kubernetes.io/name: test-lb-private
    app.kubernetes.io/instance: test-lb-private
  type: LoadBalancer <= IMPORTANT: LoadBalancer type for link

Apply.

3. Adjust LB settings

Change the name of the frontend using port 443 to the domain you want to use it on.
the “golem.ai” extension is mandatory.

In the following, we'll show you how to use a different domain extension to access the application from a different environment.

If we are accessing the application from a different environment(s) than the one it was created in, we must add the private networks:

4 . Adding a DNS entry

For HTTPS access to the application from its environment or from other environments, we need to add an A entry.

Private @IP for test.golem.ai : 172.16.0.53 (See the loadbalancer's private network @IP)

5 . Create Let's Encrypt certificate and associate it with frontend

Scaleway's Private Loadbalancer doesn’t support automatic management of let's encrypt certificates by CertManager.

So we’ve to manage their entire lifecycle and deploy them ourselves.

To avoid this, we've set up a mechanism using the APIs of Scaleway, CloudFlare and Certbot, as described below.

Requirements :

⇒ Available on the Golem.ai Github Repository : https://github.com/golem-ai/privatelb-scaleway

To use this script, you'll need certbot and the certbot-dns-cloudflare plugin installed on the runtime environment.

Two files :

  • certmanager-privatelbscw.sh for LB certificate management
  • cloudflare.ini for Certbot ACME management on CloudFlare DNS Provider

Important fields are here :

On certmanager-privatelbscw.sh

##Certbot##
lets_server=https://acme-v02.api.letsencrypt.org/directory
auth_email=XXXXXXXXX
cloudflare_config=cloudflare.ini
##Scaleway API Token##
scw_token=XXXXXXXXX
##URI : Application name without domain extension##
hostname=("test")
##Domain extension : golem.ai , toto.ai , tata.ai , titi.ai##
default_envs=("golem.ai")
##Multi Private Network : To access resources from an environment other than the one from which the SVC K8s was created##
multipns=("no")
##LBs##
#test:0d1b714f-2767-4937-a9e5-4399cfd45338
private_lbs=("0d1b714f-2767-4937-a9e5-4399cfd45338")
zones=("fr-par-1")

lets_server : Fill in the Let's encrypt environment with which you wish to generate the certificate (production or staging).

auth_email : Fill in the email address we wish to use for certificate generation

scw_token : Secret token for Scaleway API authentication

domains : Fill in the domain we created earlier (test)

multipns : Would we like to access our applications from a non-production environments ? (no / yes)

private_lbs : Our Private LB ID

Zone : Area from which we have created our Private LB

On cloudflare.ini

# Cloudflare API token used by Certbot
dns_cloudflare_api_token = XXXXXXXXX

dns_cloudflare_api_token : Fill with Cloudflare API token

Execution :

The information entered in the example will be used to access the test application via the url test.golem.ai from the production environment.

Once completed, save and run the script!

./certmanager-privatelbscw.sh 
xxxxxxxx.golem.ai
issuer=C = US, O = Let's Encrypt, CN = R3
Aug 20 13:34:06 2024 GMT
----
xxxxxxxx.golem.ai
issuer=C = US, O = Let's Encrypt, CN = R3
Aug 25 10:56:11 2024 GMT
----
xxxxxxxx.golem.ai
issuer=C = US, O = Let's Encrypt, CN = R3
Aug 25 10:47:04 2024 GMT
----
xxxxxxxx.golem.ai
issuer=C = US, O = Let's Encrypt, CN = R3
Aug 27 11:42:58 2024 GMT
----
xxxxxxxx.golem.ai
issuer=C = US, O = Let's Encrypt, CN = R3
Aug 27 11:48:00 2024 GMT
----
Could not read certificate from <stdin>
40D7FB7F107F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:
Unable to load certificate
test.golem.ai


----
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for test.golem.ai

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/test.golem.ai/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/test.golem.ai/privkey.pem
This certificate expires on 2024-08-28.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4887  100  1178  100  3709   1593   5016 --:--:-- --:--:-- --:--:--  6612
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4442    0  4442    0     0  10888      0 --:--:-- --:--:-- --:--:-- 10887
b33097cf-4e49-456b-990b-1bc8a64caef3
fd7c47b7-9214-4945-9bae-67f18bfe0070
443
{"id":"fd7c47b7-9214-4945-9bae-67f18bfe0070", "name":"test.golem.ai", "inbound_port":443, "backend":{"id":"b33097cf-4e49-456b-990b-1bc8a64caef3", "name":"915d146e-fef6-4873-92f9-58cbea07ac4e_tcp_30715", "forward_protocol":"tcp", "forward_port":30715, "forward_port_algorithm":"roundrobin", "sticky_sessions":"none", "sticky_sessions_cookie_name":"", "health_check":{"port":30715, "check_delay":5000, "check_timeout":5000, "check_max_retries":5, "check_send_proxy":false, "transient_check_delay":null, "tcp_config":{}}, "pool":["172.16.0.54", "172.16.0.7", "172.16.0.25", "172.16.0.9", "172.16.0.21", "172.16.0.18", "172.16.0.15", "172.16.0.31", "172.16.0.14", "172.16.0.17", "172.16.0.22", "172.16.0.27"], "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "send_proxy_v2":false, "timeout_server":10000, "timeout_connect":600000, "timeout_tunnel":600000, "on_marked_down_action":"on_marked_down_action_none", "proxy_protocol":"proxy_protocol_none", "created_at":"2024-05-30T11:39:17.384840Z", "updated_at":"2024-05-30T11:39:17.384840Z", "failover_host":null, "ssl_bridging":false, "ignore_ssl_server_verify":null, "redispatch_attempt_count":0, "max_retries":3, "max_connections":null, "timeout_queue":null}, "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[{"id":"9bf4b0ab-8fcd-405a-9c9c-9e246e03b057", "status":"pending", "ip_address":"", "created_at":"2024-05-30T10:23:15.724826Z", "updated_at":"2024-05-30T12:59:23.088049484Z", "region":"fr-par", "zone":"fr-par-1"}], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "timeout_client":600000, "certificate":{"id":"aba65e0d-7391-4685-b8c3-f86b88c2ade5", "type":"custom", "status":"ready", "common_name":"test.golem.ai", "subject_alternative_name":[], "fingerprint":"", "not_valid_before":"2024-05-30T11:59:18Z", "not_valid_after":"2024-08-28T11:59:17Z", "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "name":"test.golem.ai-2024-05-30", "created_at":"2024-05-30T12:59:21.483814Z", "updated_at":"2024-05-30T12:59:23.049179683Z", "status_details":null}, "certificate_ids":["aba65e0d-7391-4685-b8c3-f86b88c2ade5"], "created_at":"2024-05-30T11:39:17.735956Z", "updated_at":"2024-05-30T12:59:23.045618828Z", "enable_http3":true}

The certificate has been created and is associated with the frontend on which we want to access the application on port 443.

Renewal only within 30 days of expiry.

Access to the application :

⇒ https://test.golem.ai/

Once again, test.golem.ai is only accessible from our ZeroTrust Golem.ai Private Network.
Outside this network, the application doesn’t exist (172.16.0.53 / non-routable IP address).

nslookup test.golem.ai
Server:        127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:    test.golem.ai
Address: 172.16.0.53
Share on
Other articles about:
Load BalancerManaged ServicesHow to

Recommended articles