Kubernetes Networking Challenges

 We have spent quite a few days to troubleshoot an intermittent problem in our dev clusters.

I ended up deploying a k8s cluster by hand on Ubuntu and watching the output of each command to finally figure out the issue we were experiencing.

When I installed istio I got this output:

bin/istioctl install

        |\          

        | \         

        |  \        

        |   \       

      /||    \      

     / ||     \     

    /  ||      \    

   /   ||       \   

  /    ||        \  

 /     ||         \ 

/______||__________\

____________________

  \__       _____/  

     \_____/        

❗ detected Calico CNI with 'bpfConnectTimeLoadBalancing=TCP'; this must be set to 'bpfConnectTimeLoadBalancing=Disabled' in the Calico configuration

This will install the Istio 1.28.0 profile "default" into the cluster. Proceed? (y/N) y

✔ Istio core installed ⛵️

✔ Istiod installed 🧠

✔ Ingress gateways installed 🛬                                                                                        

✔ Installation complete

I read a little bit more on Calico and figured out where that configuration was and modified it with:

kubectl edit -n kube-system felixconfigurations.crd.projectcalico.org

Normally this would be in the calico-system namespace but on the clusters deployed with Kubespray it is in the kube-system namespace.

After we restarted the cluster, everything was stable and much faster.

I like automated deployment but they hide a lot of the output so we had no idea what we were looking for.

It is also difficult to keep on top of everything that changes in every new version of every component of our stack and to be aware of the impact of these changes. This requires a lot of time.

Versions involved:

• Kubernetes 1.31, 1.32 and 1.34
• Calico 3.31
• Istio 1.28

We still have an issue with Calico Whisker because we disabled ipv6 on the nodes but it is a known bug with a workaround.

JVM Startup

 After reading this great article:JVM Startup

I tested a few things on my Macbook and on my Ubuntu laptop to see very different startup in Java 25.

On Linux, I see that the first thing it checks is if it is in a container. I don't see that on MacOS.

On linux it checks for hugepage support which is not need or done on MacOS.

It is interesting to see the differences between the two OS.

It is nice to see all the resources allocations that are done and they vary to match the different laptops that I have. 

I feel that there are more details under Linux about the librairies and allocation of resources.

The allocation of the garbage collector seems the same on both and for a start with no parameters it is good to see the same defaults.

A very long output of trace logs but very interesting to compare and learn a little bit more about the startup of a JVM. 

Python and boto3

 I had enough trouble to get a python script to work with an S3 storage.

If you use Dell ECS or GCS, it seems that the latest version (1.39) of the recommended library (boto3) to exchange data with those S3 services is a no go.

I followed the recommendations from some users to downgrade to 1.35 and that worked well.

I then tried quite a few suggested solutions to try to get 1.39 to work and I could not make it so.

I always get an error about signature mismatch like:

An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: The Content-SHA256 you specified did not match what we received.

Involuntary QA

 How often do I find a problem when using a product?

It feels way too often.

I am trying to create a new account for a new application and I get all sorts of errors and it becomes a frustrating experience to try to use it.

Yesterday, I am trying to place an order at DQ and so you have to install the app and create an account. It forces you to pick your favourite DQ. The application is not able to find any place in the whole city. I even entered the address of the location I was at and it said it could not find any DQ. I had to quit the app and start from scratch.

There is no way for me to report this issue easily. They won't know. They won't fix this.

I want to help but it is a bigger adventure to report issues in any app or service out there.

I have the feeling that I could report a bug every day.

I am sure that I am not alone having all these issues and no one can report them easily.

As a dev, you have to make it easy for the users to tell you when thinks don't work as expected.

Better Bash Scripting - Error Handling

 I was not aware of all the options you could set at the beginning of a script to help it be more reliable.

I have decided that my script will now have:

set -euo pipefail

The set -e is for the script to exit when there is an error (non-zero) execution of a line. This can come with some side effect but there are quick ways to adapt to them. You could do:

command_wit_error || true

To get your script to continue even if the command on the left did not complete with a zero.

The set -u is good to make sure all variables are set for the script to run properly. Reminds me of a strict mode in some other tools.

The set -o pipefail is good to make sure you get the error code of the rightmost command in the pipeline.

Quick reading on this topic because we have an error in a deployment script that did not prevent the rest to be attempted and cause other issues.

Intellij IDEA adventures with Python

 I am a bit puzzled this morning.

Yesterday I built a simple python application and containerized it. It was all working in Intellij with the python plugin and I could run it with the Dockerfile. A good experience that made it easy to develop and test.

The only thing not working in IntelliJ was the AI assistant. The plugin was stuck in eap-preview and no matter than I re-installed it and restarted IntelliJ, he did not want to recognize my license. Frustrating.

I apply the update to 2023.3.1 and this morning...

I have a working AI assistant.

IntelliJ will not recognize that python is available and keeps telling me that it can't reach the Docker server. Complete Amnesia.

At the CLI, I can confirm that docker and python are available.

I have no clue what it lost in its configuration to not be able to recognize anything.

First python application using my GPU

 I simply wanted to make sure that some Python code could use my GPU and it took a little more effort than I taught.

The code I found simply makes sure you have some GPU available and fills in a matrix of 1000x1000. Very simple.

First rookie mistake was to call my file tensorflow.py which cause a circular reference that took me too long to figure out.

I also needed to install a few missing components on my fresh Ubuntu 22.04 installation.

pip install tensorflow

sudo apt install nvidia-cuda-toolkit

pip install tensorrt

Once all these components were present, the application ran without errors.

SDKMan needs curl from apt install

 I made a rookie mistake and installed the curl from the snap store and that is not compatible with SDKMan. I would get errors about curl failing:

curl: Failed to open /home/user/.sdkman/tmp/spark-3.4.0.headers.tmp

The solution was to remove the snap curl and get the apt one installed:

sudo snap remove curl

sudo apt install curl

Easy enough and I can now install all my java tools.

Keycloak Outage

 It is a strange behavior that I learned today.

We run keycloak in HA in K8S. Bitnami chart. Nothing fancy.

We have a volume for the custom theme.

If I remove the theme files and put new one in place... All pods will restart...

This was very unexpected.

Next time we replace the theme we will need to have a maintenance window.

Security Headers

 A useful site to check your deployed application:

Security Headers Scanner

It easily shows you what you have set properly and what is missing.

Gives you a grade that is color coded.

It helps to get this done before a pen test.

Istio and proxy-protocol

 When you search for solutions on the internet it is sometimes very difficult to know if what is proposed is going to work in your environment. The challenge comes from the fact that not everyone identifies the version they were working with at the time they applied this solution.

I find this snippet of YAML to be applied to my istio deployment to allow some headers to be properly passed on with the requests to the destination service.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  configPatches:
    - applyTo: LISTENER
      patch:
        operation: MERGE
        value:
          listener_filters:
            - name: envoy.listener.proxy_protocol
            - name: envoy.listener.tls_inspector
  workloadSelector:
    labels:
      istio: ingressgateway

Once applied, I tested but nothing changed and I still get the errors:

ERR_TOO_MANY_REDIRECTS

I then decide to see if restarting the istio ingress pod would fix things.

Well…

The pod is not coming back so I pull the logs and rapidly reads to identify what is causing this issue. I finally spot a few lines:

2022-07-12T20:39:29.975846Z error envoy misc Using deprecated extension name 'envoy.listener.proxy_protocol' for 'envoy.filters.listener.proxy_protocol'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this filter name is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
2022-07-12T20:39:30.421115Z error envoy misc Using deprecated extension name 'envoy.listener.proxy_protocol' for 'envoy.filters.listener.proxy_protocol'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this filter name is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
2022-07-12T20:39:30.429091Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Didn't find a registered implementation for name: 'envoy.listener.proxy_protocol'
0.0.0.0_8080: Didn't find a registered implementation for name: 'envoy.listener.proxy_protocol'

So I delete the EnvoyFilter and everything comes back.

So envoy.listener.proxy_protocol does not work with istio 1.13.x? I will have to read more to figure that part out.

Spring Beans not Instantiating

I had not seen this error before:

No qualifying bean of type 'org.springframework.security.oauth2.client.registration.ClientRegistrationRepository' available

It took a little bit of research to find that if you do not have your application.yaml with the Spring Security section for this bean, it will not be able to instatiate one and you get the error.

To make it more difficult, I was specifying this module in gradle and I was expecting things to work. Asuming…

Once I added:

spring:
  security:
    oauth2:
      client:
        registration:
          nameItSomething:
            authorization-grant-type: client_credentials
            client-id: clientNameYouHave
            client-secret: DontShareYourSecretOnTheInternet
        provider:
          nameItSomething:
            token-uri: https://oidcServer/protocol/openid-connect/token

That ClientRegistrationRepository bean started to have a life.

Next bean… Same… I need to add a section for the caching in my application.yaml.

Dear Checkstyle

 I am not a big fan of forced styles and all these errors that comes with it.

I know, I know… If you are on a team of more people, you don’t have much of a choice otherwise you end up with style chaos and naming convention that are closer to a standup comedian number than logic.

I am just not a big fan…

We have a style for the member name in our classes that only allows for letters and number but I did a class to match the json returned from Azure OIDC so it is all lowercase name with underscore. Checkstyle would not let this through so I learned quickly how to turn that off for my class

This is the module in Checkstyle:

<module name="SuppressionCommentFilter">
    <property name="offCommentFormat" value="CHECKSTYLE.OFF\: ([\w\|]+)"/>
    <property name="onCommentFormat" value="CHECKSTYLE.ON\: ([\w\|]+)"/>
    <property name="checkFormat" value="$1"/>
</module>

All you have to do is a comment:

// CHECKSTYLE.OFF: MemberName

and after my class definition I can simply turn it back on:

// CHECKSTYLE.ON: MemberName

Then I could discuss with a team mate and disagree about having an annotation (@jsonproperty(“name_with_underscore”)) for each member name to respect the checkstyle and work with the json. What is the simplest will always win in my mind and not following made up rules for the sake of the rule.