On the rationality of using go templates for generating yaml

Bulb

Kubernetes is a fairly useful tool. But you still have to somehow manage the manifests that tell it what to run and cover the differences between various testing environments and the production environment and between the AKS installation you are using and the GCE one management is contemplating on switching to. Then you can:

Use helm. Helm has one huge advantage. It keeps history of the deployed manifests, so you can easily ask it to return to the previous version if a problem comes up with the new one, and it does it automatically if the version you asked it to deploy does not start in some timeout (as per defined health checks). But writing the manifests, called charts (), is pain in the stern.

Because, you see, the manifests are by convention written in YAML. They can also be written in JSON, and are converted to JSON for sending over the API anyway, but since standard JSON does not even support comments, nobody does that and instead deals with the zillions of ways anything can be done in YAML.

But there are many environment-specific details in the manifest like the domain name to serve on the reverse proxy and the container registry as each cloud has its own. So we need to somehow make it generic.

Now all this stuff is written in go, and go has this template engine in its standard library. It is not based on mustache. It uses similarish syntax with {{}}, but the templates are actually logicfull and Turing-complete. And it turns out you need to template about ⅘ of the values in there actually. So the file ends up looking like
```
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "appname.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: {{ $fullName }}
  labels:
    {{- include "appname.labels" . | nindent 4 }}
  {{- with .Values.ingress.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  {{- if .Values.ingress.tls }}
  tls:
    {{- range .Values.ingress.tls }}
    - hosts:
        {{- range .hosts }}
        - {{ . | quote }}
        {{- end }}
      secretName: {{ .secretName }}
    {{- end }}
  {{- end }}
  rules:
    {{- range .Values.ingress.hosts }}
    - host: {{ .host | quote }}
      http:
        paths:
          {{- range .paths }}
          - path: {{ . }}
            backend:
              serviceName: {{ $fullName }}
              servicePort: {{ $svcPort }}
          {{- end }}
    {{- end }}
  {{- end }}
```
… that's the default template for “Ingress”, the reverse proxy configuration (spat out by helm create). Note especially the
- Careful use of - to trim whitespace before/after the templates.
- Use of nindent to indent the produce text the right number of columns to properly align the yaml blocks
  - nindent is a “newline indent” that additionally to indenting the text prepends a newline to it, because the one in the input was eaten by the {{- . Before nindent was introduced, all the substitutions expanding to complex yaml had to start on the start of line with just {{, additionally throwing off the indentation.
- Naming conventions mixing, because the content of .Values is the environment-specific values to be substituted, which are by convention camelCase, but the rest of the context is PascalCase, because in go things are made public by having name starting with capital letter.
- General unreadability of the resulting mess.
Use plain kubectl apply and custom deployment scripts. Reacentlysh kubectl included kompose, which allows you to write a ‘patch’ that will modify the base manifest, so you can create a readable base manifest and a mostly readable set of overrides for the environment and it generally feels like a reasonable approach to the problem.

Except, well, you lose the ability to track history. Kubectl only remembers the previous version for the purpose of merging changes when applying newer release, but it can't revert and does not watch the installation and nothing like that.
I've seen another tool which I think would be actually reasonable for doing the templating: jsonnet. But I guess none of the Kubernetes developers ever heard of it. Instead, they are talking about adding support for writing the templates in Lua. Right….

Ok, now back to trying to write some logic in that abomination of a template language.

Edit: hm, actually some kubernetes developers did hear about jsonnet. There even is a tool for it, ksonnet. The disadvantage is dropping the useful management features of helm.

Further edit: there is also k14s/kapp which seems to add the management. Writing the rant seems to have been a good idea as it prodded me to go search for alternatives one more time.

Steve_The_Cynic

@Bulb said in On the rationality of using go templates for generating yaml:

Edit: hm, actually some kubernetes developers did hear about jsonnet. There even is a tool for it, ksonnet. The disadvantage is dropping the useful management features of helm.

I gots ta know... Are those "JSON-net" and "KSON-net" or "j-sonnet" and "k-sonnet"? And am I going to regret asking the question?

Bulb

@Steve_The_Cynic They are

JSON
 sonnet

and

jksonnet

Steve_The_Cynic

@Bulb OK, thanks. I think I'll take that as a "yes, you're going to regret asking"..

dkf

FWIW, absolutely nothing to do with YAML is sane or rational, let alone safe to generate without extreme care. It has far too many weird edge cases; it's more like a programming language for generating a graph, and not one that was carefully designed for generatability either.

dfdub

@dkf Which is why you should never try to generate idiomatic YAML, but just feed a JSON document to the YAML parser. It's easy to generate and Just Works™.

Bulb

@dkf said in On the rationality of using go templates for generating yaml:

FWIW, absolutely nothing to do with YAML is sane or rational

No, it doesn't. The part that isn't rational is using go templates with it. Or to generate anything structured, really. Structured content should be generated by template engines that generate structure like genshi (for XML), jsonnet (for JSON), ytt (for YAML) etc., not ones that generate text.