Deploy with gitops

info

Feature available with Dalim 4.X+ on early adopter

Manage your deployment with flux

GitOps on k8saas subscription is based on flux. For a short introduction GitOps it's like doing DevOps with git as source of truth. Flux work as a pull-based system within your k8saas cluster. This core functionality allows your system to always be in sync with the state described in a Git repository.

Putting in place gitops approach requires to solve some challenges:

Security: GitOps revolves around using Git as the central source of truth, which means sensitive data like secrets might end up being stored in Git. It's crucial to encrypt those secrets and manage decryption keys securely.
Storing Secrets: Secrets should never be stored in plaintext in the git repository. Always use a secure method to store credentials. Kubernetes provides Secrets as a resource, or third-party solutions exist such as Helm Secrets, AWS Secrets Manager, or HashiCorp Vault.
Segregation of Permissions: It's crucial to manage roles and responsibilities precisely. Implementing a Role-Based Access Control (RBAC) model can help differentiate permissions based on job requirements. Also, use the least privilege principle where users get minimum levels of access necessary to perform their jobs.
Monitoring & Troubleshooting: Using a monitoring tool will help you keep track of system health and detect anomalies or security incidents. Set up logging for auditability of actions and system state. If anything goes wrong, logs can be a vital resource for troubleshooting and finding remediation steps.
Repository architecture: A mono-repo or multi-repo style can be used, each having their own pros and cons.

Getting Starterd

Setup tools

Install kustomize

on linux or macos Install kustomize

brew install kustomize

on windows

choco install kustomize

Install flux cli

on linux or macos Install flux cli

brew install fluxcd/tap/flux

on windows

choco install flux

Install sops

follow official documentation

Demo repositories

repositories used for following demonstration are located here

First step with gitops

This sample demonstrate a simple use of flux with one repository and 2 environments dev and qa

Create fork project platform-team-canada/k8saas-innersource/gitops-samples/basic-sample

create access token with required permissions
- scope: read_repository
- role: maintainer
create access token
- export value into GIT_TOKEN

export GIT_TOKEN=XXXXX

You can also create access token at group level

gitlab official documentation: group access token

Inspect sample repository
repository should look like:

   ├── apps
   │   └── simple-hello-world
   │       ├── configmap.yaml
   │       ├── ingress.yaml
   │       ├── kustomization.yaml
   │       └── manifest.yaml
   └── envs
       ├── dev
       │   ├── kustomization.yaml
       │   └── patches
       │       └── patch-ingress.yaml
       └── qa
           ├── kustomization.yaml
           └── patches
               ├── patch-config.yaml
               ├── patch-deployment.yaml
               └── patch-ingress.yaml

Description of repository

apps: This folder contains the infrastructure of application simple-hello-world
- simple-hello-world: application folder.
  - configmap.yaml: This file defines the ConfigMaps for application.
  - ingress.yaml: This file defines the Ingress for application.
  - manifest.yaml: This file contains the manifests of application
  - kustomization.yaml: This file identify all object that will be rendered
  - envs: This folder contains the specific environments of your project.
  - dev: default configuration
  - qa:
    - kustomization.yaml: This file is used by Kustomize to customize the configurations for the specific environment.
    - patches: This folder contains patch files that are used to modify resources for a specific environment.
      - patch-ingress.yaml, patch-config.yaml, patch-deployment.yaml: These files contain environment-specific modifications for the corresponding resources.

Setup flux git repository

create git secret

kubectl create secret generic -n customer-namespaces git-token --from-literal=username=gittoken --from-literal=password=$GIT_TOKEN

create git repository

flux create source git -n customer-namespaces hello-world-sample --url <fork_git_url> --branch=main --secret-ref=git-token

--url: flag specifies the URL of the Git repository. You should replace <fork_git_url> with the actual repository path that you want to refer to.
--branch flag tells Flux to track the main branch of the Git repository.
--secret-ref flag is used to refer to the Kubernetes Secret git-token that contains credential details used to authenticate against the Git repository.
Verify information about source repository

flux get source git -n customer-namespaces hello-world-sample

Init dev namespace

Before deploying application with flux some prerquisites are required to allow flux controller to deploy application.

Create dev namespace and assign devops cluster role to service account k8saas-generic-sa-cicd

kubectl hns create dev -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n dev

Edit configuration
- Into forked repository edit the file patches/patch-ingress.yaml in dev environment to replace CLUSTER-NAME by the name of your k8saas instance.
- Then check modifcation by running kustomize command:

kustomize build . --load-restrictor LoadRestrictionsNone   | grep "host:

finaly commit and push your changes
setup dev environment

flux create ks -n customer-namespaces  dev-system --source=GitRepository/hello-world-sample--path envs/dev --service-account k8saas-generic-sa-cicd --prune true --target-namespace dev

--source flag is used to specify the source for the Kustomization. In this instance, it's using a GitRepository source with the name 'hello-world-sample'.
--path flag specifies the path in the aforementioned Git repository where the Kustomization manifests are stored. Here, it is pointing to the 'envs/dev' directory.
--service-account flag is used to specify the name of the Kubernetes Service account to be used for this Kustomization.
--prune flag is used to remove resources from the cluster that are not present in the Git repository. This flag ensures that the cluster state always mirrors the state described in Git.
--target-namespace flag specifies the namespace to which the resources will be applied in the cluster.
Verify kustomization state

flux get ks  -n customer-namespaces  dev-system

Verify object deployed by kustomization

With flux cli:

flux tree ks -n customer-namespaces dev-system

With kubectl:

kubectl get ks  -n customer-namespaces dev-system -o jsonpath="{.status.inventory}" | jq

browse the ingress

Setup qa namespace

Create qa namespace ans assign devops cluster role to k8saas-generic-sa-cicd

kubectl hns create qa -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role   --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n qa

Edit configuration
- Into forked repository edit the file patches/patch-ingress.yaml in qa environment to replace CLUSTER-NAME by the name of your k8saas instance.
- Then check modifcation by runing kustomize command:

kustomize build . --load-restrictor LoadRestrictionsNone   | grep "host:

finaly commit your changes
setup qa nevironment

flux create ks -n customer-namespaces  qa-system --source=GitRepository/hello-world-sample--path envs/qa   --service-account k8saas-generic-sa-cicd --prune true --target-namespace qa

Verify kustomization state

flux get ks  -n customer-namespaces  qa-system

Verify object deployed by kustomization

With flux cli:

flux tree ks -n customer-namespaces qa-system

With kubectl:

kubectl get ks  -n customer-namespaces qa-system -o jsonpath="{.status.inventory}" | jq

browse the ingress
Conclusion:

In this basic sample we've shown:

how to prepare k8saas environment to use gitops
how to interact with flux with cli
how to use overlay to override some values with kustomize

As you may notice, we did not use any git flow in this section. Indeed monorepo does not make it easy to use the git flow approach, as environment are managed by folder and can introduce drifts between branches. In following section we will discuss of different approach to structure git repo to have better separation of concern between applications infrastructure and configuration. Secret management with gitopsis antother key point that has not been discussed

Git topologies

Coming Soon

Work with flux

Flux objects

the main flow resources used in a project are:

1.Source

Source represent a repository where your applications infrastructure or configuration redise. Often it's a git repository. Flux support other type of repositories:

bucket: allow to declate bucket as the source. In case of k8saas, bucket is blob stotage account

Dalim 4.1
Dalim 4.2+

     ---
     apiVersion: source.toolkit.fluxcd.io/v1beta2
     kind: Bucket
     metadata:
       name: azure-service-principal-secret
       namespace: default
     spec:
       interval: 5m0s
       provider: azure
       bucketName: <bucket-name>
       endpoint: https://<account-name>.blob.core.windows.net
       secretRef:
         name: azure-sp-auth
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: azure-sp-auth
       namespace: default
     type: Opaque
     data:
       tenantId: <BASE64>
       clientId: <BASE64>
       clientSecret: <BASE64>

gitrepository: use git as source

     apiVersion: source.toolkit.fluxcd.io/v1
     kind: GitRepository
     metadata:
       name: podinfo
       namespace: default
     spec:
       interval: 5m0s
       url: https://github.com/stefanprodan/podinfo
       ref:
         branch: master

helmrepository: use helm2 repository or oci

       apiVersion: source.toolkit.fluxcd.io/v1beta1
       kind: HelmRepository
       metadata:
         name: bitnami
         namespace: flux-system
       spec:
         interval: 10m
         url: https://charts.bitnami.com/bitnami

helmchart: helmchart is not used directly. helmchart object is used in helmrelease object below chart property.

       apiVersion: helm.toolkit.fluxcd.io/v2beta1
       kind: HelmRelease
       metadata:
         name: frontend
         namespace: default
       spec:
         interval: 5m
         chart: # <<< represent helmchart object
           spec:
             chart: podinfo
             version: ">=4.0.0 <5.0.0"
             sourceRef:
               kind: HelmRepository
               name: podinfo
               namespace: default
             interval: 1m

ocirepositories: use oci repository like azure container registry. For now only kubelet managed idenity is supported

         apiVersion: source.toolkit.fluxcd.io/v1beta2
         kind: OCIRepository
         metadata:
           name: podinfo
           namespace: default
         spec:
           interval: 5m0s
           url: oci://ghcr.io/stefanprodan/manifests/podinfo
           ref:
             tag: latest

     ---
     apiVersion: source.toolkit.fluxcd.io/v1beta2
     kind: Bucket
     metadata:
       name: azure-service-principal-secret
       namespace: default
     spec:
       interval: 5m0s
       provider: azure
       bucketName: <bucket-name>
       endpoint: https://<account-name>.blob.core.windows.net
       secretRef:
         name: azure-sp-auth
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: azure-sp-auth
       namespace: default
     type: Opaque
     data:
       tenantId: <BASE64>
       clientId: <BASE64>
       clientSecret: <BASE64>

gitrepository: use git as source

     apiVersion: source.toolkit.fluxcd.io/v1
     kind: GitRepository
     metadata:
       name: podinfo
       namespace: default
     spec:
       interval: 5m0s
       url: https://github.com/stefanprodan/podinfo
       ref:
         branch: master

helmrepository: use helm2 repository or oci

       apiVersion: source.toolkit.fluxcd.io/v1beta1
       kind: HelmRepository
       metadata:
         name: bitnami
         namespace: flux-system
       spec:
         interval: 10m
         url: https://charts.bitnami.com/bitnami

helmchart: helmchart is not used directly. helmchart object is used in helmrelease object below chart property.

       apiVersion: helm.toolkit.fluxcd.io/v2beta2
       kind: HelmRelease
       metadata:
         name: frontend
         namespace: default
       spec:
         interval: 5m
         chart: # <<< represent helmchart object
           spec:
             chart: podinfo
             version: ">=4.0.0 <5.0.0"
             sourceRef:
               kind: HelmRepository
               name: podinfo
               namespace: default
             interval: 1m

ocirepositories: use oci repository like azure container registry. For now only kubelet managed idenity is supported

         apiVersion: source.toolkit.fluxcd.io/v1beta2
         kind: OCIRepository
         metadata:
           name: podinfo
           namespace: default
         spec:
           interval: 5m0s
           url: oci://ghcr.io/stefanprodan/manifests/podinfo
           ref:
             tag: latest

Kustomize controller

This controller is in charge of:

Reconciles the cluster state from multiple sources (provided by source-controller)
Generates manifests with Kustomize (from plain Kubernetes YAMLs or Kustomize overlays)
Validates manifests against Kubernetes API
Prunes objects removed from source (garbage collection)
Impersonates service accounts (multi-tenancy RBAC)
Decrypts Kubernetes secrets with Mozilla SOPS and KMS
Health assessment of the deployed workloads
Runs pipelines in a specific order (depends-on relationship)
Reports cluster state changes (alerting provided by notification-controller)

The k8saas installation uses service account impersonation to guarantee isolation between the k8saas deployment and the client. This is why you need to create a service account and associate it with a rolebinding on the namespace in which you want to deploy your application. If no service account is specified, the controller takes the default one. In this setup crossnamepace operation context and all flux object reside in the namespace.

Another point take in consideration kustomize contoller only reconcile objects managed by itself ( found in inventory section )

Helm controller

Work with secrets

Managing secret with gitops in pull mode is one point of attention. GitOps, as a methodology, focuses on using Git as the single source of truth for your infrastructure and application definitions. bellow we describe two approach

With SOPS and shared key SOPS: Secrets OPerationS is an encryption tool that supports multiple format file such as yaml, json, ini format with Azure Key Vault. SOPS use AES256 meaning one encryption key is used to encrypt and decrypt. Integration with azure keyvault allow to operator to encrypt data on their side, push encoded secret into git and at reconcillation time flux decrypt secret and apply manifest if needed.

In k8saas setup kustomize controller use workload id to access to the key in keyvault.

First you need an azure key vault to store the key.

      keyvault="k8saas-keyvault-name"
      resource_group="k8saas-cluster-name"
      sops_key="sops-key"
      az keyvault create -g "$resource_group"  -n "$keyvault"
      az keyvault key create --name "$sops_key" \
        --vault-name "${keyvault}" \
        --protection software \
        --ops encrypt decrypt

Asssign permission decrypt to workload id

     cluster_resource_group=""
     IDENTITY_NAME="${cluster_resource_group}-flux-wid"
     export IDENTITY_ID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query principalId)"
     # depending on permission model, here vault was build with 'Vault access policy'
     az keyvault set-policy --name ${keyvault} --object-id ${IDENTITY_ID} --key-permissions decrypt
     sops_key_id="$(az keyvault key show --name "$sops_key"  --vault-name "$keyvault" --query key.kid -otsv)"

Create sops definition file .sops.yaml on root directory of project like

  cat <<EOF >>.sops.yaml
   creation_rules:
     - path_regex: "envs/.*"
       azure_keyvault: "$sops_key_id"
       encrypted_regex: ^(data|stringData)$
  EOF

fork repository k8saas-innersource/gitops-samples/secrets-sample

In this version we used a kubernetes secret to store the welcome message

Encrypt exiting file envs/qa/patches/patch-secret.yaml

sops --encrypt --in-place envs/qa/patches/patch-secret.yaml

verify that file is correcly encoded

cat envs/qa/patches/patch-secret.yaml

   apiVersion: v1
   kind: Secret
   metadata:
       name: aks-helloworld-one
   data:
       title: ENC[AES256_GCM,data:tpRm7Wvqxc5YhzvoziCiyRVh40tbOz14oBSKbVcxdHWlNwpKDNY=,iv:1DZF/7Iyg5hrquN22nbJwQz8V3P+7qvR76kFYTl8aZ tag:ObXaChrurfbYyURthamNgg==,type:str]
   sops:
       kms: []
       gcp_kms: []
       azure_kv:
           - vault_url: https://XXXXXXXX.vault.azure.net
             name: sops-key
             version: 3bf1e70f408e4736ac7fadfe67ad2bdc
             created_at: "2023-11-13T16:35:34Z"
             enc:      sdY6KkZTe900dniUIZaNgEE0Q6L-iP9O2gaakHGdQqUbkBeReakdzFKkJ2Z_lma1JcPRaHe0sImiCGT4A15yNvR2z-uewoCe0eNJew7hHQoFWTePvvnzQld7HcZzGO3PU07K OS8KMXiWgej-kQlSvXT_m-gjV     azHc1RCFUTMTdJtPVA5dCLsxbGEvYt5d_I9vTCyKyfLzH40bDO-vAaJ89A0luiCFu9PWFEzyx2cbkO0tZOwb6V-UKW6fAYzXbEpXhmAYN6MUSjDJO4Q98vf5FLn4554gWKUs wLcg2xu4HO9GzfnGqWFROSKqa       0bPQcFx8RqhOe0ixmkX7rQ
       hc_vault: []
       age: []
       lastmodified: "2023-11-13T16:35:37Z"
       mac: ENC[AES256_GCM,data:X7IwFaXc2wjdiqz1IXMLpkrVd84qGDVffFdead5g8bgJ5C9vvf/lrAQE8w2dYV83es0WKkA1Rmu3agGadLp4LaUmMdnyK oe3vVjHS2iIYGO33uOioOh4UM7qoolJM3lqO4L+zjjeK       +pH+ImFVMWsr9u9s8z7pRiH99hcJok4=,iv:NiCQac+FzMy9PtS2Z3KGrU8a2MjX+IbOyIP1Twn0Jpo=,tag:s CeY3xtqfxRsJAw7YAw==,type:str]
       pgp: []
       encrypted_regex: ^(data|stringData)$
       version: 3.7.3

Commit and push
Setup new namespace 'qa-secret'

   ns="qa-secret"
   kubectl hns create "$ns" -n customer-namespaces
   kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role   --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n "$ns"

setup flux environment
Create new git access token

   kubectl create secret generic -n customer-namespaces git-token-secret --from-literal=username=gittoken --from-literal=password=$GIT_TOKEN

Create gitrepository

  flux create source git -n customer-namespaces hello-world-sample-secret  --url <fork_git_url> --branch=main --secret-ref=git-token-secret

Create environment qa-secret

   flux create ks -n customer-namespaces  "$ns-system" --source=GitRepository/hello-world-sample-secret --path envs/qa   --service-account k8saas-generic-sa-cicd --prune true --target-namespace "$ns" --decryption-provider sops

Create port-forward then browse service

  kubectl port-forward -n qa-secret svc/aks-helloworld-one 8081:80

Open url localhost:8081 with a browser

Rotate secret in git

  sops envs/qa/patches/patch-secret.yaml

With Azure KeyVault Provider

Azure Key Vault Provider (AKVP) for Kubernetes secret store CSI Driver allows you to get secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into Kubernetes pods. Mounts secrets/keys/certs to pod using a CSI Inline volume.

Major difference with previous method is secrets are no longer managed by git.

Like in previous section we'll need a keyvault to store secret.

Add secret into keyvault

   keyvault="vltgitops123"
   resource_group="<cluster-name>"
   secret_value_base64=$(echo -n "AKVP secret message"| base64)
   az keyvault secret set --name secretMessage --vault-name "$keyvault" --value "$secret_value"

Secret has to be encoded in base64 because it will be sync with kubernetes secret at the end

Add permission to azure identity to read our secret. An identity <cluster-name>-customer-workload-id is already deployed and usable from customer-namespaces

   IDENTITY_NAME="${cluster_resource_group}-customer-workload-id"
   export IDENTITY_ID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query principalId)"
   az keyvault set-policy --name ${keyvault} --object-id ${IDENTITY_ID} --secret-permissions get list

Now we have defined how secret will be used by the application. Indeed 2 choice are possible:
1. mount secret directly into application using CSI Inline volume. To used this methode application have to read secret from local file. Main inconvenient of this method when secret is updated into keyvault, application is not aware of changes and has to watch secret for changes
2. Second option is to sync keyvault secret with kubernetes secret and use kustomize post build feature to reload application. This scenario that will be cover in following
Create SecretProviderClass

  IDENTITY_NAME="${cluster_resource_group}-customer-workload-id"
  export IDENTITY_ClientID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query clientId)"
  export tenantId="$(az account show --query tenantId -otsv)"
  cat<<EOF >>secretprovideclass.yaml
  apiVersion: secrets-store.csi.x-k8s.io/v1
  kind: SecretProviderClass
  metadata:
    name: azure-sync
  spec:
    parameters:
     clientID: $IDENTITY_ClientID
     keyvaultName: $keyvault
     tenantId: $tenantId
     usePodIdentity: "false"
     useVMManagedIdentity: "false"
     objects: |
       array:
       - |
         objectName: secretMessage
         objectType: secret
         objectVersion: ""
    provider: azure
    secretObjects:
    - data:
      - key: title   # map secretMessage keyvault secret to title key in aks-helloworld-one secret
        objectName: secretMessage
      secretName: aks-helloworld-one
      type: Opaque
  EOF
  kubectl apply -n customer-namespaces -f secretprovideclass.yaml

Create a deployemt to mount secret, this step is required to sync azure keytvault secret with kubernetes

  cat<<EOF >> secret-sync-deployment.yaml
  kind: Deployment
  apiVersion: apps/v1
  metadata:
    name: secret-sync-deployment
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: secret-sync-deployment
    template:
     metadata:
       labels:
         app: secret-sync-deployment
       name: secret-sync-deployment
     spec:
       serviceAccountName: k8saas-customer-sa-workload-id
       containers:
         - name: busybox
           image: docker.io/busybox:1.35.0
           resources:
            requests:
             cpu: 20m
             memory: 32Mi
           command:
             - "/bin/sleep"
             - "10000"
           volumeMounts:
           - name: secrets-store01-inline
             mountPath: "/mnt/secrets-store"
             readOnly: true
       volumes:
         - name: secrets-store01-inline
           csi:
             driver: secrets-store.csi.k8s.io
             readOnly: true
             volumeAttributes:
               secretProviderClass: azure-sync
  EOF
  kubectl apply -n customer-namespaces -f secret-sync-deployment.yaml

Verify that secret is created under namespace customer-namespaces

kubectl get secret -n customer-namespaces aks-helloworld-one  -ojsonpath="{.data.title}" | base64 -d

Under repo gitops-samples-secrets switch to the branch akvp

git checkout -b akvp

review changes

  git diff main..akvp

Create new environment named qa-akvp that use branch akvp
prepare env

  ns="qa-akvp"
  kubectl hns create "$ns" -n customer-namespaces
  kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role   --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n  "$ns"
  flux create source git -n customer-namespaces hello-world-sample-secret-akvp  --url <fork_git_url> --branch=akvp --secret-ref=git-token-secret

Create environment

  cat<<EOF >> ks-postbuild-manifest.yaml
  apiVersion: kustomize.toolkit.fluxcd.io/v1
  kind: Kustomization
  metadata:
    name: qa-akvp-system
    namespace: customer-namespaces
  spec:
    interval: 1m0s
    path: ./envs/qa
    prune: true
    serviceAccountName: k8saas-generic-sa-cicd
    sourceRef:
      kind: GitRepository
      name: hello-world-sample-secret-akvp
    targetNamespace: qa-akvp
    postBuild:
      substituteFrom:
      - kind: Secret
        name: aks-helloworld-one

  EOF
  kubectl apply -n customer-namespaces -f ks-postbuild-manifest.yaml

inspect secret into qa-akvp namespace

  kubectl get secret -n qa-akvp aks-helloworld-one -ojsonpath="{.data.title}" |base64 -d

Create port-forward then browse service

  kubectl port-forward -n qa-akvp svc/aks-helloworld-one 8081:80

Open url localhost:8081 with a browser

Rotate secret in keyvault and observe event in customer-namespaces and qa-akvp

Known limitations

Controllers ImageRepository, ImagePolicy, ImageUpdateAutomation are not available
not UI to Follow object like argoUI, Weave UI is still undeer experiment on our side

troubleshooting

Error message	Solution
helmreleases.helm.toolkit.fluxcd.io "keda" is forbidden: User "system:serviceaccount:flux-system:default" cannot patch resource "helmreleases" in API group "helm.toolkit.fluxcd.io" in the namespace "flux-system"	Add service account name field in the kustomization's spec & the inherited Flux resources
econciliation failed: failed to get last release revision: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:flux-system:default" cannot list resource "secrets" in API group "" in the namespace "flux-system"	Add the service account name spec in your helm release definition file and all the Flux resources
can't access 'XXX/XXX/XXX', cross-namespace references have been blocked	All Flux resources should be defined in the same namespace, but you can specify a deployment destination namespace by adding the spec targetNamespace in your Flux resource definition file

more cheatsheet

Manage your deployment with flux​

Getting Starterd​

Setup tools​

Demo repositories​

First step with gitops​

Git topologies​

Work with flux​

Flux objects​

Work with secrets​

Known limitations​

troubleshooting​