# During Resource Enrollment

## cc-ansible --playbook ./playbooks/k3s.yml issues

### Unable to read /etc/rancher/k3s/k3s.yaml

```
kubectl get pods -A -o wide
WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions 
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied
```

To overcome this issue, add the following line to `roles/k3s/templates/config.yaml.j2`:

```
write-kubeconfig-mode: "0644"
```

### Wait till the Tigera Operator has fully applied

```
k3s : Wait till the Tigera Operator has fully applied
fatal: [edge -> localhost]: FAILED! => {"changed": false, "msg": "Failed to gather information about TigeraStatus(s) even after waiting for 123 seconds"}
```

By default the timeout for the tigera operator is 120 seconds. If you face this error, increase it 5 mins by adding `wait_timeout: 300` just after `wait: yes` in `roles/k3s/tasks/config-calico.yml` corresponding tasks. Such as below:

```
- name: Wait till the Tigera Operator has fully applied
  delegate_to: "{{ groups['deployment'][0] }}"
  kubernetes.core.k8s_info:
    kind: TigeraStatus
    name: calico
    namespace: default
    wait: yes
    wait_condition:
      type: "Available"
      status: "True"
      reason: "AllObjectsAvailable"
    wait_timeout: 300

- name: Wait till the Calico Installation is created
  delegate_to: "{{ groups['deployment'][0] }}"
  kubernetes.core.k8s_info:
    kind: Installation
    name: default
    namespace: default
    wait: yes
    wait_condition:
      type: "Ready"
      status: "True"
      reason: "AllObjectsAvailable"
    wait_timeout: 300
```

In addition to that, this must be run in a completly clean state.

### Change cluster ip CIDR

In case it is colliding with any of the used subnets, it is better to change it at `roles/k3s/defaults/main.yml`. More info could be find [here](https://docs.k3s.io/reference/server-config#networking).

### Apply Calico network policies

```
TASK [k3s : Apply Calico global network policies] **********************************************************************************************************************************************************
failed: [edge-mv -> localhost] (item=default-deny) => {"ansible_loop_var": "item", "changed": false, "item": "default-deny", "msg": "Failed to find exact match for projectcalico.org/v1.GlobalNetworkPolicy by [kind, name, singularName, shortNames]"}
failed: [edge-mv -> localhost] (item=allow-ping) => {"ansible_loop_var": "item", "changed": false, "item": "allow-ping", "msg": "Failed to find exact match for projectcalico.org/v1.GlobalNetworkPolicy by [kind, name, singularName, shortNames]"}
```

Solution according to [here](https://serverfault.com/questions/1076325/apply-calico-manifest-through-ansible-kubernetes-core-module) is to add `apply: yes` to the `task.kubernetes.core.k8s`.

### Configure Neutron

```
TASK [k3s : Create calico network] *************************************************************************************************************************************************************************
fatal: [edge-mv]: FAILED! => {"action": "os_network", "changed": false, "extra_data": {"data": null, "details": "Running without keystone AuthN requires that tenant_id is specified", "response": "{\"NeutronError\": {\"type\": \"HTTPBadRequest\", \"message\": \"Running without keystone AuthN requires that tenant_id is specified\", \"detail\": \"\"}}"}, "msg": "BadRequestException: 400: Client Error for url: http://10.0.87.254:9696/v2.0/networks, Running without keystone AuthN requires that tenant_id is specified"}
```

Solution: open the webportal, create Calico network, subnet, and router from there. OR, add the following line to 'Create calico network', 'Create calico subnet', and 'Fetch existing NAT router' tasks `<task>.kolla_toolbox.module_args` in `roles/k3s/tasks/config-neutron.yml`:

```
project: "{{ keystone_admin_project }}"
```

### Generate Calico/Neutron connection script

```
TASK [k3s : Generate Calico/Neutron connection script] *****************************************************************************************************************************************************
fatal: [edge-mv]: FAILED! => {"changed": false, "checksum": "0b5e563c74380eba5d02bb1a041e4703aac71e28", "msg": "Destination /etc/rancher/k3s not writable"}
```

The solution is to add `become: yes` to the task `Generate Calico/Neutron connection script` in `roles/k3s/tasks/config-neutron.yml`.

## NOTE: Run these when a new snapshot is loaded:

1. Configs

   ```
   sudo chown root:expeca-deploy /opt
   sudo chmod g+rw /opt
   ```
2. Add `enable_k3s: yes` to

   ```
   sudo vim /opt/site-config/defaults.yml
   ```
3. Add `wait_timeout: 300` to 'Wait till the Tigera Operator has fully applied' and 'Wait till the Calico Installation is created' tasks just after `wait: yes` in `roles/k3s/tasks/config-calico.yml`.
4. Change cluster ip CIDR at `roles/k3s/defaults/main.yml`.
5. Add `apply: yes` to the last task in `roles/k3s/tasks/config-calico.yml`.
6. Add the following line to 'Create calico network', 'Create calico subnet', and 'Fetch existing NAT router' tasks `<task>.kolla_toolbox.module_args` in `roles/k3s/tasks/config-neutron.yml`:

   ```
   project: "{{ keystone_admin_project }}"
   ```
7. Add `become: yes` to the task `Generate Calico/Neutron connection script` in `roles/k3s/tasks/config-neutron.yml`.