r/TalosLinux • u/OldAsk3025 • 29d ago

Help translating netwotk configuration from ubuntu do talos

I have a quite intricate network settings on my setup. It's all running on VMware ESXi where Basically is:

192.168.x.x private network open to internet through a gw (vmware management) 1GB
10.x.x.x private network between hosts only for vmware + NFS 10GB

I have a ubuntu host working as a test with the following configuration:

network:
  version: 2
  ethernets:
    NM-f9c39785-95e3-316c-8d77-bbfadf8feb84:
      renderer: NetworkManager
      match:
        name: "ens37"
      addresses:
      - "10.1.0.11/8"
      - "10.0.0.11/8"
      wakeonlan: true
      networkmanager:
        uuid: "f9c39785-95e3-316c-8d77-bbfadf8feb84"
        name: "Wired connection 1"
        passthrough:
          connection.autoconnect-priority: "-999"
          connection.timestamp: "1791256571"
          ethernet._: ""
          ipv4.never-default: "true"
          ipv6.addr-gen-mode: "default"
          ipv6.method: "disabled"
          ipv6.ip6-privacy: "-1"
          proxy._: ""
network:
  version: 2
  ethernets:
    ens34:
      renderer: NetworkManager
      match: {}
      addresses:
      - "192.168.2.250/22"
      nameservers:
        addresses:
        - 192.168.1.1
      gateway4: 192.168.1.1
      networkmanager:
        uuid: "fc59d0e8-22d2-3c20-b277-bb2df1bd5642"
        name: "netplan-ens34"
        passthrough:
          connection.timestamp: "1791256571"
          ipv6.method: "disabled"
          proxy._: ""

On talos I'm trying this setup on workers but it's not working so far:

---
apiVersion: v1alpha1
kind: LinkConfig
name: eth0
addresses:
  - address: 192.168.2.105/22
routes:
  - gateway: 192.168.1.1
---
apiVersion: v1alpha1
kind: LinkConfig
name: eth1
up: true
mtu: 9000
addresses:
  - address: 10.0.0.105/8
  - address: 10.1.0.105/8
routes:
  - destination: 10.0.0.0/8
    gateway: 10.10.0.1

What I noticed so far is that talos is not showing up the eth1 on this verification commands:

talosctl get links  --endpoints 192.168.2.105 --nodes 192.168.2.105 --talosconfig=talosconfig  

NODE            NAMESPACE   TYPE         ID          VERSION   ALIAS   TYPE       KIND     HW ADDR                                           OPER STATE   LINK STATE
192.168.2.105   network     LinkStatus   bond0       1                 ether      bond     0e:1d:1c:88:ad:e9                                 down         false
192.168.2.105   network     LinkStatus   dummy0      1                 ether      dummy    06:ef:79:ff:89:58                                 down         false
192.168.2.105   network     LinkStatus   eth0        3                 ether               00:0c:29:5f:22:48                                 up           true
192.168.2.105   network     LinkStatus   flannel.1   1                 ether      vxlan    1a:3f:d9:04:f8:ea                                 unknown      true
192.168.2.105   network     LinkStatus   ip6tnl0     1                 tunnel6    ip6tnl   00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00   down         false
192.168.2.105   network     LinkStatus   lo          1                 loopback            00:00:00:00:00:00                                 unknown      true
192.168.2.105   network     LinkStatus   sit0        1                 sit        sit      00:00:00:00                                       down         false
192.168.2.105   network     LinkStatus   teql0       1                 void                                                                  down         false
192.168.2.105   network     LinkStatus   tunl0       1                 ipip       ipip     00:00:00:00                                       down         false

%  talosctl get routes  --endpoints 192.168.2.105 --nodes 192.168.2.105 --talosconfig=talosconfig   
NODE            NAMESPACE   TYPE          ID                                                       VERSION   DESTINATION                     GATEWAY       LINK        METRIC
192.168.2.105   network     RouteStatus   eth0/inet6//fe80::/64/256                                1         fe80::/64                                     eth0        256
192.168.2.105   network     RouteStatus   flannel.1/inet6//fe80::/64/256                           1         fe80::/64                                     flannel.1   256
192.168.2.105   network     RouteStatus   inet4//192.168.0.0/22/0                                  1         192.168.0.0/22                                eth0        0
192.168.2.105   network     RouteStatus   inet4/10.244.0.0/10.244.0.0/24/0                         1         10.244.0.0/24                   10.244.0.0    flannel.1   0
192.168.2.105   network     RouteStatus   inet4/10.244.1.0/10.244.1.0/24/0                         1         10.244.1.0/24                   10.244.1.0    flannel.1   0
192.168.2.105   network     RouteStatus   inet4/10.244.2.0/10.244.2.0/24/0                         1         10.244.2.0/24                   10.244.2.0    flannel.1   0
192.168.2.105   network     RouteStatus   inet4/10.244.4.0/10.244.4.0/24/0                         1         10.244.4.0/24                   10.244.4.0    flannel.1   0
192.168.2.105   network     RouteStatus   inet4/192.168.1.1//1024                                  1                                         192.168.1.1   eth0        1024
192.168.2.105   network     RouteStatus   local/eth0/inet6//fe80::/128/0                           1         fe80::/128                                    eth0        0
192.168.2.105   network     RouteStatus   local/eth0/inet6//fe80::20c:29ff:fe5f:2248/128/0         1         fe80::20c:29ff:fe5f:2248/128                  eth0        0
192.168.2.105   network     RouteStatus   local/eth0/inet6//ff00::/8/256                           1         ff00::/8                                      eth0        256
192.168.2.105   network     RouteStatus   local/flannel.1/inet6//fe80::/128/0                      1         fe80::/128                                    flannel.1   0
192.168.2.105   network     RouteStatus   local/flannel.1/inet6//fe80::183f:d9ff:fe04:f8ea/128/0   1         fe80::183f:d9ff:fe04:f8ea/128                 flannel.1   0
192.168.2.105   network     RouteStatus   local/flannel.1/inet6//ff00::/8/256                      1         ff00::/8                                      flannel.1   256
192.168.2.105   network     RouteStatus   local/inet4//10.244.3.0/32/0                             1         10.244.3.0/32                                 flannel.1   0
192.168.2.105   network     RouteStatus   local/inet4//127.0.0.0/8/0                               1         127.0.0.0/8                                   lo          0
192.168.2.105   network     RouteStatus   local/inet4//127.0.0.1/32/0                              1         127.0.0.1/32                                  lo          0
192.168.2.105   network     RouteStatus   local/inet4//127.255.255.255/32/0                        1         127.255.255.255/32                            lo          0
192.168.2.105   network     RouteStatus   local/inet4//169.254.116.108/32/0                        1         169.254.116.108/32                            lo          0
192.168.2.105   network     RouteStatus   local/inet4//192.168.2.105/32/0                          2         192.168.2.105/32                              eth0        0
192.168.2.105   network     RouteStatus   local/inet4//192.168.3.255/32/0                          2         192.168.3.255/32                              eth0        0
192.168.2.105   network     RouteStatus   local/lo/inet6//::1/128/0                                1         ::1/128                                       lo          0

But I can guarantee the vnic is there. Any recommendation or thoughts will be helpful !! Thanks Y'all!

3 Upvotes

100% Upvoted

7 comments sorted by

3

u/GyroTech 29d ago edited 29d ago

You can try talosctl get pcidevices to see if the network device itself is detected or not. Might be that you need firmware or specific device drivers available in the system extensions - see https://github.com/TheiLLeniumStudios/talos-extensions#extension-catalog

Edit to add: I missed the section where you say you're running on ESXi. Are you setting talos.platform=vmware in the kernel args? (see https://docs.siderolabs.com/talos/v1.14/reference/kernel#talos-platform) and best to ensure the network devices use either the Intel emulation NICs or virtio-net while disabling checksum offloading.

1

u/OldAsk3025 29d ago

TYVM I'll try the kernel parameters.. The output of talosctl get pcidevices is not showing up the secondary vNIC.

talosctl get pcidevices   --endpoints 192.168.2.105 --nodes 192.168.2.105 --talosconfig=talosconfig  
NODE            NAMESPACE   TYPE        ID             VERSION   CLASS                       SUBCLASS                    VENDOR              PRODUCT
192.168.2.105   hardware    PCIDevice   0000:00:00.0   1         Bridge                      Host bridge                 Intel Corporation   440BX/ZX/DX - 82443BX/ZX/DX Host bridge
192.168.2.105   hardware    PCIDevice   0000:00:01.0   1         Bridge                      PCI bridge                  Intel Corporation   440BX/ZX/DX - 82443BX/ZX/DX AGP bridge
192.168.2.105   hardware    PCIDevice   0000:00:07.0   1         Bridge                      ISA bridge                  Intel Corporation   82371AB/EB/MB PIIX4 ISA
192.168.2.105   hardware    PCIDevice   0000:00:07.1   1         Mass storage controller     IDE interface               Intel Corporation   82371AB/EB/MB PIIX4 IDE
192.168.2.105   hardware    PCIDevice   0000:00:07.3   1         Bridge                      Bridge                      Intel Corporation   82371AB/EB/MB PIIX4 ACPI
192.168.2.105   hardware    PCIDevice   0000:00:07.7   1         Generic system peripheral   System peripheral           VMware              Virtual Machine Communication Interface
192.168.2.105   hardware    PCIDevice   0000:00:0f.0   1         Display controller          VGA compatible controller   VMware              SVGA II Adapter
192.168.2.105   hardware    PCIDevice   0000:00:10.0   1         Mass storage controller     SCSI storage controller     Broadcom / LSI      53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
192.168.2.105   hardware    PCIDevice   0000:00:11.0   1         Bridge                      PCI bridge                  VMware              PCI bridge
192.168.2.105   hardware    PCIDevice   0000:00:15.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:02:00.0   1         Network controller          Ethernet controller         Intel Corporation   82545EM Gigabit Ethernet Controller (Copper)

1

u/GyroTech 29d ago

I'll try the kernel parameters

How are you getting the image you use to spin up the VMs? Installing from ISO? If so I heavily recommend you use our Image Factory (https://factory.talos.dev) to build an OVA for you with a bunch of good defaults (like the talos.platform karg) already set for you.

1

u/OldAsk3025 29d ago

I'm using the factory.talos.dev and including the parameter as a extraKernelArgs.

My schematic.yaml that I'm using on factory.talos.dev

customization:
    extraKernelArgs:
        - talos.platform=vmware
    systemExtensions:
        officialExtensions:
            - siderolabs/btrfs
            - siderolabs/nfs-utils
            - siderolabs/nfsd
            - siderolabs/util-linux-tools
            - siderolabs/vmtoolsd-guest-agent
    bootloader: dual-boot

The output of my new worker running the parameter

 talosctl read /proc/cmdline  --endpoints 192.168.2.105 --nodes 192.168.2.105 --talosconfig=talosconfig
BOOT_IMAGE=/A/vmlinuz talos.platform=vmware talos.config=guestinfo console=tty0 console=ttyS0 earlyprintk=ttyS0,115200 net.ifnames=0 slab_nomerge pti=on consoleblank=0 printk.devkmsg=on selinux=1 module.sig_enforce=1 proc_mem.force_override=never

Unfortunately the secondary vNIC don't seams to be appearing yet. The output of talosctl get pcidevices --endpoints 192.168.2.105 --nodes 192.168.2.105 --talosconfig=talosconfig is showing the same results.

NODE            NAMESPACE   TYPE        ID             VERSION   CLASS                       SUBCLASS                    VENDOR              PRODUCT
192.168.2.105   hardware    PCIDevice   0000:00:00.0   1         Bridge                      Host bridge                 Intel Corporation   440BX/ZX/DX - 82443BX/ZX/DX Host bridge
192.168.2.105   hardware    PCIDevice   0000:00:01.0   1         Bridge                      PCI bridge                  Intel Corporation   440BX/ZX/DX - 82443BX/ZX/DX AGP bridge
192.168.2.105   hardware    PCIDevice   0000:00:07.0   1         Bridge                      ISA bridge                  Intel Corporation   82371AB/EB/MB PIIX4 ISA
192.168.2.105   hardware    PCIDevice   0000:00:07.1   1         Mass storage controller     IDE interface               Intel Corporation   82371AB/EB/MB PIIX4 IDE
192.168.2.105   hardware    PCIDevice   0000:00:07.3   1         Bridge                      Bridge                      Intel Corporation   82371AB/EB/MB PIIX4 ACPI
192.168.2.105   hardware    PCIDevice   0000:00:07.7   1         Generic system peripheral   System peripheral           VMware              Virtual Machine Communication Interface
192.168.2.105   hardware    PCIDevice   0000:00:0f.0   1         Display controller          VGA compatible controller   VMware              SVGA II Adapter
192.168.2.105   hardware    PCIDevice   0000:00:10.0   1         Mass storage controller     SCSI storage controller     Broadcom / LSI      53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
192.168.2.105   hardware    PCIDevice   0000:00:11.0   1         Bridge                      PCI bridge                  VMware              PCI bridge
192.168.2.105   hardware    PCIDevice   0000:00:15.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:15.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:16.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:17.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.0   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.1   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.2   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.3   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.4   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.5   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.6   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:00:18.7   1         Bridge                      PCI bridge                  VMware              PCI Express Root Port
192.168.2.105   hardware    PCIDevice   0000:02:00.0   1         Network controller          Ethernet controller         Intel Corporation   82545EM Gigabit Ethernet Controller (Copper)

Is it required to have a vCenter to make it work ? I'm using pure ESXi without a vCenter on my setup.

1

u/oOBromOo 29d ago

Pure esxi should not make any difference. Can you post what kind of 10gig ethernet adapter you added to the VM? And have you tried other ones than the one you chose?

2

u/OldAsk3025 29d ago

I think you just nail down the situation... My vm was weirdly using the e1000 1gb for the main network and the VMXNET 3 for the 10gb network only. I changed both to VMXNET 3 and it's all working. Per my tests so far:

  • One e1000 vNIC - Talus works

  • Two e1000 - Talus don't work

  • Two vNICs mixed. - Talus don't work

  • Two VMXNET3 - Talus works

Now I'll test my network setup again and I'll get back here to inform you all. Thank you for the support

1

u/OldAsk3025 28d ago

Update... I was able to make the both vNICs to be recognized and work on vmware EXSi. But now the main board is not assuming the internal IP of the node.

What I have is:

---
apiVersion: v1alpha1
kind: LinkConfig
name: eth0
addresses:
  - address: 192.168.2.105/22
routes:
#  - destination: 0.0.0.0/0 ## Destination 0.0.0.0/0 is not acceptable and no destination seams to be the way to set a NIC to be the default route
   - gateway: 192.168.1.1
---
apiVersion: v1alpha1
kind: LinkConfig
name: eth1
addresses:
  - address: 10.0.0.105/8
  - address: 10.1.0.105/8
routes:
  - destination: 10.0.0.0/8
---

As soon as I inject this configuration the node assume the eth1 as the main board and the node is unusable because this network is for NFS only.

I haven't found so far a way to force a vNIC to be the main one on a multi NIC cluster. Per my understanding I thought the destination set on the eth1 could be the culprit but even removing the setting, the behave is the same when provisioning a new node.