CoreOS DNS Discovery.

Hi Guys, This is the blog post for my CoreOS DNS Discovery video, I have been having some issues with deploying CoreOS in my XenServer environment because of a limitation within Xenserver or CoreOS that was preventing the environment variable $private_ipv4 from allowing me to configure to do a normal discovery setup like so,

#cloud-config

coreos:  
  etcd2:
    discovery: "https://discovery.etcd.io/<token>"
    advertise-client-urls: "http://$public_ipv4:2379"
    initial-advertise-peer-urls: "http://$private_ipv4:2380"
    listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001"
    listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001"

So after some time trying to figure out how to get discovery working one of the admins over at #coreos on freenode pointed me to this great piece of documentation.

That basically was exactly what I was looking for to configure Discovery. After some tweaking to get things running right I was able to successfully bootstrap my etcd2 cluster.

Here are the cleaned up Versions of the cloud-config I used to complete this task.

Bootstrap the First Node in the Cluster:

#cloud-config

hostname: %VMNAMETOHOSTNAME%  
ssh_authorized_keys:  
  - ssh-rsa <Your SSH Key Goes Here> # This is Required so that you can log into the machine over SSH on boot.
  - ssh-rsa %CONTAINERRSAPUB% # This is Required by the docker tools for container management in XenServer.
coreos:  
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

    # Hypervisor Linux Guest Agent
    - name: xe-linux-distribution.service
      command: start
      content: |
        [Unit]
        Description=Hypervisor Linux Guest Agent
        After=docker.service

        [Service]
        ExecStartPre=/media/configdrive/agent/xe-linux-distribution /var/cache/xe-linux-distribution
        ExecStart=/media/configdrive/agent/xe-daemon

  etcd2:
    name: %VMNAMETOHOSTNAME% # This Sets the Hostname for the Node
    discovery-srv: domain.com # This sets the Domain that discovery will take place on.

    # The Following settings are used for the same purpose as before only now they use DNS records to work out their IP's.
    advertise-client-urls: http://coreosnode1.domain.com:2379
    initial-advertise-peer-urls: http://coreosnode1.domain.com:2380
    listen-client-urls: http://0.0.0.0:2379 # This is always set to 0.0.0.0 to listen on all interfaces to make it easier to work with etcdctl and fleet.
    listen-peer-urls: http://coreosnode1.domain.com:2380

write_files:  
  # Enable ARP notifications for smooth network recovery after migrations
  - path: /etc/sysctl.d/10-enable-arp-notify.conf
    permissions: 0644
    owner: root
    content: |
      net.ipv4.conf.all.arp_notify = 1

Bootstrap Member Nodes after the initial node is up:

Refer to This Document for more Instructions on this process.

#cloud-config

hostname: %VMNAMETOHOSTNAME%  
ssh_authorized_keys:  
  - ssh-rsa <Your SSH Key Goes Here> # This is Required so that you can log into the machine over SSH on boot.
  - ssh-rsa %CONTAINERRSAPUB% # This is Required by the docker tools for container management in XenServer.
coreos:  
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

    # Hypervisor Linux Guest Agent
    - name: xe-linux-distribution.service
      command: start
      content: |
        [Unit]
        Description=Hypervisor Linux Guest Agent
        After=docker.service

        [Service]
        ExecStartPre=/media/configdrive/agent/xe-linux-distribution /var/cache/xe-linux-distribution
        ExecStart=/media/configdrive/agent/xe-daemon

  etcd2:
    name: %VMNAMETOHOSTNAME%
    initial-cluster: coreosnode2=http://coreosnode2.domain.com:2380,coreosnode1=http://coreosnode1.domain.com:2380 # This Line is defined by the outpu of the commands refrenced in the link above this clode block in my blog post.
    initial-cluster-state: existing # This is Required to inform etcd that the cluster already exists and is documented above.

    # These work as they usually do with the exeption of using DNS instead of ip's
    advertise-client-urls: http://coreosnode4.domain.com:2379
    listen-client-urls: http://0.0.0.0:2379
    listen-peer-urls: http://coreosnode4.domain.com:2380

write_files:  
  # Enable ARP notifications for smooth network recovery after migrations
  - path: /etc/sysctl.d/10-enable-arp-notify.conf
    permissions: 0644
    owner: root
    content: |
      net.ipv4.conf.all.arp_notify = 1

I hope this Helps others in configuring their XenServer CoreOS cluster :)

Related Article