I am building a system that includes a hEX Lite at its core with a WAN on ether1, workstation on ether2, then one or more hAP AC Lite devices on ether{3,4,5}. This is ultimately a very simple small network from a physical layout perspective.
Where this gets fun is that I'm managing a large stack of vlans and SSIDs that need to be dynamically made available on specific ports on the hAP's and SSIDs made available to specific hAP's. This is the base level infrastructure of a robotics competition where between matches the network will be reconfigured to match the teams that are scheduled to be in the next match. This reconfiguration and config juggling is not the problem I am currently facing, as I have a large external supervisor program that already manages the schedule and generates the relevant configuration information to apply via Terraform. Where I am hung up is the bootstrapping workflow.
Out of the box it is my intention to have contest organizers reinstall the OS to guarantee a known base state. I have this already well automated and wrapped under a thin veneer of tooling I wrote that makes options available using terms and concepts that my userbase already understands. This tooling does little more than template an `rsc` file onto disk and then run netinstall-cli with the correct flags set.
This obviously results in devices that have an extremely minimal configuration loaded, which is fine, since the bulk of the configuration will be loaded later via API. What I am running into is how to setup the system so that I can reach the hAP's to configure them. My first attempt was to simply create a dhcp client on ether1 and then connect that to the right port on the hEX that will present my management VLAN as an untagged network. This worked well and the hAP got an IP via DHCP, which was the correct IP due to a preallocated lease. What did not work well was that when I add ether1 as a bridge port on the hAP, I can no longer communicate with it, and have to reset it.
My next attempt was to re-use the solution I attempted with the hEX which was to create a special bootstrapping vlan and make the port a member of that (this time with a static address). This so far has not worked as I have been unable to get the hEX to present the VLAN across its bridge as though it were a switch for that VLAN. I again run into problems where making the port a member of the bridge causes a loss of connectivity.
How should I be doing this so that I can have netinstall put just the bare minimum configuration on the device to connect, and then I can add further configuration later without losing connectivity? Is the special provisioning vlan an idea with merit or is there a different way to do bridge ports that won't cause the DHCP assigned address to become invalid when the underlying device becomes a member of the bridge?
Please don't waste your time and mine by trying to re-architect the control plane or functional application, there are many more requirements than I cared to type here and explain the backstory. The problem at hand is bootstrapping config on cascading devices without a loss of connectivity.
Re: Network topology for bootstraping.
Can you post the "bootstrapped" config of hEX? The one before trying to add ether1 to bridge (which breaks your connectivity)?
Re: Network topology for bootstraping.
Sure, since "bootstrapped" is slightly overloaded, I'll post all the configs that could be referring to. Some of these contain passwords, I'm not worried about it since those get regenerated on every install cycle.
Right after install, this is the config on the hEX:
And this is the config on the hAP
After applying the terraform with a special bootstrap flag which prevents certain firewall rules from locking me out until all are applied (which is why the drop all rule is disabled), this is the config on the hEX:
Immediately prior to losing communication with the hAP, the configuration is as follows:
These configs are from the hEX having a special bootstrap0 interface that can be reached over vlan 2, and the hAP being accessible via ether1 directly.
Right after install, this is the config on the hEX:
Code: Select all
/interface/vlan/add comment="Bootstrap Interface" interface=ether2 name=bootstrap0 vlan-id=2
/ip/address/add address=100.64.1.1/24 interface=bootstrap0
/certificate
add name=ca common-name=local_ca key-usage=key-cert-sign
add name=self common-name=localhost
sign ca
sign self
/ip service
set www disabled=no
set www-ssl certificate=self disabled=no
/user/group/add name=readonly policy=read,ssh,web
/user/add name=gizmo-fms group=full password=36db8f0f067949e9be38ff023ed0c56b
/user/add name=gizmo-ro group=readonly password=AmicablyStrewnDial
Code: Select all
/ip/dhcp-client/add interface=ether1 disabled=no
/certificate
add name=ca common-name=local_ca key-usage=key-cert-sign
add name=self common-name=localhost
sign ca
sign self
/ip service
set www disabled=no
set www-ssl certificate=self disabled=no
/user/group/add name=readonly policy=read,ssh,web
/user/add name=gizmo-fms group=full password=36db8f0f067949e9be38ff023ed0c56b
/user/add name=gizmo-ro group=readonly password=AmicablyStrewnDial
Code: Select all
# 1970-01-02 00:24:17 by RouterOS 7.14.2
# software id = LIBF-JR99
#
# model = RB750UPr2
# serial number = HFE090F2A1Z
/interface bridge
add frame-types=admit-only-vlan-tagged ingress-filtering=no name=br0 \
vlan-filtering=yes
/interface vlan
add comment="Bootstrap Interface" interface=ether2 name=bootstrap0 vlan-id=2
add comment="FMS Network" interface=br0 name=fms0 vlan-id=10
add comment="Peer Networks" interface=br0 name=peer0 vlan-id=30
add comment=team451 interface=br0 name=team451 vlan-id=500
add comment=team452 interface=br0 name=team452 vlan-id=501
add comment=team453 interface=br0 name=team453 vlan-id=502
add comment=team454 interface=br0 name=team454 vlan-id=503
add comment=team455 interface=br0 name=team455 vlan-id=504
add comment=team456 interface=br0 name=team456 vlan-id=505
add comment=team457 interface=br0 name=team457 vlan-id=506
add comment=team458 interface=br0 name=team458 vlan-id=507
add comment=team459 interface=br0 name=team459 vlan-id=508
add comment=team460 interface=br0 name=team460 vlan-id=509
add comment=team461 interface=br0 name=team461 vlan-id=510
add comment=team462 interface=br0 name=team462 vlan-id=511
add comment=team463 interface=br0 name=team463 vlan-id=512
add comment=team464 interface=br0 name=team464 vlan-id=513
add comment=team465 interface=br0 name=team465 vlan-id=514
add comment=team466 interface=br0 name=team466 vlan-id=515
add comment=team467 interface=br0 name=team467 vlan-id=516
add comment=team468 interface=br0 name=team468 vlan-id=517
add comment=team469 interface=br0 name=team469 vlan-id=518
add comment=team470 interface=br0 name=team470 vlan-id=519
add comment=team471 interface=br0 name=team471 vlan-id=520
add comment="Upstream Networks" interface=br0 name=wan0 vlan-id=20
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add comment=team457 name=team457 ranges=10.4.57.2-10.4.57.10
add comment="FMS Default IP Pool" name=fms ranges=100.64.0.1-100.64.0.30
add comment=team464 name=team464 ranges=10.4.64.2-10.4.64.10
add comment=team458 name=team458 ranges=10.4.58.2-10.4.58.10
add comment=team456 name=team456 ranges=10.4.56.2-10.4.56.10
add comment=team452 name=team452 ranges=10.4.52.2-10.4.52.10
add comment=team471 name=team471 ranges=10.4.71.2-10.4.71.10
add comment=team470 name=team470 ranges=10.4.70.2-10.4.70.10
add comment=team451 name=team451 ranges=10.4.51.2-10.4.51.10
add comment=team460 name=team460 ranges=10.4.60.2-10.4.60.10
add comment=team466 name=team466 ranges=10.4.66.2-10.4.66.10
add comment=team467 name=team467 ranges=10.4.67.2-10.4.67.10
add comment=team459 name=team459 ranges=10.4.59.2-10.4.59.10
add comment=team465 name=team465 ranges=10.4.65.2-10.4.65.10
add comment=team462 name=team462 ranges=10.4.62.2-10.4.62.10
add comment=team468 name=team468 ranges=10.4.68.2-10.4.68.10
add comment=team461 name=team461 ranges=10.4.61.2-10.4.61.10
add comment=team454 name=team454 ranges=10.4.54.2-10.4.54.10
add comment=team469 name=team469 ranges=10.4.69.2-10.4.69.10
add comment=team455 name=team455 ranges=10.4.55.2-10.4.55.10
add comment=team453 name=team453 ranges=10.4.53.2-10.4.53.10
add comment=team463 name=team463 ranges=10.4.63.2-10.4.63.10
/ip dhcp-server
add address-pool=fms comment="FMS Default DHCP Server" interface=fms0 \
lease-time=10m name=FMS
add address-pool=team462 comment=team462 interface=team462 lease-time=10m \
name=team462
add address-pool=team468 comment=team468 interface=team468 lease-time=10m \
name=team468
add address-pool=team457 comment=team457 interface=team457 lease-time=10m \
name=team457
add address-pool=team470 comment=team470 interface=team470 lease-time=10m \
name=team470
add address-pool=team471 comment=team471 interface=team471 lease-time=10m \
name=team471
add address-pool=team453 comment=team453 interface=team453 lease-time=10m \
name=team453
add address-pool=team469 comment=team469 interface=team469 lease-time=10m \
name=team469
add address-pool=team456 comment=team456 interface=team456 lease-time=10m \
name=team456
add address-pool=team463 comment=team463 interface=team463 lease-time=10m \
name=team463
add address-pool=team459 comment=team459 interface=team459 lease-time=10m \
name=team459
add address-pool=team464 comment=team464 interface=team464 lease-time=10m \
name=team464
add address-pool=team460 comment=team460 interface=team460 lease-time=10m \
name=team460
add address-pool=team467 comment=team467 interface=team467 lease-time=10m \
name=team467
add address-pool=team452 comment=team452 interface=team452 lease-time=10m \
name=team452
add address-pool=team465 comment=team465 interface=team465 lease-time=10m \
name=team465
add address-pool=team461 comment=team461 interface=team461 lease-time=10m \
name=team461
add address-pool=team455 comment=team455 interface=team455 lease-time=10m \
name=team455
add address-pool=team454 comment=team454 interface=team454 lease-time=10m \
name=team454
add address-pool=team451 comment=team451 interface=team451 lease-time=10m \
name=team451
add address-pool=team458 comment=team458 interface=team458 lease-time=10m \
name=team458
add address-pool=team466 comment=team466 interface=team466 lease-time=10m \
name=team466
/user group
add name=readonly policy="ssh,read,web,!local,!telnet,!ftp,!reboot,!write,!pol\
icy,!test,!winbox,!password,!sniff,!sensitive,!api,!romon,!rest-api"
/interface bridge port
add bridge=br0 interface=ether3 internal-path-cost=10 path-cost=10 pvid=10
add bridge=br0 interface=ether1 internal-path-cost=10 path-cost=10 pvid=20
add bridge=br0 comment="Upstream Networks" interface=wan0 internal-path-cost=\
10 path-cost=10 pvid=20
add bridge=br0 interface=ether4 internal-path-cost=10 path-cost=10 pvid=10
add bridge=br0 interface=ether5 internal-path-cost=10 path-cost=10 pvid=10
add bridge=br0 interface=ether2 internal-path-cost=10 path-cost=10 pvid=10
add bridge=br0 comment="Peer Networks" interface=peer0 internal-path-cost=10 \
path-cost=10 pvid=30
add bridge=br0 comment="FMS Network" interface=fms0 internal-path-cost=10 \
path-cost=10 pvid=10
add bridge=br0 comment=team462 interface=team462 internal-path-cost=10 \
path-cost=10 pvid=511
add bridge=br0 comment=team452 interface=team452 internal-path-cost=10 \
path-cost=10 pvid=501
add bridge=br0 comment=team459 interface=team459 internal-path-cost=10 \
path-cost=10 pvid=508
add bridge=br0 comment=team466 interface=team466 internal-path-cost=10 \
path-cost=10 pvid=515
add bridge=br0 comment=team463 interface=team463 internal-path-cost=10 \
path-cost=10 pvid=512
add bridge=br0 comment=team469 interface=team469 internal-path-cost=10 \
path-cost=10 pvid=518
add bridge=br0 comment=team455 interface=team455 internal-path-cost=10 \
path-cost=10 pvid=504
add bridge=br0 comment=team465 interface=team465 internal-path-cost=10 \
path-cost=10 pvid=514
add bridge=br0 comment=team456 interface=team456 internal-path-cost=10 \
path-cost=10 pvid=505
add bridge=br0 comment=team461 interface=team461 internal-path-cost=10 \
path-cost=10 pvid=510
add bridge=br0 comment=team457 interface=team457 internal-path-cost=10 \
path-cost=10 pvid=506
add bridge=br0 comment=team464 interface=team464 internal-path-cost=10 \
path-cost=10 pvid=513
add bridge=br0 comment=team458 interface=team458 internal-path-cost=10 \
path-cost=10 pvid=507
add bridge=br0 comment=team467 interface=team467 internal-path-cost=10 \
path-cost=10 pvid=516
add bridge=br0 comment=team470 interface=team470 internal-path-cost=10 \
path-cost=10 pvid=519
add bridge=br0 comment=team468 interface=team468 internal-path-cost=10 \
path-cost=10 pvid=517
add bridge=br0 comment=team460 interface=team460 internal-path-cost=10 \
path-cost=10 pvid=509
add bridge=br0 comment=team454 interface=team454 internal-path-cost=10 \
path-cost=10 pvid=503
add bridge=br0 comment=team471 interface=team471 internal-path-cost=10 \
path-cost=10 pvid=520
add bridge=br0 comment=team451 interface=team451 internal-path-cost=10 \
path-cost=10 pvid=500
add bridge=br0 comment=team453 interface=team453 internal-path-cost=10 \
path-cost=10 pvid=502
/interface bridge vlan
add bridge=br0 untagged=ether2 vlan-ids=10
add bridge=br0 tagged=ether1 vlan-ids=30
add bridge=br0 untagged=ether1 vlan-ids=20
add bridge=br0 comment="Bridge Networks" tagged=br0 vlan-ids="10,20,30,500,501\
,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,5\
20"
add bridge=br0 tagged="team451,team452,team453,team454,team455,team456,team457\
,team458,team459,team460,team461,team462,team463,team464,team465,team466,t\
eam467,team468,team469,team470,team471,ether3,ether4,ether5" vlan-ids="500\
,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,5\
19,520"
/ip address
add address=100.64.1.1/24 interface=bootstrap0 network=100.64.1.0
add address=100.64.0.1/24 interface=fms0 network=100.64.0.0
add address=10.4.56.1/24 interface=team456 network=10.4.56.0
add address=10.4.67.1/24 interface=team467 network=10.4.67.0
add address=10.4.61.1/24 interface=team461 network=10.4.61.0
add address=10.4.64.1/24 interface=team464 network=10.4.64.0
add address=10.4.52.1/24 interface=team452 network=10.4.52.0
add address=10.4.51.1/24 interface=team451 network=10.4.51.0
add address=10.4.54.1/24 interface=team454 network=10.4.54.0
add address=10.4.71.1/24 interface=team471 network=10.4.71.0
add address=10.4.68.1/24 interface=team468 network=10.4.68.0
add address=10.4.69.1/24 interface=team469 network=10.4.69.0
add address=10.4.63.1/24 interface=team463 network=10.4.63.0
add address=10.4.58.1/24 interface=team458 network=10.4.58.0
add address=10.4.62.1/24 interface=team462 network=10.4.62.0
add address=10.4.60.1/24 interface=team460 network=10.4.60.0
add address=10.4.70.1/24 interface=team470 network=10.4.70.0
add address=10.4.66.1/24 interface=team466 network=10.4.66.0
add address=10.4.53.1/24 interface=team453 network=10.4.53.0
add address=10.4.65.1/24 interface=team465 network=10.4.65.0
add address=10.4.59.1/24 interface=team459 network=10.4.59.0
add address=10.4.55.1/24 interface=team455 network=10.4.55.0
add address=10.4.57.1/24 interface=team457 network=10.4.57.0
/ip dhcp-client
add comment="External Upstream" interface=wan0 use-peer-dns=no use-peer-ntp=\
no
/ip dhcp-server lease
add address=100.64.0.10 comment="Field 1" mac-address=78:9A:18:7E:54:5D \
server=FMS
/ip dhcp-server network
add address=10.4.51.0/24 comment=team451 dns-server=10.4.51.1 gateway=\
10.4.51.1
add address=10.4.52.0/24 comment=team452 dns-server=10.4.52.1 gateway=\
10.4.52.1
add address=10.4.53.0/24 comment=team453 dns-server=10.4.53.1 gateway=\
10.4.53.1
add address=10.4.54.0/24 comment=team454 dns-server=10.4.54.1 gateway=\
10.4.54.1
add address=10.4.55.0/24 comment=team455 dns-server=10.4.55.1 gateway=\
10.4.55.1
add address=10.4.56.0/24 comment=team456 dns-server=10.4.56.1 gateway=\
10.4.56.1
add address=10.4.57.0/24 comment=team457 dns-server=10.4.57.1 gateway=\
10.4.57.1
add address=10.4.58.0/24 comment=team458 dns-server=10.4.58.1 gateway=\
10.4.58.1
add address=10.4.59.0/24 comment=team459 dns-server=10.4.59.1 gateway=\
10.4.59.1
add address=10.4.60.0/24 comment=team460 dns-server=10.4.60.1 gateway=\
10.4.60.1
add address=10.4.61.0/24 comment=team461 dns-server=10.4.61.1 gateway=\
10.4.61.1
add address=10.4.62.0/24 comment=team462 dns-server=10.4.62.1 gateway=\
10.4.62.1
add address=10.4.63.0/24 comment=team463 dns-server=10.4.63.1 gateway=\
10.4.63.1
add address=10.4.64.0/24 comment=team464 dns-server=10.4.64.1 gateway=\
10.4.64.1
add address=10.4.65.0/24 comment=team465 dns-server=10.4.65.1 gateway=\
10.4.65.1
add address=10.4.66.0/24 comment=team466 dns-server=10.4.66.1 gateway=\
10.4.66.1
add address=10.4.67.0/24 comment=team467 dns-server=10.4.67.1 gateway=\
10.4.67.1
add address=10.4.68.0/24 comment=team468 dns-server=10.4.68.1 gateway=\
10.4.68.1
add address=10.4.69.0/24 comment=team469 dns-server=10.4.69.1 gateway=\
10.4.69.1
add address=10.4.70.0/24 comment=team470 dns-server=10.4.70.1 gateway=\
10.4.70.1
add address=10.4.71.0/24 comment=team471 dns-server=10.4.71.1 gateway=\
10.4.71.1
add address=100.64.0.0/24 comment="Options for FMS" dns-server=100.64.0.1 \
domain=gizmo gateway=100.64.0.1
/ip firewall address-list
add address=127.0.53.53 comment="Name collision occurence" list=bogons_v4
add address=100.64.0.0/24 comment="NAT Source Pool" list=nat_sources
add address=192.168.0.0/16 comment=RFC1918 list=bogons_v4
add address=0.0.0.0/8 comment="'This' network" list=bogons_v4
add address=240.0.0.0/4 comment="Reserved for future use" list=bogons_v4
add address=127.0.0.0/8 comment=Loopback list=bogons_v4
add address=255.255.255.255 comment="Limited Broadcast" list=bogons_v4
add address=203.0.113.0/24 comment=TEST-NET-3 list=bogons_v4
add address=192.0.2.0/24 comment=TEST-NET-1 list=bogons_v4
add address=192.0.0.0/24 comment="IETF Protocol Assignments" list=bogons_v4
add address=172.16.0.0/12 comment=RFC1918 list=bogons_v4
add address=198.51.100.0/24 comment=TEST-NET-2 list=bogons_v4
add address=100.64.0.0/10 comment=CG-NAT list=bogons_v4
add address=198.18.0.0/15 comment=\
"Network interconect device benchmark testing" list=bogons_v4
add address=169.254.0.0/16 comment="Link Local" list=bogons_v4
add address=224.0.0.0/4 comment=Multicast list=bogons_v4
add address=10.0.0.0/8 comment=RFC1918 list=bogons_v4
/ip firewall filter
add action=accept chain=input comment=accept-established connection-state=\
established,related,untracked
add action=accept chain=input in-interface=peer0
add action=drop chain=input comment=drop-invalid connection-state=invalid
add action=accept chain=input protocol=icmp
add action=drop chain=input comment=deny-bogons-to-self in-interface=wan0 \
src-address-list=bogons_v4
add action=drop chain=input comment=default-deny in-interface=!fms0
/ip firewall nat
add action=masquerade chain=srcnat comment=nat-masquerade out-interface=wan0 \
src-address-list=nat_sources
/ip service
set telnet disabled=yes port=21
set ftp disabled=yes
set www disabled=yes
set www-ssl certificate=self disabled=no
set api disabled=yes port=8278
set winbox disabled=yes
set api-ssl disabled=yes
/system identity
set name=gizmo-edge
/system note
set show-at-login=no
Code: Select all
# 1970-01-02 00:11:44 by RouterOS 7.14.2
# software id = 1J90-DG0X
#
# model = RB952Ui-5ac2nD
# serial number = HF6090SBZFK
/interface bridge
add frame-types=admit-only-vlan-tagged ingress-filtering=no name=br0 \
vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
set [ find default-name=wlan2 ] ssid=MikroTik
/interface vlan
add comment=team451 interface=br0 name=team451 vlan-id=500
add comment=team452 interface=br0 name=team452 vlan-id=501
add comment=team453 interface=br0 name=team453 vlan-id=502
add comment=team454 interface=br0 name=team454 vlan-id=503
add comment=team455 interface=br0 name=team455 vlan-id=504
add comment=team456 interface=br0 name=team456 vlan-id=505
add comment=team457 interface=br0 name=team457 vlan-id=506
add comment=team458 interface=br0 name=team458 vlan-id=507
add comment=team459 interface=br0 name=team459 vlan-id=508
add comment=team460 interface=br0 name=team460 vlan-id=509
add comment=team461 interface=br0 name=team461 vlan-id=510
add comment=team462 interface=br0 name=team462 vlan-id=511
add comment=team463 interface=br0 name=team463 vlan-id=512
add comment=team464 interface=br0 name=team464 vlan-id=513
add comment=team465 interface=br0 name=team465 vlan-id=514
add comment=team466 interface=br0 name=team466 vlan-id=515
add comment=team467 interface=br0 name=team467 vlan-id=516
add comment=team468 interface=br0 name=team468 vlan-id=517
add comment=team469 interface=br0 name=team469 vlan-id=518
add comment=team470 interface=br0 name=team470 vlan-id=519
add comment=team471 interface=br0 name=team471 vlan-id=520
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/user group
add name=readonly policy="ssh,read,web,!local,!telnet,!ftp,!reboot,!write,!pol\
icy,!test,!winbox,!password,!sniff,!sensitive,!api,!romon,!rest-api"
/interface bridge port
add bridge=br0 interface=ether3 internal-path-cost=10 path-cost=10 pvid=500
add bridge=br0 interface=ether5 internal-path-cost=10 path-cost=10 pvid=500
add bridge=br0 interface=ether4 internal-path-cost=10 path-cost=10 pvid=500
add bridge=br0 interface=ether2 internal-path-cost=10 path-cost=10 pvid=500
add bridge=br0 comment=team457 interface=team457 internal-path-cost=10 \
path-cost=10 pvid=506
add bridge=br0 comment=team468 interface=team468 internal-path-cost=10 \
path-cost=10 pvid=517
add bridge=br0 comment=team462 interface=team462 internal-path-cost=10 \
path-cost=10 pvid=511
add bridge=br0 comment=team456 interface=team456 internal-path-cost=10 \
path-cost=10 pvid=505
add bridge=br0 comment=team460 interface=team460 internal-path-cost=10 \
path-cost=10 pvid=509
add bridge=br0 comment=team453 interface=team453 internal-path-cost=10 \
path-cost=10 pvid=502
add bridge=br0 comment=team455 interface=team455 internal-path-cost=10 \
path-cost=10 pvid=504
add bridge=br0 comment=team451 interface=team451 internal-path-cost=10 \
path-cost=10 pvid=500
add bridge=br0 comment=team466 interface=team466 internal-path-cost=10 \
path-cost=10 pvid=515
add bridge=br0 comment=team471 interface=team471 internal-path-cost=10 \
path-cost=10 pvid=520
add bridge=br0 comment=team452 interface=team452 internal-path-cost=10 \
path-cost=10 pvid=501
add bridge=br0 comment=team458 interface=team458 internal-path-cost=10 \
path-cost=10 pvid=507
add bridge=br0 comment=team459 interface=team459 internal-path-cost=10 \
path-cost=10 pvid=508
add bridge=br0 comment=team470 interface=team470 internal-path-cost=10 \
path-cost=10 pvid=519
add bridge=br0 comment=team454 interface=team454 internal-path-cost=10 \
path-cost=10 pvid=503
add bridge=br0 comment=team464 interface=team464 internal-path-cost=10 \
path-cost=10 pvid=513
add bridge=br0 comment=team467 interface=team467 internal-path-cost=10 \
path-cost=10 pvid=516
add bridge=br0 comment=team469 interface=team469 internal-path-cost=10 \
path-cost=10 pvid=518
add bridge=br0 comment=team465 interface=team465 internal-path-cost=10 \
path-cost=10 pvid=514
add bridge=br0 comment=team461 interface=team461 internal-path-cost=10 \
path-cost=10 pvid=510
add bridge=br0 comment=team463 interface=team463 internal-path-cost=10 \
path-cost=10 pvid=512
/interface bridge vlan
add bridge=br0 comment="Bridge Networks" tagged=br0 vlan-ids="500,501,502,503,\
504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520"
add bridge=br0 tagged="team451,team452,team453,team454,team455,team456,team457\
,team458,team459,team460,team461,team462,team463,team464,team465,team466,t\
eam467,team468,team469,team470,team471" vlan-ids="500,501,502,503,504,505,\
506,507,508,509,510,511,512,513,514,515,516,517,518,519,520"
/ip dhcp-client
add interface=ether1
/ip service
set telnet disabled=yes port=21
set ftp disabled=yes
set www disabled=yes
set www-ssl certificate=self disabled=no
set api disabled=yes port=8278
set winbox disabled=yes
set api-ssl disabled=yes
/system identity
set name=gizmo-field-1
/system note
set show-at-login=no
Re: Network topology for bootstraping.
I'll comment on "just before loosing contact" config on hAP:
- you should never add vlan interface back to anchor. Like this:
The second part is the wrong part (the first one is fine).Code: Select all
/interface vlan add comment=team451 interface=br0 name=team451 vlan-id=500 /interface bridge port add bridge=br0 comment=team451 interface=team451 internal-path-cost=10 \ path-cost=10 pvid=500
Consequently you should not add these invalid ports as VLAN members under /interface bridge vlan - This command sequence is problematic (and probably the break point):
It probably actually fails, but you don't see it due the way config is applied ...Code: Select all
/interface bridge vlan add bridge=br0 untagged=ether1 vlan-ids=20 add bridge=br0 comment="Bridge Networks" tagged=br0 vlan-ids="10,20,30,500,501\ ,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,5\ 20"
The thing is that you can only have one add vlan-ids=XX per vlan-id. The first command creates table entry for VLAN ID 20 and second one tries to create another one (which includes same VID). Instead you should have something like this:
Note that you don't have to add ports as untagged if they have pvid set in /interface/bridge/port section, that's done automatically.Code: Select all/interface bridge vlan add bridge=br0 vlan-ids=20 tagged=br0 untagged=ether1 add bridge=br0 vlan-ids=500 tagged=br0,ether2,ether4,ether5
Re: Network topology for bootstraping.
If I understand you correctly I should make the following changes:
* Do not add the vlan interfaces themselves to the bridge, merely creating them with the parent interface set to be the bridge is sufficient.
* When adding vlan-IDs to the bridge, each vlan must appear exactly once, with all the interfaces that are to receive it tagged or untagged appearing in that command.
* It is not necessary to explicitly add a port to an untagged vlan, setting the pvid value when adding it to the bridge is enough.
Based on that, I think that my next course of action is to remove a significant number of resource edits and re-apply, which should result in the vlans not being added to the bridge twice. My understanding, however, is that I'll still lose connectivity because as soon as I add ether1 to the bridge, its pvid will be 10, which I expect, but the IP that I am connecting to is still directly on the interface which I believe is now unreachable. Is that actually what happens here or is there something more nuanced going on?
* Do not add the vlan interfaces themselves to the bridge, merely creating them with the parent interface set to be the bridge is sufficient.
* When adding vlan-IDs to the bridge, each vlan must appear exactly once, with all the interfaces that are to receive it tagged or untagged appearing in that command.
* It is not necessary to explicitly add a port to an untagged vlan, setting the pvid value when adding it to the bridge is enough.
Based on that, I think that my next course of action is to remove a significant number of resource edits and re-apply, which should result in the vlans not being added to the bridge twice. My understanding, however, is that I'll still lose connectivity because as soon as I add ether1 to the bridge, its pvid will be 10, which I expect, but the IP that I am connecting to is still directly on the interface which I believe is now unreachable. Is that actually what happens here or is there something more nuanced going on?
Re: Network topology for bootstraping.
I did another test incorporating the changes in my last post and I've now positively identified the point at which I lose connection to be enabling ether1 as a port on br0. I can set it up disabled and using safe mode I was able to work out that enabling the bridge port was where it fails.
Re: Network topology for bootstraping. [SOLVED]
It shouldn't come as a surprise. After an interface is "enslaved" as port of a bridge, it shouldn't be used as interface (i.e. carry any L3 functions) anymore. If you want to keep using ether1 in similar manner after adding it to bridge, you shoukd set up bridge so that it provides same level of connectivity. In advance.I did another test incorporating the changes in my last post and I've now positively identified the point at which I lose connection to be enabling ether1 as a port on br0.
E.g.:
- add "provisioning" VLAN interface, anchored to br0 (e.g. with vlan-id=666)
- set br0 as tagged member of VLAN 666
- add DHCP client to "provisioning" VLAN interface
- at the end add ether1 as port of br0 with PVID set to 666
After that you can remove DHCP client on ether1.
Or something like that. The point I'm trying to make is that at some point the way to connect to hAP will be different. How exactly depends on intended final hAP configuration.
BTW in a LAN topology with VLANs it's good to have a dedicated management VLAN, connecting all gadgets (which obviously have to allow management via that VID ... in my example that's VID 666 and an AP/switch only needs IP address in that VLAN ... router needs IP addresses in all VLANs and firewall setup which limits access to management VLAN from other VLANs). I couldn't identify such VLAN in your setup.
Re: Network topology for bootstraping.
In my setup that would be vlan 10, which is bound to fms0 on the hEX. What I had considered doing was to just extend my vlan2 (bootstrap0) from the hEX as a straight swtiched network to the hAPs and have them get static IPs in that vlan for a truly segregated management LAN. I had trouble getting this to work though and its not entirely clear where that's failing yet. I had thought that it was possible to have vlans from one interface land on different targets (i.e. one vlan directly tied to the interface, with others on the bridge). Perhaps that is not the case though?
Re: Network topology for bootstraping.
What about not using API to configure the VLANs and whatnot at the end, instead generate a little .rsc with the appropriate config/commands in it, and /fetch and /import it?
Re: Network topology for bootstraping.
If you're thinking of a combo "interface is bridge port, but is anchor for a vlan interface" ... then no, it shouldn't be done like that (it falls into category "it shouldn't be used as interface").
The problem in your setup procedure is that you're effectively changing L2 topology ... and it's almost impossible to do it without loosing access. The only way not to loose access is if you dedicate a port for (sort of) out-of-band management, this way you're not "cutting your own branch".
The only question is then: how to regain management access. One way is as I outlined in my previous post (i.e. allow management access via bridge port, to which your management PC is connected, with appropriate bridge config applied in advance), the other is to configure everything to have management via trunk port (connecting to main router) and in this case you'd have to move management PC from device being provisioned to (appropriately configured) port of main router (or another properly configured switch).
The problem in your setup procedure is that you're effectively changing L2 topology ... and it's almost impossible to do it without loosing access. The only way not to loose access is if you dedicate a port for (sort of) out-of-band management, this way you're not "cutting your own branch".
The only question is then: how to regain management access. One way is as I outlined in my previous post (i.e. allow management access via bridge port, to which your management PC is connected, with appropriate bridge config applied in advance), the other is to configure everything to have management via trunk port (connecting to main router) and in this case you'd have to move management PC from device being provisioned to (appropriately configured) port of main router (or another properly configured switch).
Re: Network topology for bootstraping.
Increasingly I'm trending towards forming the bridge and the management vlan in the bootstrap.rsc, then configuring everything else on top of that via API so that I don't have a point where I've modified the layer 2 over itself.What about not using API to configure the VLANs and whatnot at the end, instead generate a little .rsc with the appropriate config/commands in it, and /fetch and /import it?
It seems like the most foolproof method here then is to change my setup procedure such that I don't at any point lose the out of band access. With the most straightforward way to do that seeming to be to create the out of band access at netinstall time and then not modifying it after that. I'd hoped to avoid that since it means I have configuration spread across multiple points, but it seems like it may be unavoidable.
Re: Network topology for bootstraping.
The solution of adding the dhcp client to the vlan on the bridge is working well. I need to add some guardrails to my provisioning wrappers to tell people to reboot things at the appropriate points, but otherwise everything is working great. Thanks for all the advice, I think this is solved now. I wound up pinning the admin-mac on the bridge to that of ether1, so that it would get the same address again since that's pinned to the mac address at the hEX.