the peering xchange : [a network engineer’s blog]: Juniper vMX on VMWare ESXi

During my studies, I wanted to get my hands on more Juniper and JunOS experience. I did play around with Juniper Olive, and while that did provide a good avenue to learn a lot on JunOS, it wasn’t fully featured and certain technologies were not supported (at least for the labs I wanted to do).

In doing a little research, I found that Juniper offered their MX router platform as a full featured virtualized machine. What was even better was that they offered a 60-day free trial. This was great as I had no access to a physical Juniper MX box in a test environment. So I took it upon myself to try and setup vMX on an ESXi standalone server I had. Here are my deployment notes…

VM OVA Deployment

Download OVAs from Juniper’s site. You must have an account to do this. There will be 2 VMs that are required to be deployed to operate a single router instance. One is for the router’s control plane and other is for the forwarding plane. The control plane VM is called the VCP and the forwarding plane VM the VFP.

Deploy the VCP OVA first.

Give the VM a name. In my deployment, I named each router “Rxx”, with xx being a 2 digit number, such as 01. With 2 VMs required per router, it got confusing quickly, so giving them a descriptive name was essential.

I opted to Thin provision to save some disk space.

During the OVA deployment, the “br-ext” interface is the VCP’s management interface. I kept this in an isolated vSwitch for now. The “br-int” interface is the internal interface that is used to communicate with the forwarding plane VM (VFP). There will be more information on the 2 interfaces later.

Click Finish to deploy.

Once the VCP has been deployed, repeat the procedure for the VFP. I believe the network mapping during the VFP deployment will use the same names (i.e., br-ext and br-int). Again, the br-ext is the management interface for the VFP itself and the br-int interface is the internal communication interface used between the VFP to VCP.

VM Settings

Virtual Control Plane (VCP)

Edit the VCP’s VM settings. Based on my experience, I was able to successfully run the VCP with 1 vCPU with 2 GB of memory.

Network Adapter 1 is the routing engine’s management port (interface fxp0 in JunOS). By default it will use DHCP, so it will grab an address if it’s in a vSwitch that has a DHCP server. In my deployment, I isolated it in its own vSwitch since I didn’t plan to use it (more on this management interface later).

Network Adapter 2 is for internal communication between the VCP and VFP. A dedicated vSwitch and VLAN are required. I would recommend this vSwitch not touch the physical network (i.e., do not use a vSwitch mapped to physical NIC and connected to a network etc.). I created a single dedicated vSwitch for all my vMX instances and created individual VLANs for each routers’ internal communications interfaces. For example, I created vSwitch 998 with Port-groups/VLANs 900 for R01, 901 for R02 and 902 for R03 etc.

Virtual Forwarding Plane (VCP)

Next edit the VFP’s VM settings. The default settings of the VFP are set with 1 vCPU with 32MB of memory. This obviously wasn’t enough. I’m not sure why the deployment process didn’t set this automatically but based on my experience, 3 vCPUs with 8 GB of memory worked well during my testing.

There will be only 2 network adapters by default, network adapters 1 and 2. Any additional adapters must be installed manually if more are required.

Network Adapter 1 is this VM’s management interface and is intended to be used for administrative access purposes, such as SSH etc. If this interface is in a vSwitch/VLAN with a DHCP server, it will grab an address automatically. The VFP VM runs Wind River Linux and its management interface is labeled “ext”. In my deployment, I used DHCP for IP address assignment for this interface.

Network Adapter 2 is labeled “int” and is the interface that’s used for the internal communications between the VFP to VCP. Assign the Port-group/VLAN that is dedicated for this router. The IP address is automatically assigned using its own allocation (no need for a DHCP server), so there isn’t anything to be done there.

Network Adapter 3 – 6 as seen below were manually added. Adapter 3 will be represented as "ge-0/0/0" in JunOS, Adapter 4 as "ge-0/0/1" and so on. You can add as many as you require. Select “E1000” as the adapter type if not already selected.

Powering on VMs

Power on the VCP (Control Plane) first then the VFP (Forwarding Plane) about 10 seconds apart. For best results, I tried to get the VCP to boot before the VFP. Give them time for both to fully boot up.

Once at the login prompt, use the username "root" (no password at this point).

To login into the VFP, use the username "root" with password "root".

While in the VFP, to list all interfaces, use the command "ifconfig". The interface labeled “ext” is the management interface. The IP assigned here (via DHCP) can be used for remote management to this VM if desired.

Administrative Settings

Once in JunOS, configure some basic administrative system settings, such as setting the root password, creating a new user account and enabling SSH.

Configure a “root” password.

Configure to allow SSH.

Configure a new username and password. Commit all changes.

If the fxp0 interface was set in a vSwitch with DHCP, the IP address assigned can be used to SSH directly into the VCP for administration.

VCP to VFP Communications

VCP to VFP communication is essential for vMX to function correctly. Without it, the interfaces will not present itself and no data forwarding will occur.

Use the command below to check if the VCP is talking to the VFP. If Slot 0 is Online with a relatively low CPU total, then it’s working fine. If, on the other hand, the CPU total seems to be running unusually high, reboot both the VCP and VFP. I have seen this before and a reboot typically helps.

If the VPC is correctly communicating with the VFP, all the Ethernet interfaces will be visible. If you can see the ge-0/0/0 - ge-0/0/x in the list (depending on how many adapters were added in the VFP setup), things are working fine.

In the example below, I added 4 extra network adapters so interfaces ge-0/0/0 - ge-0/03 are showing Up/Up.

License Install

This command show current licensing status.

Add the evaluation license key via the terminal.

Appendix

The following were additional configuration items implemented during my deployment. They were not required for vMX to run, however they were helpful for my purposes.

Dedicated Management Interface

Based on the JunOS version I tested (17.2R1.13), the routing engine’s management interface (interface fxp0) could not be placed into a separate routing-instance (i.e., VRF) for management traffic isolation. I believe this feature was available starting with JunOS version 17.3R1 and later.

This was very annoying because having a management default route in the default routing-instance can cause routing conflicts if another default route was required in a particular scenario. In order to mitigate this issue, I simply used one of the Ethernet interfaces (e.g., ge-0/0/0) and placed that into a new routing-instance (and disabled the fxp0 interface). I was then able to achieve the same functionality of a true OOB management interface.

One small problem with this was that if for any reason the VFP was having issues communicating with the VCP, none of the interfaces (including ge-0/0/0) would be presented and remote administration wouldn't be possible. Since the fxp0 interface was attached to the VCP itself, and as long as the VCP was up and running, remote administration to the control-plane would always be possible (to help troubleshoot the VFP). However, in my case, this wouldn't not be possible if the VFP was down.

To setup a dedicated management interface using a VFP provided interface, first configure the desired management IP address to an unused Ethernet interface (e.g., ge-0/0/0).

Create a new routing-instance called MGMT.

Set the ge-0/0/0 interface to the MGMT routing instance, set the default route and commit changes.

Test SSH using the new management address.

If SSH'ing to the new interface was successful, the VCP's management interface (fxp0) can be disabled.

Notice ge-0/0/0 is now the management interface and fxp0 was shutdown.

Once I know the new management interface was working fine, I changed the Port-group on Network Adapter 1 (fxp0) on the VCP to something isolated. I did this as a precautionary measure.

Auto Image Upgrade Message

I experienced this annoying “Auto Upgrade Image” console message upon logging into the router. Please use the following command to disable this feature.