|Router Security||Self-Updating Firmware||
Website by |
The general thinking is that a router that can update its own firmware is better than one that requires manual updating. This is true, but overly simplistic. There are many things that go wrong with a router that self-updates it firmware.
One obvious danger is that the update breaks something. In August 2017 this happened to the a smart lock, model LS-6i, made by Lockstate. The company sent the wrong firmware and bricked the $470 locks. See their apology note.
Another danger is mass infections. In a September 2018 article on IoT security, security expert Robert Graham writes "You might think a good solution to this is automated patching, but only if you ignore history. Many rate 'NotPetya' as the worst, most costly, cyberattack ever. That was launched by subverting an automated patch ... Automated patching ... makes it much more likely mass infections will result from hackers targeting the vendor".
To plan for something like this, you would need to be able to schedule a pending update for a relatively safe time, and, be able to back out a bad update. For the most part, routers are not at that level of sophistication yet.Borrowing from the Security Checklist page, below is my list of things to look for in determining if a self-updating router is doing a good job of self-updating.
As for answering these questions, my experience with self-updating routers has been limited to Eero, the Turris Omnia and Google Wi-Fi. Someone from Linksys provided answers to these questions for their "Smart-Wifi" branded routers (model numbers starting with EA or WRT). The Linksys Velop mesh router system is something else, although, it too, can self-update. Google Wifi responses below came from Google. I slightly revised the "What to look for" questions in Feb. 2017. Both Linksys and Google answered the questions as they were in Jan. 2017.
A list of self-updating routers is on the Resources page.
|Table of Contents|
|What to look for||Google Wifi|
Q: Is there an audit log of each firmware update issued by the router vendor? Something along the lines of what Microsoft provides for Windows 10.
A: Google pushes all firmware updates from the cloud. We have dashboards so we can monitor progress of the firmware releases, which is done in carefully controlled phases.
My Translation: No. As of mid-April 2017 Google does not seem to have a single web page with a history of firmware releases. Each firmware update seems to get a new web page. I asked Google about this in mid-April 2017. They said that this page, Release Notes should have the full firmware history. I could not tell, immediately, if this was true or not. But, as of Aug. 15, 2017 it most definitely is not true.
May 8, 2017. I was right. Google updated the firmware on my Gwifi router May 3, 2017. There is no single web page with a release history. And the description of this update is shameful. The full description is "General stability & performance improvements". See a screen shot.
Q: Is there an audit log of each firmware update installed on your router? Only by comparing these two logs can you verify that the auto-update system is working correctly.
A: Cloud pushes notifications to the mobile app. Release notes accessed via the app state dates on which firmware was updated. Firmware revision can be noted per mesh node via the app.
Q: How often does the router check for updates? Can you control this?
A: Google pushes updates via the cloud roughly every six weeks.
Q: Can you be notified of firmware updates beforehand? Afterwards? If so, what type of notification?
A: Since the user does not need to manually intervene, no. Updates are done carefully in a phased rollout to ensure no regressions.
Q: If you are notified beforehand, can you schedule the firmware installation and the necessary reboots it entails?
Q: Even if you are not notified of available updates, can you set a schedule for when installation/reboots are allowed? That is, reboot at 3am but not at 3pm.
A: Reboots are auto-scheduled within 24 hours and when the systems are idle.
Q: Can you force the router to check for new firmware?
A: No. Firmware is pushed directly via the cloud.
Q: Can you force the router to update to newly available firmware, or do you have to wait for its regular check-in?
A: N/A. Firmware pushed from the cloud.
Q: If you do nothing, how quickly will newly released firmware be installed? Eero promises to install new firmware "within a few weeks"
A: We release new firmware every six weeks. It is phased in a rollout over a few days.
Q: When the router phones home looking for updates does it do so securely with TLS?
A: N/A, but yes, all communication with the cloud is secure TLS
Q: When the router downloads new firmware does it so securely with TLS? Is newly downloaded firmware validated in any way, such as being digitally signed?
A: Yes, all code is signed by Google. A Trusted Platform Module (HW TPM) verifies signatures of the code.
Q: Does the router support multiple installed firmwares? (so you can fall back in case an update causes a problem) If not, then can you install old firmware if a new version caused a problem?
A: Yes. Multiple partitions with fallback in case of a failed software update.
Q: Is there a manual over-ride mechanism for installing new firmware in case the auto-updating system fails?
A: N/A generally, but yes, firmware can be downloaded and flashed via USB key in recovery mode.
Q: Does the vendor document the changes in each firmware update? If so, do they do it well?
Q: Can you tell what version of the firmware is now running? If its a multi-device mesh router/system, then the question applies to each device.
A: Yes. Via mobile app.
Q: How smart is the auto-updating system? Specifically, can it self-update within the same firmware version, but update when there is a major new firmware release? Synology offers this on their NAS boxes. You can configure the NAS to self-update from version 5.1 to 5.2 to 5.3, but not to automatically update to version 6.
A: Users can turn off updates by opting out of cloud services
Q: Can you backup the router settings to a file? Pretty much any router can do this, but with auto-updating I wonder if that feature still exists.
Q: In a mesh system involving multiple devices, do all the devices update their firmware at same time? If not, how is it handled?
Q: In a mesh, what happens if one device gets new firmware but another device does not? Can the system run if the three devices are not on the exact same firmware release?
A: It is a managed rollout where all nodes are updated during the same session.
Separately, Google Wifi security features says "All software updates are signed by Google. Google Wifi cant download or run any software that isnt signed and verified" and "All communication between Google Wifi and Google is secured by Transport Layer Security (TLS)".
As of early May 2017, I was able to answer some of these questions myself. I own a Google WiFi router and it self-updated. My observations are here 7 mistakes Google made updating my Google Wifi router.
In June 2017, a Google Wifi router had been powered off for a couple weeks. During that time, the firmware had been updated from version 9334.41.3 to 9460.40.5. Within a couple hours, the router self-updated to the new firmware. This is a huge contrast to the much slower rollout used by eero.
In August 2017, I setup three new Google Wifi hockey pucks. Each one self-updated its firmware immediately after being put online.
This thread at a Google tech support forum illustrates how poorly Google does firmware self-updating. People can't schedule updates, are not warned beforehand, not told afterwards and can't fall back to a previous version if the new version is a problem. On my Google Wifi system (shown at the right), it updated firmware on Oct 2, 2017 at 2:51pm, right in the middle of the afternoon. Shameful. An update on Nov. 28, 2017 was done at 1:56 in the morning. Perhaps there is hope?
As of March 29, 2018 the last firmware update was Feb. 6, 2018. What version? It's not clear. When click for "More Details" in the Android app (version jetstream-BV10144_RC0003), I just see a generic list of Help article. Digging in the app turns up version 10032.86.2. There is no link to a detailed description of the changes in this version.
See more about Google Wifi routers.
NOTE: The information below was provided by someone who works for Linksys in February 2017. See also How to automatically update the firmware of the Linksys Smart Wi-Fi Routers.
According to Linksys, all of their "Smart-Wifi" branded routers can self-update. These devices usually have model numbers starting with EA or WRT. The information below applies to these routers and not to their Velop mesh router system, which can also self-update.
An "audit log" of sorts can be found on the manual firmware download page for each SKU under "release notes". The release notes contain a timestamp of the release, the firmware version, and some patch notes. See sample.
The router checks for updates by querying the Linksys Smart WiFi cloud. During setup, there's a checkbox that you can uncheck to prevent the router from automatically checking for and installing updates. This option is also togglable under the connectivity settings.
All cloud operations between the router and the cloud are over a TLS connection. This includes checking for and downloading firmware updates.
A change log is provided on the manual firmware download page under "release notes".
Current firmware version can be found under the connectivity settings.
There is currently no mechanism implemented for notification that an automatic firmware update has happened or is going to happen.
By default, the device checks for automatic updates during a 240 minute window starting at midnight local time. There is an API for setting the time and duration of that window, but I do not believe there is any UI supporting it.
There is a button under the connectivity settings that you can use to force a firmware update check. Otherwise, the device checks for new firmware daily.
Router can be forced to update to the latest firmware at any time. If a new firmware update is found after clicking the "check for updates" button, a new button is added to start download/installation.
On the connectivity settings page there's a section for uploading your own firmware images, or firmware images downloaded from the linksys support site.
The automating updating system only supports a single "update channel" that installs the newest possible firmware.
Router configuration can be backed up and restored under the Troubleshooting -> Diagnostics page.
The Linksys EA and WRT series routers are single devices and this section applies to them. Linksys just released a Velop mesh system, but the person I corresponded with was not familiar with how the Velop deals with updating three devices.
Firmware updates are downloaded over TLS. The majority of the EA series devices use signed firmware images that require signature verification before installation. None of the WRT devices have firmware signature restrictions. Technically this shouldn't matter - the TLS connection provides verification that the firmware image is coming from Linksys - firmware signing is just an added protection.
As far as I know, all EA and WRT series devices have two firmware partitions - the active partition and a fallback partition. The fallback partition is whatever firmware was previously installed on the router. That way if the main partition ever becomes unbootable, the bootloader can automatically switch to the backup partition to effectively unbrick the device. You can manually trigger a switch to the fallback partition by clicking "Restore previous firmware" under the Troubleshooting -> Diagnostics page.
According to this June 2017 review, Linksys Velop: Powerful But Erratic Mesh Router, "The firmware updates are encrypted during delivery, but the firmware doesn't self-authenticate during startup."
A history of eero firmware releases is here: eero Software Release Notes. Judge for yourself how well they document the changes in each firmware release. Personally, I am not impressed.
If you do nothing, how quickly does eero install new firmware after it has been released?
According to the Software Release Notes page: "eero software updates are released on a rolling basis, so your eero may not be updated immediately after a new version is released. New versions will be pushed to all customers' networks within a few weeks of public release." A few weeks, seems a bit slow, but this is a matter of opinion.
I asked Eero about the rollout of new firmware in April 2017 and the response was:
"We send out automatic updates every night in small numbers. Usually within approx 4 weeks of a new firmware release all eero networks should have gotten the automatic update push."
AUDITING EERO FIRMWARE UPDATE SPEED
Are software updates really pushed out with a few weeks? I have access to one Eero system and occasionally audit the firmware updates. They seem to be installed faster than that. Version numbers below are simplified, they actually include another number after a dash.
On the whole, Eero, much like Google Wifi, does a poor job informing the end user about firmware updates.
Perhaps most importantly, the app does not indicate that a software update is available, front and center. Instead, you have to dig around in the app to learn that you are behind in patching. In contrast, AmpliFi tells the user about an available update on the home screen of the app. And, you can't miss the notice.
Available, but not yet installed, updates are indicated in the app in two places. If you click on a specific eero, it shows the currently installed firmware version and a notice that "An update is available! Tap to begin update." If you go to Network Settings -> Network software version, it says "Update available." Click on this to see the version number of the not-yet-installed software and an option to install it now. This was latest verified with Android app version 2.14.0 in Feb. 2018.
The currently installed software version is reported in two different places. In Network Settings it just says that the software is "up-to-date." Period. If you, instead, click on an Eero device, it reports the currently installed firmware version. Period. When the software was installed is none of your business. Likewise, what changed in the last firmware update and its release date are none of your business. This is worse than Google Wifi which, at least, indicates the installation date of the current firmware.
Needless to say, there is no audit trail of every firmware update or install. Google Wifi can do this.
The eero app also fails to keep a history of speed tests. It seems to perform a speed test every day but only reports the last one. In contrast, both AmpliFi and Google Wifi keep a log of previous speed tests, which makes it easy to spot any trends. On the plus side, Eero reports the date and time of the last speed test, Google Wifi only reports the date. And, although you can't tell it when to run the speed test, it seems to do it at 3AM.
The eero app has no provision for saving the current settings, a trait shared by Google Wifi and AmpliFi. If something goes wrong, it will probably have to be re-configured from scratch.
After Linksys provided answers to these questions, I contacted some other router vendors asking them to explain the details of their firmware auto-updating system. Not one company did. Each company was contacted on Feb. 5, 2017 (13 days ago when I last reviewed this). The cowards are
I don't have a lot of information on the F-Secure Sense router. There is a summary of it on the Resources page. In September 2017, it got the ability to schedule firmware updates. I say this based on this tweet and their Whats New page. The tweet also shows they do not keep a complete history of firmware changes.
The Fingbox is not a router; it is a device that plugs into a router and offers LAN scanning and some security features. I mention it here because it self-updates. According to this Oct. 2017 review, "You cannot disable, throttle or change anything. If Fing wants to apply a new update, you'll get it whether you want it or not, and if that messes with your network, so be it."
The AmpliFi mesh router system does not self-update, but it does check for updates on its own. This, arguably, is the worst of all possible worlds.
I administer two AmpliFi setups, both remote from me. To fix the KRACK flaw, in October 2017, AmpliFi released new firmware, as did many router vendors. So, I went to update each AmpliFi system. The first one reported that it was running firmware version 2.4.2 and that 2.4.3 was available to be installed. Fine.
The second system was also running firmware 2.4.2 but it was ignorant of the newly released firmware. The mobile app has no manual check for update feature, so all I could do was wait until it detected the new firmware on its own. Only then, could I manually update it.
Synology routers do a good job, perhaps the best, of self-updating themselves. It seems the company has copied the procedures used by their NAS devices.
You can schedule when new firmware is auto-installed. Or, you can turn off auto update altogether. You can choose to either get the latest software auto-installed all the time, or, just get fixes to current release. That is, they give you the option to get bug fixes for your current release, but not be auto-updated to a new release. Router reboots can also be scheduled.
You can backup the current configuration with: Control Panel -> System -> Update & Restore. Finally, you can add extra software to the router at the Synology Package Center. One of the add-on packages is Intrusion Prevention. It too, can self-update its signatures and these updates can be scheduled.
In June 2018, Dwight Silverman wrote My router now auto-updates its firmware, but that's not always good. In a nutshell, his point is "Auto-updating is a great solution for a growing problem, but it needs to be implemented smartly." He owns a Netgear R6400 and it just got a new feature - the ability to self-update. However, when it does the update is none of your business. Netgear does not say. But, working for the Houston Chronicle, Silverman was able to get a response from the company, the update and reboot happens between 1AM and 4AM local time. You have no control over the timing. As for some type of notification system or the ability to defer updates - don't hold your breath.
The Turris Omnia can self-update but the feature is a mixed bag.
When it is auto-updating, there is no button to check for updates now, so you are at the mercy of its hidden schedule. Does it check daily? weekly? Automatic updating supports an option, just like Windows, to check for updates but not install them until they are manually approved. However, when you give approval, there is no status of the update. Pretty much every other router tells you that the update is downloading and then that its installing, along with some type of progress bar. With the Omnia, you have to know to check for notifications and a notification is only issued after the update is done.
There is also an option to auto-install updates with a delay (you chose the number of hours to wait). I am not sure what purpose this serves.
If you want to insure that updates are, in fact, being automatically installed, Turris maintains a change log of firmware updates. On October 1, 2018 it appeared as if the log was no longer being updated, it was missing the last two updates. By October 9, 2018, the log had been updated to include two updates issued in Sept. 2018.
If you don't want auto-updating, it can be disabled. However, when it is disabled, there is no check for update button. Beats me how you get updates in this case.
With most routers, the firmware is a single binary blob. However, with the Omnia and the two Synology routers, the firmware is modular. You can add additional software (called packages) at any time. Thus, some software on the Omnia can be updated without requiring a restart, but other software does require the router to reboot. You can set your preferred time for the router to reboot itself.
I have a page devoted to the Turris Omnia router.
The Nokia Beacon mesh routers self-update at 3AM, by default. If you prefer, you can change the time.