Bugged by a ChromeBug

It’s been a tense couple of weeks for the AppsAnywhere dev team as we battled with an issue that arose with the potential to cause a really big headache for the team!

A couple of weeks ago a ticket was raised by one of our customers that described an issue with the user’s app list failing to load after validation had been triggered. From the offset, it was apparent that this issue was limited to Google Chrome and it appeared to be related to a recent update that had just been pushed out by Google.

Initially, the team were sceptical of the issue as we had seen something similar when google pushed out their last major update (documented here) which turned out to be somewhat of a false flag. We quickly realized however that this issue was replicable and spent an intense couple of hours quickly trying to limit the scope of the issue and narrow down on what we thought the cause of the problem might be.

Eventually we narrowed the issue down to a very specific area and decided that it was most likely an unintended consequence of changes made in the most recent Chrome update and as such, we decided to raise the issue with the Chrome dev team. In the grand scheme of things, sending a bug report to Google is a very bold decision to make as it’s hard to imagine a company with their resources allowing any major bug into production but we were confident that the issue wasn’t anything to do with S2Hub and so it was raised.

Admittedly, I wasn’t holding out a great deal of hope. If anything, an issue caused by someone else’s code is a thousand times worse than one caused by your own as the resolution is completely out of your hands. Given the potential scope of the problem, even when raising the bug, I was considering a wide range of alternative plans for how we could work around the issue if there was nothing that could be done in the short term.

As it happens, the Google dev team were very responsive. Within hours they were categorizing the issue and referring it to specific people within the team. Over the next couple of days, they came back with questions and we responded as quickly as we could with as much detail as possible to help further define the issue. We demonstrated how to reproduce the issue in the most minimal way possible and after only 4 days the underlying cause of the issue had finally been identified.

Then came the big question.. “can you quantify how many users are affected?”. They were basically asking, as I do when triaging bugs at AppsAnywhere, how important is it that a fix is provided and how urgently should the fix be prioritized. Luckily, I had an awesome response to provide; we know for sure that at least a handful of our customers use Chrome as their default browser on-site, which probably represents at least 100k users, and globally we have a user base of over 1.5 million students that could be potentially affected which even if you assume Chrome’s average browser share of somewhere around 60%, that’s a lot of users! Through the power of the Software community, the issue was set as high priority and dealt with as a critical bug.

Resolving a critical bug is a massive issue as it means bypassing the usual ‘alpha, beta, stable’ release process and pushing out a fix that hasn’t undergone the usual amount of community testing. To do this people on the Chrome team had to manually review the suggested fix and directly merge it into the next stable mid-release build to avoid our massive user base being affected. Crazy!

This, to me, is what AppsAnywhere is all about; the community, everyone coming together to be more powerful than they are individually. Without the massive user base that we have for AppsAnywhere, this could have been a major headache but as we are, it was resolved in an unimaginably small amount of time.

4^th September - Affected update released

6^th September - Ticket raised by customer

7^th September - Chrome bug raised by AppsAnywhere

12^th September - Problem identified

13^th September - Problem resolved

14^th September - Resolution update released

Actual time it took Chrome to fix the bug we reported: 7 days!

A massive thanks to the Google Chrome team for their professionalism in dealing with this matter in such a timely manner and for taking the time to pay so much attention to each bug report they receive. While we did our best to quickly demonstrate in a concise manner that this was a genuine issue, there must be a great deal of noise that they must deal with and to give this issue the full attention it required in the timeframe they did is a testament to their process.

All in all, as I said, a fun two weeks!

If you wish to see the full interactions on this issue, check out the Chrome bug report:

https://bugs.chromium.org