Firefox 26, A Retrospective in Quality

[Edit: @ttaubert informed me the charts weren’t loading so I’ve uploaded new images instead of linking directly to my document]

The release of Firefox 27 is imminently upon us, next week will mark the end of life for Firefox 26. As such I thought it’d be a good time to look back on Firefox 26 from a Quality Assurance (QA) perspective. It’s kind of hard to measure the impact QA has on a per release basis and whether our strategies are working. Currently, the best data source we have to go on is statistics from Bugzilla. It may not be a foolproof but I don’t think that necessarily devalues the assumptions I’m about to make; particularly when put in the context of data going back to Firefox 5.

Before I begin, let me state that this data is not indicative of any one team’s successes or failures. In large part this data is limited to the scope of bugs flagged with various values of the status-firefox26 flag. This means that there are a large amount of bugs that are not being counted (not every bug has the flag accurately set or set at all) and the flag itself is not necessarily indicative of any one product. For example, this flag could be applied to Desktop, Android, Metro, or some combination with no way easy way to statistically separate the two. Of course one could with some more detailed Bugzilla querying and reporting, but I’ve personally not yet reached a point where I’m able or willing to dig that deep.

Unconfirmed Bugs

Unconfirmed bugs are an indication of our ability to deal with the steady flow of incoming bug reports. For the most part these bugs are reported from our users (trusted Bugzilla accounts are automatically bumped up to NEW). The goal is to be able to get down to 0 UNCONFIRMED bugs before we release a product.

chart_incoming_bugs

What this data tells us is that we’ve held pretty steady over the months, in spite of the ever increasing volume, but things have slipped somewhat in Firefox 26. In raw terms, Firefox 26 was released with 412 of the 785 reported bugs being confirmed. The net result is a 52% confirmation rate of new bug reports.

However, if we look at these numbers in the historical context it tells us that we’ve slipped by 10% confirmation rate in Firefox 26 while the volume of incoming bugs has increased by 64%. A part of me sees this as acceptable but a large part of me sees a huge opportunity for improvement.

The lesson here is that we need to put more focus on ensuring day to day attention is paid to incoming bugs, particularly since many of them could end up being serious regressions.

Regressions

Regressions are those bugs which are a direct result of some other bug being resolved. Usually this is caused by an unforeseen consequence of a particular code change. Regressions are not always immediately known and can exist in hiding until a third-party (hardware, software, plug-in, website, etc) makes a change that exposes a Firefox regression. These bugs tend to be harder to investigate as we need to track down the fix which ultimately caused the regression. If we’re lucky the offending change was a recent one as we’ll have builds at our disposal. However, in rare cases there are regressions that go far enough back that we don’t have the builds needed to test. This makes it a much more involved process as we have to begin bisecting changesets and rebuilding Firefox.

The number of regressions speaks to a potential failure, something that was missed, not accounted for, or unable to be tested either by the engineer or by QA. In a perfect world a patch would be tested taking into account all potential edge cases. This is just not feasible in reality due to the time and resources it would take to cover all known edge cases; and that’s to say nothing of the unknown edge cases. But that’s how open source works: we release software we think is ready, users report issues, and we fix them as fast and as through as we reasonably can.
In the case of Firefox 26 we’ve seen a continued trend of a reduction in known regressions. I think this is due to QA taking a more focused approach to feature testing and bug verifications. Starting with Firefox 24 we brought on a third QA release driver (the person responsible for coordinating testing of and ultimately signing-off on a release) and shifted toward a more surgical approach to bug testing. In other words we are trying to spend more time doing deeper and exploratory testing of bug fixes which are more risky. We are also continuing to hone our processes and work closer with developers and release managers. I think these efforts are paying off.

chart_regressions

The numbers certainly seem to support this theory. Firefox 26 saw a reduction of regressions by 20% over Firefox 25, 25% over Firefox 24, and 57% better than Firefox 17 (our worst release for regressions).

Stability

Stability bugs are a reflection of how frequently our users are encountering crashes. In bugzilla these are indicated using the crash keyword. The most serious crashes are given the topcrash designation. The more crash bugs we ship in a particular release does not necessarily translate for more crashes per user, but it is indicative of more scenarios under which a user may crash. Many of these bugs are considered a subset of the regression bugs discussed earlier as these mostly come about by changes we have made or that a third-party has exposed.
In Firefox 26 we again see a downward trend in the number of crash bugs known to affect the release. I believe this speaks to the success of educating more people in the skills necessary to participate in reviewing the data in crash-stats dashboard, converting that into bugs, and getting the necessary testing done so developers can fix the issues. Before Firefox 24 was on the train the desktop QA really only had one or two people doing day to day triage of the dashboards and crash bugs. Now we have four people looking at the data and escalating bug reports on a daily basis.

chart_crashes

The numbers above indicate that Firefox 26 saw 12% less crash bugs that Firefox 25, 41% less than Firefox 24, and 59% less than Firefox 15, our most unstable release.

Reopened

Reopened bugs are those bugs which developers have landed a fix for but the issue was proven not to have been resolved in testing. These bugs are a little bit different than regressions in that the core functionality the patch was meant to address still remains at issue (resulting in the bug being reopened), whereas a regression is typically indicative of an unforeseen edge case or user experience (resulting in a new bug being filed to block the parent bug).

That said, a high volume of reopened bugs is not necessarily an indication of poor QA; in fact it could mean the exact opposite. You might expect there to be a higher volume of reopened bugs if QA was doing their due diligence and found many bugs needing follow up work. However, this could also be an indication of feature work (and by consequence a release) that is of higher risk to regression.

chart_reopened_bugs

As you can see with Firefox 26 we’re still pretty high in terms of the number of reopened bugs. We’re only 5% better than Firefox 23, our worst release in terms of reopened bugs. I believe this to be a side-effect of us doing more focused feature testing as indicated earlier. However it could also be indicative of problems somewhere else in the chain. I think it warrants looking at more closely and is something I will be raising in future discussions with my peers.

Uplifts vs Landings

Landings are those changes which land on mozilla-central (our development branch) and ride the trains up to release following the 6-week cadence. Uplifts are those fixes which are deemed either important enough or low enough risk that it warrants releasing the fix earlier. In these cases the fix is landed on the mozilla-aurora and/or mozilla-beta branches. In discussions I had last year with my peers I raised concerns about the volume of uplifts happening and the lack of transparency in the selection criteria. Since then QA has been working closer with Release Management to address these concerns.

chart_fixed_by_date

I don’t yet have a complete picture to compare Firefox 26 to a broad set of historical data (I’ve only plotted data back to Firefox 24 so far). However, I think the data I’ve collected so far shows that Firefox 26 seemed far more “controlled” than previous releases. For one, the volume of uplifts to Beta was 29% less than Firefox 25 and there were 42% less uplifts across the entire Firefox 26 cycle compared to Firefox 25. It was also good to see that uplifts trailed off significantly as we moved through the Aurora cycle into Beta.

However, this does show a recent history of crash-landings (landing a crash late in a cycle) around the Nightly -> Aurora merge. The problem there is that something that lands on the last day of a Nightly cycle does not get the benefit of Nightly user feedback, nor does it get much, if any, time for QA to verify the fix before it is uplifted. This is something else I believe needs to be mitigated in the future if we are to move to higher quality releases.

Tracked Bugs

The final metric I’d like to share with you today is purely to showcase the volume of bugs. In particular how much the volume of bugs we deal with on a release-by-release basis has grown significantly over time. Again, I preface these numbers with the fact that these are only those bugs which have a status flag set. There are likely thousands of bugs that are not counted here because they don’t have a status flag set. A part of the reason for that is because the process for tracking bugs is not always strictly followed; this is particularly true as we go farther back in history.

chart_tracked_bugs

As you can see above, the trend of ever increasing bug volume continues. Firefox 26 saw a 21% increase in tracked bugs over Firefox 25, and a 183% increase since we started tracking this data.

Firefox 26 In Retrospective

So there we have it. I think Firefox 26 was an incremental success over it’s predecessors. In spite of an ever increasing volume of bugs to triage we shipped less regressions and less crashes than previous releases. This is not just a QA success but is also a success borne of other teams like development and release management. It speaks to the success of implementing smarter strategies and stronger collaboration. We still have a lot of room to improve, in particular focusing more on incoming bugs, dealing with crash landings in a responsible way, and investigating the root cause of bugs that are reopened.

If we continue to identify problems, have open discussions, and make small iterative changes, I’m confident that we’ll be able to look back on 2014 as a year of success.

I will be back in six weeks to report on Firefox 27.

My Philippine Adventure

A few weeks ago I had the opportunity to embark on a two week vacation to the Philippines with my boyfriend, Genesis. His family being from the Philippines it was a natural first destination to Asia for me. As luck would have it the adventure started well before we were due to depart.

Our original itinerary included a week in the northern island of Luzon and a week on the southern island of Palawan. Each would offer us a different experience: Luzon with an exploration of Philippine civilization and Palawan with it’s natural beauty. Having booked our vacation back in September we had no idea what the future held. Unfortunately the Philippines would have to endure horrible devastation at the hand of Typhoon Yolanda (aka Haiyan), Palawan being directly in her path. While I’m confident they will recover and be stronger than ever, my sympathies go out to the people affected.

Not wanting these events to impact my vacation I decided to turn the week in Palawan into a week exploring various locales around Luzon. As luck would have it Luzon was unaffected by the storm, giving me the opportunity to explore the wonderful history and culture of the Philippines.

Day 1: Departure

It was Friday, our bags were packed, and we were on our way to the airport. I was so excited for this trip. Arriving at the airport, balikbayan box in tow, we discovered our flight was delayed. We’d have to wait until 1:30 in the morning to board but I was not deterred. Nothing was going to diminish my excitement for this vacation.

Sunrise over the Pacific Ocean

Some time later, after a couple movies and a few hours of sleep, I was pleasantly rewarded with an amazing sunrise. We were a couple hours the other side of the international date line and inching our way closer to the Philippines.

When we landed we were greeted by some of Genesis’ family who had hired a van for the remainder of our trip. Stepping out of the airport in Manila at about 7am local time I recall feeling like I had stepped into the midday sun of a Canadian summer.

Market in Manila

After clearing the airport traffic we stopped at a market for some breakfast. We bought some fresh red snapper, crab, shrimp, and squid which they cooked for us. It was an amazing breakfast having spent 14 hours on a plane, and looking forward to another 8 hours to travel by van.

Later that evening we arrived in Solano, Genesis’ hometown. Coincidentally it was his mother’s birthday so we were welcomed with a bit of a feast. After eating and a visit with more of the family it was time to turn in.

Day 2: Bayombong

Our second day in the Philippines was spent exploring the provincial capital, Bayombong; but first a traditional Ilocano breakfast complete with dried fish, banana, veggies, rice, and pandesal (fresh from the trike vendor).

Saint Dominic Church

Following breakfast we spent the bulk of the morning exploring the market in Solano, getting fresh food for that night’s supper. It was a bit overwhelming at first, so many people and so much food, but I soon settled in to my surroundings. After the market we traveled into the provincial capital, Bayombong, visiting the local museum and Saint Dominic Cathedral.

Day 3: Waterfalls and Parrots

On the third day Genesis, his brother Adonis, and I ventured out to visit a nearby waterfall. Traveling by trike, one of the more common forms of transport in Nueva Vizcaya, we headed up a dirt path into the hills near Solano. After about 30 minutes along the rocky, muddy path we reached the trail head. From here it was a short hike across a rickety suspended foot bridge to a waterfall view.



Unfortunately we weren’t able to hike any further since a recent rain made the rocks far too slippery to attempt climbing. On the plus side this hike gave us a bit of an appetite. Upon returning back to Solano, we picked up Genesis’ mother and his aunt to go out for lunch (yes, more food). I thoroughly enjoyed the presentation at this restaurant; the carrot parrot was a nice touch.

Carrot Parrot

Day 4: Banaue

On the fourth day we traveled to Banaue, a town famous for its rice terraces. This trip gave me the opportunity to travel by another popular form of transportation in the Philippines, the Jeepney; often wildly decorated world war 2 era transport vehicles. It was not the most comfortable form of transportation but it was extremely affordable, only costing a few dollars per trip.

Jeepney to Banaue

After a few hours winding our way slowly up the road into the mountains we arrived in Banaue. Dismounting the Jeepney we walked around the village and up to the view point of the rice terraces. Unfortunately we arrived following the harvest and were welcomed by a thick mountain rain, cutting our trip to Banaue a bit shorter than expected. Even still I found the view stunning and well worth the trip.

Town of Banaue
Banaue Rice Terraces

Day 6: Road to Baguio

The sixth day marked the end of our visit in the Cayagan Valley, today we were off to the mountain city of Baguio. The road to Baguio is one of the most scenic roads I’ve ever had the pleasure of traveling. As you leave the Cayagan Valley, the road winds westward up into the mountains, reaching elevations of 1400 meters. Along the way we came across Ambuklao Dam, a hydroelectric facility on the Agno river in Benguet province. Having been on the road for a few hours we decided to stop, stretch our legs, and take in the view.

Ambuklao Dam

After the break we continued on our way. The road continued to snake ever higher until we finally reached the city of Baguio. Much to my surprise the climate here was completely different than what I’d expected from the Philippines; 23C and minimal humidity; a stark contrast to the 35C and plenty of humidity we experienced days before back in Solano.

Mines Park Lookout, Baguio City
Baguio City

After checking in to our hotel we went to Wright Park for a picnic and walked around the tourist attractions in the area. Unfortunately after a couple hours we would have to say good bye to Genesis’ family. It was time for them to head home and for Gensis and I to carry on with our vacation. The departure was a bit emotional but I’m grateful I was able to share some of my introduction to the Philippines with his family.

Day 7: Exploring Baguio

On the seventh day, being our only full day in Baguio, we wanted to explore as much of it as possible. The first stop was to the botanical garden where we got acquainted with some of the native plants, artwork, and peoples.

Baguio Botanical Garden

Of course the trip wouldn’t be complete without a trip to the SM shopping mall and a stop a Jolibee. As luck would have it, it started to rain soon after we finished up at the botanical garden. The mall provided the perfect distraction to wait out the rain. Once it let up we went to visit the Baguio museum where we got to learn the city’s history and culture.

Baguio Museum

Following the museum we continued on to Burnham Park, named for the American architect (Daniel Burnham) who designed several buildings in Baguio as part of the Philippine Commission in the early 1900s. We strolled around the park for a while before heading back to the mall for some more shopping and a movie before we turned in for the evening.

Burnham Park

Day 8: The Road to Laoag

Following a good night’s rest and a traditional Baguio breakfast of dried fish, rice, and egg it was time to pack our bags and catch a bus to Laoag. The ride itself would take us west through the mountains to the coast and then north to Laoag through several (very old) cities and towns. As with our drive too Baguio the view going down the other side of the mountains was absolutely stunning.

Mountains west of Baguio

 

After a couple hours of our bus winding down the mountain we were nearing the coast. At this point the road veered north, worming its way along the Philippine coast, teasing us with sights of the sea several times along the way. It would be several hours before we reached the historic city of Vigan. Unfortunately for us this was just a stop along the way. The bus would stop here for a few minutes to transfer some passengers before continuing to our destination of Laoag.

Gateway to Vigan

A couple hours later we arrived on the outskirts of the city of Laoag. From here it was a short ride on a trike through the countryside to the coast.

Fort Ilocandia

As luck would have it we arrived at our destination, Fort Ilocandia, just in time to watch the sun set.

Sunset at Fort Ilocandia

After a quick stroll around the grounds of Fort Ilocandia it was time to turn in for the night.

Fort Ilocandia Fountain at Night

Day 9: Laoag City

We had a pretty lazy start to our ninth day in the Philippines. We slept in, had breakfast, went for a long stroll along the beach, had lunch, and then went for a swim. Following that we decided to spend the afternoon exploring Laoag city. It was a short ride from the hotel on the coast to the city.

Laoag Bell Tower

 

One of the main tourist attractions to Laoag is the Sinking Bell Tower. The tower is believed to have been built in 1612 by the Augustinians and leans slightly to the north. It’s location is fairly central to other historical architecture like the St William’s cathedral, Ilocose Norte capital building, and court house. I can’t help but be reminded while visiting these places that the Philippines is a country with deep roots in Catholocism and they’ve done well preserving their roots.

After spending a couple hours exploring Laoag it was time to head back to Fort Ilocandia where we were treated to another amazing sunset.

Sunset at Fort Ilocandia

Day 10: Relaxing at Fort Ilocandia

On our tenth day in the Philippines we decided to stay close to home and just relax; I think we needed a bit of a break from all the travel. After breakfast we went down to the beach and watched some local fishermen bringing in their nets they had set out the previous evening.



Following this we went for a swim in the ocean. I was really quite surprised with how warm the water was. This wasn’t my first time swimming in an ocean (I swam in the waters off Prince Edward Island when I was a kid), nor was it my first time in the Pacific (I took a dip in Hawaii last year), but it was my first time in the South China Sea and it was the warmest natural body of water I have experienced to date. We spent a couple hours playing in the waves, trying and failing to keep from swallowing the water. Once we achieved optimum sodium levels we decided it was time for lunch.



On our way to lunch we discovered Fort Ilocandia was home to a small zoo, we decided this would be a good way to spend the afternoon. Their zoo was home to several animals; not least of which were crocodiles, monkeys, and a pair ostrich.

After a walk with the animals we wanted to explore more of what Fort Ilocandia had to offer. Unfortunately some of the more exciting activities (like snorkeling and hot air ballooning) were only open to groups of four or more. Being off-season we basically had the hotel to ourselves so these activities were not accessible to us. It’s a bit regrettable that we weren’t able to enjoy these activities but perhaps we’ll have better luck next time, or come with a larger group next time.

The rest of the day was spent relaxing by the water.

Day 11: Paoay City

Our eleventh day in the Philippines we decided to get back to exploring. Following breakfast we hired ourselves a trike with the goal of seeing Paoay City. The main attraction in Paoay, like many other cities in Ilocos Norte, was an opportunity to take a step back in time and witness some centuries old architecture. In the case of Paoay this would mean a visit to St. Augustine’s Cathedral, a church built in 1710.

St Augustine Cathedral, Paoay
St Augustine Cathedral, Paoay

After strolling around the grounds of St Augustine’s cathedral we were escorted by our trike driver to the Ferdinand E. Marcos Presidential Center. The visit itself was a bit somber as I would learn this to be the final resting place of Ferdinand E Marcos, former president of the Philippines. That aside, it was interesting to see the way homes used to be built in the Philippines; an interesting mixture of cement, local wood, and windows made from translucent capiz shells.

Ferdinand E Marcos Presidential Center

Our next stop on the journey was a bit more exciting. Our trike driver took us back up the coast to a place somewhat off the beaten path. It was here that I would enjoy probably my most thrilling experience on this trip, riding in the back of a 4×4 across the Paoay Dunes.



It was quite windy that day, but as our vehicle danced playfully across the dunes, sand blasting in our face, holding on for fear of being catapulted, I found myself forgetting about all the worries in my life. I could think of nothing else but the fun I was having. I was living in the moment.

Paoay Dunes

After our joy ride was over, and a few minutes to calm ourselves down, we were off to our next destination. We drove back out to the main road, following it around Paoay lake for several kilometers until we reached the far side. Here we reached a rather large house which I would soon learn was one of the many homes of the Marcos family. This home in particular was set up as a museum, not only of the former president’s life but of the people of the Ilocos region.

Malaca├▒ang of the Norte

 

We wandered the grounds of this amazingly beautiful property for quite a while. As luck would have it, upon leaving we met a local man who knew the area well and offered to be our tour guide. We decided to hire him for the next, and what would be our last, two days in Ilocos Norte. Upon our return to Fort Ilocandia that evening we were gifted another amazing sunset.

Sunset over South China Sea

Day 12: Pagudpud

On our twelfth day we ventured north along the coast toward Pagudpud. The driver we met the day before escorted us the entire way, showing us some sights we may have missed if we had tried to go it alone. Our first stop was a salt mill just off the road to Pagudpud.

Salt Milling

It was interesting to see how they made the salt, something I had perhaps taken for granted before visiting the Philippines. After farming and milling the rice grain the waste product is the grain casing. Instead of throwing out this casing they use it as fuel for fire. The fire is used to heat salt water from the ocean to its boiling pont. As the water boils the sodium content distills into salt crystals which are then farmed our of the water. It was really interesting to watch this process unfold before my eyes.

The next stop along the way was a very old lighthouse perched on top of a rocky hill just east of the coast. The lighthouse was constructed and first lit in 1892 and still functions today, marking the northwestern most point of the Philippines.

Cape Bojeador Lighthouse

We continued our journey up the coast to the town Burgos to visit the Kapurpurawan rock formation, a limestone monument scuplted by the elements over thousands of years.

Kapurpurawan Rock Formation

 

It was a little bit of a hike to get down the hillside to visit the rocks but it was well worth the journey. The geology was unlike anything I had seen in the Philippines and I had found myself, for the second time, challenging my assumptions of this being a country of beaches and palm trees. The first time of course was seeing pine trees and experiencing temperate weather in Baguio. The hike also presented me with an opportunity to make a new friend.



I found this little guy trying to cross the pathway. After capturing this shot I helped him safely to the other side.

Further up the coast we stopped by a wind farm. It was somewhat unexpected but unsurprising at the same time, as the coast in these parts was really quite windy.

Philippine Wind Farm

 

It was rather curious to see a country without the financial capabilities of Canada embracing this technology.

We carried on to our destination, soon arriving in Pagudpud. As we drove further along the coast, coastline turned to mountains and we found ourselves in a bit of rain but that did not deter us.

Welcome sign in Pagudpud

 

The road continued and eventually lead us back out to the coast where we came across a sleepy little resort. We decided this would be a good place to stop for some food before heading back down to Laoag for the evening. We walked into a small roadside restaurant where a woman cooked us up some fresh tuna and prawns.

Pagudpud Beach

 

After a healthy dose of seafood we made our way back “home” to rest up for our final day in Ilocos Norte. Our driver, Mario, to return in the morning.

Day 13: Vigan

Our thirteenth day in the Philippines was also our last day in Ilocos Norte. Since we didn’t get to see much of Vigan on our first way through we decided we wanted to see it before we left. The trip to Vigan was a few hours round trip so we had plenty of time to fit it in before flying to Manila later that evening.

Old City of Vigan

We arrived in Vigan shortly before lunch and spent much of the mid-day walking around this old city. I was yet again amazed to see the preservation of their rich history and ways of life. Many of the buildings, built hundreds of years ago, still standing despite decades of pummeling by nature. Many of the people still practicing crafts using the traditional techniques taught to them by their ancestors.

Pottery Factory in Vigan
Filipino making a Vase

I felt so grateful that I was able to see this city on foot and to meet the people of Vigan. Unfortunately I was unable to stay for long as we had to eat then return to Laoag so we could catch our flight to Manila.

Much later that evening we had landed safely in Manila and after quite some time in Manila’s infamous traffic we checked into our hotel.

Makati

We were only in Manila for two nights before we had to head home. I found Manila to be quite hectic in contrast to the much more relaxing lifestyle I had experienced throughout my journey around Luzon. We spent much of our time walking to and from restaurants and shopping malls. Much of my experience in Manila reminded me of why I left Toronto (too busy, too noisy, too smoggy).

Don’t get me wrong though. I enjoyed my time in Manila. It was just different, and a bit of a disorienting.

That said I still look at my time Manila as positive experience. After all, it gave me the opportunity to visit with Adonis (Genesis’ brother) and I able to experience a filipino hilot massage, complete with hot banana leaf across my back.

Back to Reality

This whole vacation has been an amazing journey. It has been a feast for the eyes and the mind; and at times just a feast.It was relaxing, busy, enlightening, and rejuvenating. I feel like I was able to have an authentic Philippine experience with a dash of frivolity. Perhaps the best part was being able to share some of that with Genesis and his family.

In the end my time in the Philippines was successful. I came back to my life in Vancouver re-energized and enlightened, something I hope to achieve in all my vacations.

I’m already looking forward to returning to the Philippines someday, perhaps next year. There is so much more this country has to offer and I can’t wait to experience it.

 


If you want to see more pictures of my trip, I’ve posted a photo album on trovebox.

Firefox 25 Bug Stats

We hit another milestone this week. After 24 weeks, 12 Betas, and 3 RCs, Firefox 25 was tested, signed-off, and shipped to the general public. Since Firefox 10 I’ve been collecting data using the bugzilla status flags in an effort to determine what impact our policies and efforts are having on the quality of the product. I’ve decided it would be good of me to move that project over to my blog. For one, it makes this data a little bit more discoverable. For another, this will give me an excuse to do something I don’t have a good track record of following through on: blogging.

The Data

Before I go into the numbers let me state that I have no background in Metrics or Statistics, whatsoever. I will not, nor can I make any conclusions about the data. I am merely presenting it here to you, the community, in a way I feel to be interesting. Feel free to comment on this or any related posts if you have any suggestions about how I may improve my methodology or to make more accurate, informed conclusions.

With that, here are the numbers for Firefox 25. Anything in green is a success that we need to continue to build on. Anything in red is an area that needs greater attention as we move forward.

  • 382 verified fixes (1% improvement)
  • 649 unverified fixes (14% improvement)
  • 439 unfixed bugs (14% improvement)
  • 245 wontfix bugs (10% improvement)
  • 129 unconfirmed bugs (34% degradation)

We’ve made continuous improvement to our processes around fix verification. This includes focusing more on high priority bugs in the pushlogs, verifying fixes earlier in the cycle whenever possible, documenting our processes, and trying to involve more volunteers. tells a story that I’ve seen repeated the last few releases. The addition of a third QA release lead (Tracy Walker) has allowed us to scale somewhat in the last couple of cycles (we are now spending more time on Aurora than we were before). I suspect the move to two Betas per week has also contributed here in that it’s allowed us to do more focused testing on fewer changes more frequently. Unfortunately we aren’t putting enough effort into unconfirmed bug triage, something I hope to improve upon as I switch focus to Firefox 28 next week.

The Data Visualized

Now I’d like to share the visualization of this data. Please keep in mind that I’m not drawing any conclusions here, I’m merely visualizing the data in ways I believe to be interesting. I hope you do too.

Firefox25-Bug-Breakdown
Breakdown of bugs by status per Firefox version
Number of fixes landed in each branch for each Firefox version
Number of fixes landed in each branch for each Firefox version
Percentage of fixes verified prior to release for each Firefox version
Percentage of fixes verified prior to release for each Firefox version
Active instances of Firefox compared to unconfirmed bugs shipped each release
Active instances of Firefox compared to unconfirmed bugs shipped each release
Active instances of Firefox compared to number of fixes verified in that release
Active instances of Firefox compared to number of fixes verified in that release

 More to Come

That’s Firefox 25 from my perspective. I hope you have found this interesting. I will continue to share this data and update the visualizations every six weeks. You are encouraged to provide me feedback and ask questions on the data, the comparisons I’ve visualize, and my methodology.

Until next release, enjoy.

Summer 2013 Testday Results

It’s hard to believe another quarter has come and gone. I’m writing today to reveal the contributions made through QA organized testdays over the last three months. I’m happy to say that the summer was kind to us. We saw an end to the slump in the number of testdays organized (fig. 1) and 11 of the bugs reported through testdays resulted in fixes to Firefox.

Fig 1: Frequency of Testdays

Of course none of this would be possible without the contributors themselves. There are too many to mention by name but I would like to call out and thank the following volunteers for reporting bugs which resulted in fixes this quarter:

Naturally it is not all good news. I have noticed a slight stagnation in the participation trend (fig 2.).

Fig 2: Testday Attendance and Participation

I hope that we can reverse only 3 of the 11 events this quarter were attended better than average. I’d like to see if there is something we can do to improve that going forward. I think a part of this may have to do with the lack of mentorship to get someone from being a newcomer to becoming repeat contributor. Over 40% of our attendees this quarter were newcomers which tells me that we’re doing enough to attract people but could be doing more to mentor people (fig 3.).

Fig 3: Breakdown of Testday Contributors

As I close out this post I want to thank everyone who contributed recently and I hope to see you all return to make the next three months better than the last.