Closed Bug 763099 Opened 13 years ago Closed 13 years ago

Design: Graphs for FN14 GA benchmarks

Categories

(Marketing :: Design, task)

Product:
Marketing
Component:
Design
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jfu, Assigned: tyronflanagan)

References

Details

Attachments

(9 files, 2 obsolete files)

Benchmark Graph - FFxAurora - Printable PDF
536.51 KB, application/pdf
Details
Benchmark Graph - FFxAurora - Printable TIF - CMYK 300dpi
179.01 KB, application/zip
Details
Benchmark Graph - FFxAurora - Web 640x480
32.47 KB, image/png
Details
Benchmark Graph - FFxAurora - Web 1280x932
69.15 KB, image/png
Details
Benchmark Graph - FFxAurora - Canvas Rendering - Web
30.22 KB, image/png
Details
Benchmark Graph - FFxAurora - Website Panning - Web
32.44 KB, image/png
Details
Benchmark Graph - FFxAurora - Canvas Rendering - Print
1.93 MB, application/postscript
Details
Benchmark Graph - FFxAurora - Website Panning - Print
1.98 MB, application/postscript
Details
Previous JS benchmark example
120.54 KB, image/png
Details
Hi John/Tara, We have some benchmark numbers for Firefox Android that we'd like to make pretty so we can put them into press kits/blog posts etc. We're in a bit of a rush though, would EOD June 11 be possible? The higher the numbers the better - how can we make it optically look like this means faster even though the numbers are bigger? Graph title: Page load speed Subtext: The higher the number the better Firefox for Android (latest version): 220 Firefox for Android (previous version): 80 Default Android Browser: 65 Opera: 112 Chrome for Android: 102 Dolphin Browser: 74 Thank you!
**horizontal bar graph
Also clarification on the unit of measurements: Number of unique frames in a capture. Higher is better (higher # of frames = faster rendering = page load speed)
(In reply to Jaclyn Fu from comment #2) > Also clarification on the unit of measurements: > > Number of unique frames in a capture. Higher is better (higher # of frames = > faster rendering = page load speed) Minor correction: this benchmark doesn't have anything to do with page load speed. "higher # of frames = faster rendering" is perfectly accurate though.
Thanks for the correction! Let's move forward with calling this graph "Page rendering" (William - that's accurate?)
(In reply to Jaclyn Fu from comment #4) > Thanks for the correction! > > Let's move forward with calling this graph "Page rendering" (William - > that's accurate?) "Canvas rendering" would be more precise. Page rendering sounds pretty vague to me -- this benchmark measures performance only for a specific type of page.
(In reply to William Lachance (:wlach) from comment #3) > (In reply to Jaclyn Fu from comment #2) > > Also clarification on the unit of measurements: > > > > Number of unique frames in a capture. Higher is better (higher # of frames = > > faster rendering = page load speed) > > Minor correction: this benchmark doesn't have anything to do with page load > speed. > > "higher # of frames = faster rendering" is perfectly accurate though. It also occurred to me last night that I could give you numbers in terms of frames per second on this benchmark. The proportions would be approximately the same, but the metric would probably be more familiar to the technical press. Frames per second is commonly used in graphics card benchmarks, for example. e.g. http://www.tomshardware.com/charts/2011-gaming-graphics-charts/Aliens-vs.-Predator-Gamer,2667.html
ok cool, let's do frames per second. Is it still higher the number the better and measuring canvas rendering?
(In reply to Jaclyn Fu from comment #7) > ok cool, let's do frames per second. Is it still higher the number the > better and measuring canvas rendering? Yep, I'll post the new numbers shortly.
Frames per second benchmarks: Firefox for Android (previous version): 13.2001638884 Firefox for Android (latest version): 39.273391046 Opera: 21.6094041976 Chrome for Android: 19.6808266828 Dolphin Browser: 14.1638832422 Default Android Browser: 12.0694154442
I've asked Ty to help with this. EOD today isn't really doable, but we'll see what we can get done tmrw.
Do we need to include which Android OS version was used to obtain these benchmarks?
William - can you provide what device and version you used? thanks!
(In reply to Jaclyn Fu from comment #12) > William - can you provide what device and version you used? thanks! Samsung Galaxy Nexus - Android 4.0.4.
Love the graphic, thanks Ty. One thought: we should consider changing the title of the graphic...calling it "Canvas rendering speed" actually makes us look pretty slow if you don't read the fine print (the Firefox bar is much larger than the rest, which make it look like we take longer). In other words, I like the visuals, but we should make sure it's clear to people what they're saying. Could even give it an overall headline like "Firefox for Android is Fast!" (note - that's a terrible headline, don't really use it) and then add a subhead that explains what the graphic is. Jaclyn, how would you like to position this?
+1 love the visual! Agreed, at first glance the perception might be these results show the opposite of what it really is. How about something like... Do more, faster with Firefox for Android Canvas rendering performance Is that lame.
William - did you use Opera Mini or Opera Browser?
Matej, can you give us a quick copy consult here? See comment #18 and 19 for details. Thanks!
Ty - and let's add a note about what site we used (William, doubling checking that we used nyt.com) "*Results based on using nyt.com on a Samsung Galaxy Nexus on 4.0.4" Thanks!
On the whole, I don't feel like "canvas rendering speed" is adding a lot as something big and prominent at the top of the image. Could that go at the bottom with the fine print? "Canvas rendering speed based on a Samsung Galaxy Nexus running Android 4.0.4 on nyt.com" I would actually suggest putting the "Frames Per Second" and all that copy at the top of the graph because that clearly states that a higher number is better. That way we wouldn't need a subhead, either, and the headline could be one of the following: Get up to speed with Firefox for Android Browse faster with Firefox for Android Get to the Web fast with Firefox for Android One other question: Is there a reason we're using green for our performance? Seems like we've been using blue lately for mobile. Or, obviously, orange.
Replying to stuff in bulk: On the graph: The "better page load time" on the bottom of the graph is inaccurate. This measurement doesn't have anything to do with page load time. I'd either take it out or replace it with "smoother animations" (e.g. higher # of frames = faster rendering = smoother animations). On the page: This isn't the New York Times, it's a synthetic benchmark I created myself for measuring canvas performance with Eideticker. An example of what the benchmark looks like is here: http://wrla.ch/eideticker/dashboard/videos/video-1338408277.15.webm In general this graph would indicate that the new Firefox for Android would have better performance in highly interactive sites and games. On version of Opera (Opera mini vs. Opera browser): I used the Opera browser. I don't think this benchmark would actually work on Opera mini.
Oh, and one random comment on the graph: I don't think we need to get down to 4 digits of precision. ;) The results of this test are fairly noisy, and you might get values 1 or 2 different per run. I'd use at most one digit. e.g. Firefox for Android (previous version): 13.2 Firefox for Android (newest version): 39.3 etc.
Thanks! hm if it's smoother animations, then I don't we can say this graph shows how fast Firefox for Android is. Unless we just say higher # of frames = faster rendering and just leave it at that Ok Ty - let's not add the bit about nyt.com in the description then. Can we add "Browser" to "Opera"? And also let's simplify the digits slater/matej - what should we do about the copy about rendering and how to describe?
(In reply to Jaclyn Fu from comment #26) > hm if it's smoother animations, then I don't we can say this graph shows how > fast Firefox for Android is. Unless we just say higher # of frames = faster > rendering and just leave it at that The technical press/advertising often talks about how "fast" graphics are based on metrics like these. For example, look at the section on "outstanding performance" at http://www.apple.com/imac/performance.html. I'm obviously not an advertising expert, but I don't think it's too much of a stretch to talk about speed here.
Ok good to know. We can spin it as "performance" then.. What about.. Canvas rendering performance Frames per second: higher # of frames = faster graphics rendering
I would still keep "canvas rendering performance" out of the header area (I don't think that's going to mean a lot to people), but would make what the graph is representing more prominent (the frames per second stuff). Based on the last few comments, it sounds like "Get up to speed with Firefox for Android" would still work for the headline.
I think since this is mainly going to be a Press graphic, they should know what canvas rendering is To sum up: For the note on the bottom, *Results based on new Eideticker Canvas Benchmark using Samsung Galaxy Nexus on 4.0.4. Opera -> Opera Browser Simply digits to just one digit after the decimal is fine
We also have a second set of benchmark results we'd like to create into a graph as well: This one measures performance while panning a website while using cnn.com: Title - Website Panning Performance Subtext: Frames per second over capture (higher # of frames = smoother rendering): Firefox (old): 10.2 Opera Browser: 9.6 Chrome for Android:11.9 Dolphin Browser:14.2 Default Android Browser: 13.9 Firefox for Android (new): 20.5 Thanks!
What's the timing for this 2nd graph? Ideally we'd be given more heads up for this sort of thing.
It would be great if we could have both graphs in the reviewer's guide...but if it's not possible then can we get it in time for the blog post? (June 26)
Ok. Ty, let us know what's doable. At the least, we should get the first one handed off today.
Please let me know if this covers all the changes you asked for including copy tone.
Once all changes have been made/approved, i'll go ahead and produce a printable version of each graph
These look great to me! Everything looks technically correct as well. I have just a few final suggestions on clarifying content (I guess it's up to Jaclyn to approve these, this is just IMO): 1. For the first graph, instead of just saying 4.0.4, I would say "Android 4.0.4" (makes it more clear what you're talking about) 2. For the second graph, I'd put down the version of the operating system as well (so: "Results based on performance measures while panning a website using http://www.cnn.com using Samsung Galaxy Nexus on Android 4.0.4")
Thanks Ty! Jaclyn, please give the signoff as soon as you can so we can get these over to Rhonda for the reviewer's guide.
Just to chime - I agree with William's feedback - think we should add those small descriptors to make it very clear how we got these #'s
> 1. For the first graph, instead of just saying 4.0.4, I would say "Android > 4.0.4" (makes it more clear what you're talking about) +1 > 2. For the second graph, I'd put down the version of the operating system as > well (so: "Results based on performance measures while panning a website > using http://www.cnn.com using Samsung Galaxy Nexus on Android 4.0.4") +1 Also for the second graph can we label the old version as "Firefox for Android (previous version)" to keep it consistent with the first graph? Thanks Ty!
Attachment #633137 - Attachment is obsolete: true
Attachment #633138 - Attachment is obsolete: true
Hi Ty, spoke to slater about this, but we're getting in some new benchmark data tonight and will need a tight turn around to make into a pretty graph for tomorrow morning will post in here as soon as we get it, thank you!
Sounds good. If youre able to get me anything tonight, I can get it created this for reviewing tomorrow in the AM.
If we can provide the metrics this evening, when's the earliest we can have a review tomorrow morning EDT time? We're up for waking up early to review
Assignee: tshahian → tyronflanagan
Status: NEW → ASSIGNED
any luck on getting those metrics tonight?
Status: ASSIGNED → NEW
Hey Ty, sorry...looks like they're slowly trickling in. Don't wait up for it tonight, they'll be in your inbox when you're up tomorrow? :)
Thanks Ty! Jaclyn, just to be clear, this is basically just updating the earlier graphic with new data?
With different data (Javascript benchmark) - but same in the same format as the previous data
Blocks: 719894
Hey Ty, We finally have numbers! Sorry about the delay. This is a different benchmark from the one we previously did, and I have an example of something we did in the past for JS benchmarks. Can we do something similar? Thank you! We have results from 3 benchmarks: Kraken, Sunspider, V8 Sorting results by Android 2.2 (Froyo), 2.3 (Gingerbread), 4.0 (ICS) Firefox for Android vs Stock Browser Besides on the axis, we don't need to show the actual numbers of the results KRAKEN (lower the better) 2.2 -Firefox: -Stock: 2.3 -Firefox: 55721 -Stock: Browser not responsive 4.0 -Firefox: 34058 -Stock: 41950 SUNSPIDER (lower the better) 2.2 -Firefox: -Stock: 6836 2.3: -Firefox: 3139 -Stock: 5313 4.0 -Firefox: 1687 -Stock: 1875 V8 (higher the better) 2.2 -Firefox: 420 -Stock: 178 2.3 -Firefox: 459 -Stock: 215 4.0 -Firefox: 877 -Stock: 1173 We have a couple of holes we sitll need to fill in but is this enough to get started? Will ping with the remaining data Thank you!!!! What do you estimate the turnaround to be?
Just to verify, I'm creating 3 separate graphs correct? (1) KRAKEN; (2) SUNSPIDER; (3) V8 Turnaround is all dependent on how soon you can get me the rest of the data.
I hate to throw a wrench in this now, but seeing that we're only 25% faster, I think that's going to be hard to make look awesome in a graph. Since we are trying to change the conversation and point to Eideticker, I think we should try to create a graph around those results that shows up much faster. Jaclyn - can you update w/Eideticker results here?
I'm obviously delirious - Ty already made those graphs in this bug! Jaclyn/Dave - I assume we want to use the panning one but please confirm and let Ty know if there are any changes to be made. Otherwise, Ty I think we'll just need your help dropping this is the rev guide file.
Ping Ty - can we drop this into the rev guide?
Adding Val and Tetsuya to access the high res print and web performance graph attachments- there is one for panning and one for canvas
Can we close this? If so, please close.
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: