Testing the JIFFEE System

Overview

Browsers are infamous for not all handling JavaScript in quite the same way. On the other hand, a critical design goal of JIFFEE was to make it easy-to-use, which means "it should just work." The only way to reconcile these two things is to test the whole JIFFEE framework on as many browser configurations as possible. (There are also some important design techniques used, such as sticking to the "core" language as much as possible, and keeping all the HTML code and DOM access isolated to one or two small sections of code.)

I may well be the only guy on my block to have a bazillion different browsers and browser versions installed on his laptop, and this is probably even more versions than most commercial software houses routinely test or support, but I don't think this is overkill. Remember that an important part of my target audience consists of students using old outdated school computers, so I really do want JIFFEE to function (correctly!) on as many of these combinations as I can reasonably manage to make work.

Manual testing of this many configurations would be completely impractical, so I use the same suite of over 100 automated tests that keep JIFFEE development on track. All the automated tests are run with JsUnit version 2.1.

The following codes indicate the level and results of testing:

pass This version of JIFFEE running on this configuration passed all the automated tests. This doesn't guarantee perfection, but it improves the odds enormously.
smoke This version of JIFFEE running on this configuration passed a "smoke-test," i.e. I ran a game and nothing blew up. Typically I settle for this if the browser is so old that it can't run the automated test system (JsUnit).
fail This version of JIFFEE running on this configuration does not work - it fails either an automated test or a manual test. It is possible that I might someday figure out how to get these configurations to pass, but don't hold your breath. These entries are included in the table so you won't waste your time on things that are known to be too old to work.

The current version of JIFFEE is 0.3. Whenever there is a significant new version of JIFFEE released I try to run the tests on as many configurations as possible, but there are some to which I do not have regular access.

Testing results

Operating System Browser Browser
Version
Year JIFFEE
Version(s)
results
Windows 98
Second Edition 4.10
Internet Explorer 5.0 1999 0.3 fail
Firefox 1.0.8 2005 0.3 pass
Windows NT 4.0
Service Pack 6
Internet Explorer 6.0.2800.1106 2001 0.3 pass
Firefox 1.0.8 2005 0.3 pass
Windows 2000
Professional 5.0
Service Pack 4
Internet Explorer 6.0.2800.1106 2001[1]
0.3 fail
Firefox 1.0.6 2005 0.3 pass
Windows XP
Home Edition
Service Pack 3
Internet Explorer 7.0 2006 0.3 pass
Firefox 1.0 2004 0.3 pass
1.5 2005 0.3 pass
2.0 2006 0.3 pass
3.0.3 2008 0.3 pass
Netscape Navigator 4.80 2002 0.3 fail
6.01 2000 0.3 pass
7.01 2002 0.3 pass
8.0.1 2005 0.3 pass
9.0.0.1 2007 0.3 pass
Safari 3.1.2 2008 0.3 pass
Opera 7.50 2004 0.3 fail
8.0 2005 0.3 pass
9.60 2008 0.3 pass
Chrome 0.2.149.30 2008 0.3 pass
Windows Vista Not yet tested, but no problems anticipated with any modern browsers.
iMac G3
OS X 10.3.9
Safari 1.3.2 2005 0.3 fail
Macintosh
OS X 10.5.5
Safari 3.1.2 2008 0.3 pass
Ubuntu
8.04 LTS
(Hardy Heron)
Firefox 3.0.3 2008 0.3 pass
OLPC
XO-1
Supplied with Sugar Initial production run 2007 0.3 smoke
Mobile Phones T-Mobile G1 Initial release 2008 0.3 pass
Apple iPhone ? 2008? 0.3 pass

[1] Yes, this is exactly the same browser that worked fine under Windows NT.

Mobile Platforms

Isn't it a bit silly to be testing JIFFEE on cell phones? Well, look at it this way: As of 2008 the number of data-capable mobile phones in the world is roughly triple the number of PCs. Nokia alone ships a million handsets a day. More and more of your potential audience is going to have mobile, and many of them will have only mobile. A potential audience of over 3 billion and growing isn't silly at all. If you want more evidence, go read the annual report of any big Internet company and see if they think it's a good idea to ignore mobile phones.

To make JIFFEE really usable on a mobile phone will require a separate HTML template specialized for the smaller screens. I haven't had time to create one of those yet, but it's coming, but in the meantime it makes sense to keep the core game engine compatible.