Testing the JIFFEE System

Overview

Browsers are infamous for not all handling JavaScript in quite the same way. On the other hand, a critical design goal of JIFFEE was to make it easy-to-use, which means "it should just work." The only way to reconcile these two things is to test the whole JIFFEE framework on as many browser configurations as possible. (There are also some important design techniques used, such as sticking to the "core" language as much as possible, and keeping all the HTML code and DOM access isolated to one or two small sections of code.)

I may well be the only guy on my block to have a bazillion different browsers and browser versions installed on his laptop, and this is probably even more versions than most commercial software houses routinely test or support, but I don't think this is overkill. Remember that an important part of my target audience consists of students using old outdated school computers, so I really do want JIFFEE to function (correctly!) on as many of these combinations as I can reasonably manage to make work.

Manual testing of this many configurations would be completely impractical, so I use the same suite of over 100 automated tests that keep JIFFEE development on track. All the automated tests are run with JsUnit version 2.1.

The following codes indicate the level and results of testing:

pass	This version of JIFFEE running on this configuration passed all the automated tests. This doesn't guarantee perfection, but it improves the odds enormously.
smoke	This version of JIFFEE running on this configuration passed a "smoke-test," i.e. I ran a game and nothing blew up. Typically I settle for this if the browser is so old that it can't run the automated test system (JsUnit).
fail	This version of JIFFEE running on this configuration does not work - it fails either an automated test or a manual test. It is possible that I might someday figure out how to get these configurations to pass, but don't hold your breath. These entries are included in the table so you won't waste your time on things that are known to be too old to work.

The current version of JIFFEE is 0.3. Whenever there is a significant new version of JIFFEE released I try to run the tests on as many configurations as possible, but there are some to which I do not have regular access.

Testing results

Operating System	Browser	Browser Version	Year	JIFFEE Version(s)	results
Windows 98 Second Edition 4.10	Internet Explorer	5.0	1999	0.3	fail
Windows 98 Second Edition 4.10	Firefox	1.0.8	2005	0.3	pass
Windows NT 4.0 Service Pack 6	Internet Explorer	6.0.2800.1106	2001	0.3	pass
Windows NT 4.0 Service Pack 6	Firefox	1.0.8	2005	0.3	pass
Windows 2000 Professional 5.0 Service Pack 4	Internet Explorer	6.0.2800.1106	2001^[1]	0.3	fail
Windows 2000 Professional 5.0 Service Pack 4	Firefox	1.0.6	2005	0.3	pass
Windows XP Home Edition Service Pack 3	Internet Explorer	7.0	2006	0.3	pass
	Firefox	1.0	2004	0.3	pass
		1.5	2005	0.3	pass
		2.0	2006	0.3	pass
		3.0.3	2008	0.3	pass
	Netscape Navigator	4.80	2002	0.3	fail
		6.01	2000	0.3	pass
		7.01	2002	0.3	pass
		8.0.1	2005	0.3	pass
		9.0.0.1	2007	0.3	pass
	Safari	3.1.2	2008	0.3	pass
	Opera	7.50	2004	0.3	fail
		8.0	2005	0.3	pass
		9.60	2008	0.3	pass
	Chrome	0.2.149.30	2008	0.3	pass
Windows Vista	Not yet tested, but no problems anticipated with any modern browsers.
iMac G3 OS X 10.3.9	Safari	1.3.2	2005	0.3	fail
Macintosh OS X 10.5.5	Safari	3.1.2	2008	0.3	pass
Ubuntu 8.04 LTS (Hardy Heron)	Firefox	3.0.3	2008	0.3	pass
OLPC XO-1	Supplied with Sugar	Initial production run	2007	0.3	smoke
Mobile Phones	T-Mobile G1	Initial release	2008	0.3	pass
Mobile Phones	Apple iPhone	?	2008?	0.3	pass

^[1] Yes, this is exactly the same browser that worked fine under Windows NT.

Mobile Platforms

Isn't it a bit silly to be testing JIFFEE on cell phones? Well, look at it this way: As of 2008 the number of data-capable mobile phones in the world is roughly triple the number of PCs. Nokia alone ships a million handsets a day. More and more of your potential audience is going to have mobile, and many of them will have only mobile. A potential audience of over 3 billion and growing isn't silly at all. If you want more evidence, go read the annual report of any big Internet company and see if they think it's a good idea to ignore mobile phones.

To make JIFFEE really usable on a mobile phone will require a separate HTML template specialized for the smaller screens. I haven't had time to create one of those yet, but it's coming, but in the meantime it makes sense to keep the core game engine compatible.