Currently, quite a bit of manual testing is required before release; partly this is due to the fact that some of the browsers we support are a bit buggy, but partly this is because we just don't run all of our automated tests as part of the build.
- Multiple Browser Versions: IE6, IE7, IE7 on Vista; Firefox 1.5.x, 2.0.x ... The best way to handle this is to use the Selenium RC Self Test which really just tests the browser launchers and Selenium Core. (Automation: Possible, except for crashing bugs in the browsers. IEHTA bug SEL-390
, konqueror bug SEL-342, Opera on Linux bug SEL-350.)
- Element highlighting: We don't have a good way to test to ensure that the visual effect actually happens... so you'll need to try it on all of our supported browsers to look and see. More generally, make sure the rest of our CSS GUI looks more-or-less OK on all browsers. (Automation: Hardest. [Impossible?])
- IEHTA "Simple": You can run TestRunner.hta directly from the command-line. Run it like this:
testrunner.hta "auto=true&test=../tests/TestSuite.html&save=true&resultsUrl=results.html&close=true"
Automators beware: It's difficult to accurately kill an IEHTA run that goes awry (and sadly, they go awry more often than we'd like) because it spawns sub-processes to do the work in multiple windows, and communicates with them via DCOM. If automating this, you'd probably want to run the regular selenium-server-coreless "*iehta" tests first; run the IEHTA "Simple" tests only if the *iehta tests pass in multiple modes. Automating IEHTA "Simple" is easier in singleWindow mode than in multiWindow mode.
- User Extensions Samples: Install the -sample.js as the real one and run the test for it. (Automation: Medium)
- Permission denied: Navigate to a site you can't access. Make sure the error looks basically reasonable and that the code doesn't crash out or anything. (Perhaps this should be part of the dogfood suite.) (Automation: Medium/Hard. How do you know the error message "looks reasonable?")
