Posted January 23, 2015
DyNaer: there was a problem if i remember well with the witcher ;) , and yeah one day (like dragonsphere in the past :D)
if a notification doesn't go away after (i'm generous there) 1 hour, there's a problem....
HypersomniacLive: The "almost a day" was not in reference to the TW1 update (and surprisingly I didn't suffer from the Dragonshere one), but I can't remember which game it was. if a notification doesn't go away after (i'm generous there) 1 hour, there's a problem....
ssokolow: They should really grab a test harness and start writing regression tests for everything they fix. It's not as comprehensive as a proper, directed effort to write unit and functional tests, but "teach the computer to do it whenever you'd normally test something manually" does still trend toward full test coverage in the long term.
HypersomniacLive: *in Judas voice* Please contact support.
EDIT: I've submitted the following as a suggestion.
Given the prevalence of site bugs lately, it'd probably be a good idea for you guys to have some kind of improved testing strategy.
Yes, retrofitting a full test suite would be prohibitively time-consuming, but building a functional test suite as you go can be very easy, takes minimal additional effort, and would start paying dividends almost immediately.
Basically, whenever you want to test something, teach the computer to do it instead. This is most common with regression test suites (whenever you fix a bug, you write a test so something like it can't creep back in) but the principle applies everywhere.
The first thing you'll need is a test harness that makes writing tests roughly as quick and easy as testing by hand. I'm not sure what GOG is implemented in, but there are plenty of automated test harnesses for all sorts of testing.
The support tooling comes in two basic types:
First, Unit/Integration testing:
Unit testing is testing the smallest possible pieces of your code so you can be thorough without a combinatorial explosion. (Unit tests are the fastest-running but hardest to retrofit to a codebase not already factored into neat little units) Integration testing is testing two things that are unit tested to make sure they work together properly. (eg. Catching situations like "X expects UTF-8 but Y produces a decoded stream of Unicode characters")
You probably wouldn't want to retrofit many (if any) of those because refactoring a codebase not designed with unit testing in mind is a time-consuming endeavour.
Second, functional testing: This is what you'll want to focus on.
Functional testing basically involves either using a mock browser or automating a real browser to walk through a scripted version of what you'd normally do manually. (eg. click Y , verify that thing X is true, log a screenshot, etc.)
Functional testing can be accomplished at four different levels of trade-off:
1. Libraries like FakeWeb (Ruby) and HTTPretty (Python) replace the underlying TCP socket calls with a shim that pretends to be an HTTP server so you can fake HTTP request-response conversations with lightning speed. They're probably too low-level for your uses, but be aware of them as a tool for performance optimization.
https://github.com/chrisk/fakeweb
http://falcao.it/HTTPretty/
2. Tools like Mechanize (mock browser), PhantomJS (WebKit), SlimerJS (Firefox/Gecko), and trifleJS (IE/Trident) provide a headless browser that can be scripted (via Javascript in most cases) to perform actions, check their results, take screenshots, etc. (Note: Mechanize can't take screenshots)
http://search.cpan.org/dist/WWW-Mechanize/
http://wwwsearch.sourceforge.net/mechanize/
http://docs.seattlerb.org/mechanize/
https://github.com/GistLabs/mechanize
https://github.com/srveit/mechanize-js
http://phantomjs.org/
http://slimerjs.org/
http://triflejs.org/
For most cases, PhantomJS and friends are what you want. Instead of manually testing something, write a little Javascript which tests it for you. That way, after making a change, you can let the computer re-run every test you ever thought to try.
(You can even use them to verify that a CSS change had only the intended effect by having them take before/after screenshots and then performing a pixel-wise difference operation on them... or combine that with something like `git bisect` to identify when a problem was introduced.)
3. Finally, there's Selenium. Selenium is the heaviest, but it gives you either record/replay-based control over Firefox or scripted control over just about every browser under the sun (Firefox, Internet Explorer, Safari, Opera, Chrome, Android, iPhone, etc.) so you can do truly comprehensive, high-level functional testing.
http://docs.seleniumhq.org/ (Core and drivers for Firefox, IE, Safari, Chrome, and Presto-based Opera Desktop/Mobile versions)
https://github.com/operasoftware/operachromiumdriver (Driver for Opera 15.x+)
http://appium.io/ (Drivers for Android and iOS simulator)
https://code.google.com/p/selenium/wiki/IPhoneDriver (Driver for real iPhones)
Whatever approach you use, I highly recommend you give automated testing a try. (If you want some example code for doing before/after image diffing, feel free to ask. Just let me know what languages you'd prefer.)
Yes, retrofitting a full test suite would be prohibitively time-consuming, but building a functional test suite as you go can be very easy, takes minimal additional effort, and would start paying dividends almost immediately.
Basically, whenever you want to test something, teach the computer to do it instead. This is most common with regression test suites (whenever you fix a bug, you write a test so something like it can't creep back in) but the principle applies everywhere.
The first thing you'll need is a test harness that makes writing tests roughly as quick and easy as testing by hand. I'm not sure what GOG is implemented in, but there are plenty of automated test harnesses for all sorts of testing.
The support tooling comes in two basic types:
First, Unit/Integration testing:
Unit testing is testing the smallest possible pieces of your code so you can be thorough without a combinatorial explosion. (Unit tests are the fastest-running but hardest to retrofit to a codebase not already factored into neat little units) Integration testing is testing two things that are unit tested to make sure they work together properly. (eg. Catching situations like "X expects UTF-8 but Y produces a decoded stream of Unicode characters")
You probably wouldn't want to retrofit many (if any) of those because refactoring a codebase not designed with unit testing in mind is a time-consuming endeavour.
Second, functional testing: This is what you'll want to focus on.
Functional testing basically involves either using a mock browser or automating a real browser to walk through a scripted version of what you'd normally do manually. (eg. click Y , verify that thing X is true, log a screenshot, etc.)
Functional testing can be accomplished at four different levels of trade-off:
1. Libraries like FakeWeb (Ruby) and HTTPretty (Python) replace the underlying TCP socket calls with a shim that pretends to be an HTTP server so you can fake HTTP request-response conversations with lightning speed. They're probably too low-level for your uses, but be aware of them as a tool for performance optimization.
https://github.com/chrisk/fakeweb
http://falcao.it/HTTPretty/
2. Tools like Mechanize (mock browser), PhantomJS (WebKit), SlimerJS (Firefox/Gecko), and trifleJS (IE/Trident) provide a headless browser that can be scripted (via Javascript in most cases) to perform actions, check their results, take screenshots, etc. (Note: Mechanize can't take screenshots)
http://search.cpan.org/dist/WWW-Mechanize/
http://wwwsearch.sourceforge.net/mechanize/
http://docs.seattlerb.org/mechanize/
https://github.com/GistLabs/mechanize
https://github.com/srveit/mechanize-js
http://phantomjs.org/
http://slimerjs.org/
http://triflejs.org/
For most cases, PhantomJS and friends are what you want. Instead of manually testing something, write a little Javascript which tests it for you. That way, after making a change, you can let the computer re-run every test you ever thought to try.
(You can even use them to verify that a CSS change had only the intended effect by having them take before/after screenshots and then performing a pixel-wise difference operation on them... or combine that with something like `git bisect` to identify when a problem was introduced.)
3. Finally, there's Selenium. Selenium is the heaviest, but it gives you either record/replay-based control over Firefox or scripted control over just about every browser under the sun (Firefox, Internet Explorer, Safari, Opera, Chrome, Android, iPhone, etc.) so you can do truly comprehensive, high-level functional testing.
http://docs.seleniumhq.org/ (Core and drivers for Firefox, IE, Safari, Chrome, and Presto-based Opera Desktop/Mobile versions)
https://github.com/operasoftware/operachromiumdriver (Driver for Opera 15.x+)
http://appium.io/ (Drivers for Android and iOS simulator)
https://code.google.com/p/selenium/wiki/IPhoneDriver (Driver for real iPhones)
Whatever approach you use, I highly recommend you give automated testing a try. (If you want some example code for doing before/after image diffing, feel free to ask. Just let me know what languages you'd prefer.)
Post edited January 23, 2015 by ssokolow