All sorts of automated tests happen on projects at ThoughtWorks. There are Unit Tests, Functional tests, GUI tests, and others that make sure the software we’re building runs well from the most back-end function all the way to the user’s interface.
One thing we don’t test, however, is where particular elements land on a screen. We make sure that the drop-down box you were expecting to be on a given page is there, and that it holds the values that should be there, but we don’t make sure it lines up right next to the label that explains what it is for.
I’ve brought this up a few times while on projects, and developers have told me that we don’t test for screen position because we just can’t. I wonder how true this is.
When it comes to visual design, the placement of elements on an interface does matter. When doing a visual assessment, it’s important to me that all the pixels on the left-hand side of the screen are aligned to a grid line. It’s important that that drop-down is placed right next to its label. It’s vital that the company’s logo is always on the top left. And even on a low resolution monitor, this input box needs to align with the others. On and on…there are plenty examples.
I understand that testing for screen position is a hard problem, and is impossible with our current automated testing tools. But is it really impossible altogether?
Hey all you developers out there, weigh in with your opinions. And for anybody else out there…how important would a tool that does this be to you?
9 responses to “Screen-placement testing tool”
You’re talking about two distinct well-known testing problems here.
First is the distinction between function and aesthetics. Computers are very good about deciding that something exists. Computers are very bad at deciding whether something is beautiful. Even if you get your placement tool, what if the whole display turns chartreuse? How would you know?
If you decide that you’re going to do this testing anyway, that leads you inevitably into comparing bitmaps. This is an unpleasant process that usually produces many false failures. Most of the commercial GUI-testing tools will in fact do bitmap comparisons for you if you insist on it, so you could definitely explore that avenue and see a) how far you can go and b) if there is any way to improve the state of the art.
Josh,
Testing for screen position is definitely possible. In fact, when I’m developing Java Swing UIs, I use a base test case that among other things, allows me to assert whether a specified list of components are vertically or horizontally aligned. There’s also an assertion in there that checks if a specified list of controls are all of the same size.
It should be possible to do this for web UIs at well via javascript, using a combination of a component’s absolute position, height and width.
Hmm, sounds like an interesting idea for an Open Source project 🙂
It’s not really about being *impossible* – it has more to do with being *costly*. If positioning is extremely important, you can do some checks, but it is very hard to automate. Chris hit upon the reason why this is.
One practice I’ve seen with generated output is to take a capture and compare it to something that was previously generated – doing a checksum or size check or something. This is obviously very brittle but done right at small levels, can be done.
It would still be interesting to collect other approaches and write something up about it.
I agree with Chris and Pat. I’m not sure the value is there. If you’re working in an agile, high feedback environment, I don’t think that these types of small problems will go unnoticed. I believe that most organizations, if given the choice of testing for screen positioning or more features would choose the latter.
Thanks for the comments so far, folks. Some comments of my own:
Chris – I’m definitely not advocating that the computer should be testing for aesthetics. I understand that it’s not something a computer is good at. I’m saying that the designer/analyst should be able to string together simple commands that ensure the system follows layout & style rules. Your bitmap solution could work for this…and I’ll look into that further.
Vivek – Sweet! I’ll look into that stuff too!
Pat – Thanks for that. One thing to understand is that from a visual design point of view, positioning IS important, even if elements are positioned in a non-static format. I think positioning is one thing that people are apt to discredit, but you have no idea how many clients are so happy with my work after a single day simply because I took the time to line up form fields. Users appreciate this as well.
Brandon – As noted above, these small problems are, in fact, important. Oh, and if you didn’t notice, I generally care more about users than organizations. 🙂
Josh
Its not impossible as everyone above said it. In fact a lot of testing tools just do that(Winrunner) trace/learn your mouse/keyboard movements on the controls and just repeat that after you. So its doable. But you need to balance it with value. Also since developers do not see any day to day need they are not bothered about it. It has to be someone like you to push that.. and I agree it is required. If you think we should do it.. lets make it happen.
But I am sure the tool that we develop will not produce that ‘aha’ emotion on the users face it will be a dumb robot that compares your mockup(hi-fi) to the application.
Sudhindra
As the others said, certainly not impossible. In my admittedly inexperienced estimation, it’s far from impossible. But I wonder if it would be worth the time to test layout and style.
Like Selenium tests, writing the layout and style tests could be monotonous. Like unit tests, maintaining them could be time consuming. Spending this time on unit tests is usually worthwhile because they detect problems that would otherwise go unnoticed. Layout and style issues are much easier to notice.
Check out Zephyr at http://www.degarrah.com/products/zephyr/download.php
for the documentation and http://sourceforge.net/projects/zephyrtest for the source.
I used it briefly so can’t comment on its stability or long term use viability. What I can say is that this tool allowed us to test our installer. Its a Bitmap based image comparison tool that allowed you to add test actions (find&click being the most popular one used). It allows you to say find and click then lets you select an area anywhere on your screen. (it will use the original position you picked as its default first try.) Obviously this kind of test is sensitive to color changes and so on but with proper box size selection it could be helpful for those more mature products out there.
Ran across your blog while looking for screen real estate information. A good discussion here.
Perhaps the blog here is designed for non IE 6.0? The blog text is unreadable unless its highlighted — it appears as a faint barely detectible blue (at least to these aging eyes!).
That brings up another area of importance that doesn’t get tested frequently which is accessibility for the large percentages of colorblind users where color is used to provide meaning. From what I’ve read, images are probably the most problematic given there aren’t readily available ways to know what colors are within the image/graphic and, even when known they don’t catch all possible variations that could cause someone who is colorblind problems (think various color shades and combinations as well as the fact that there are several types of colorblindness — quite a large matrix to test!).
More on topic though, it seems that an avenue to test for placement though helpful has been left to the backburner due to other competing priorities where the cost-benefit is clearer. Not that these details aren’t important, but rather they fall lower on the ‘to do’ list (and often remain left off it as other items bump them back down).