This is a  walk-through on the presentation I had at  Czech Test.

I wanted to make a parallel between testing and solving puzzles.  Reason for this is because I think there should be more room for creative thinking in testing and also the analytic and logical thinking should have higher priority.

I started with a non-testing related puzzle.

The shown equation above is incorrect, but by changing the position of only one digit in can be improved and become correct (like in the example in the bottom of the slide). Click here to see the solution

Usually when someone says the word “puzzle” we think at something like this

But there are of course other types of popular puzzles

Introducing main theme “Testing is like solving puzzles“

Then I tried to link the concepts of “testing” and “puzzles” by the definitions

Examples of “testing puzzles”:

I see testing puzzles split in two categories: micro-puzzles and macro-puzzles. Micro-puzzles and the simple situations , where you know the mission immediately (ex: test some input validation functionality). Macro-puzzles are the situations where you have to deal with an entire project and you have to take care of many components: time, coverage, risk, resources, information.

Starting with presenting the micro-puzzles

To create a better image you can try this exercise

for this application

The video demo for it:

Trying to solve that would give a better idea on the fusing situations between testing and solving puzzles.

Sometimes we ship the product after we test it, and we get a “crazy” customer that finds some unexpected state and demands explanations. In the next slide I have a screenshot of the application at http://testalways.com/1/ . You can try two puzzles here: 1) Reproduce the screenshot 2) Find the pattern

Some description about an important element in many puzzles: Patterns

This is actually an older application I posted 1 and a half years ago to show as example:

There were many answers to that puzzle back then. One way that was solved was using an excel file where all the data was put and analysed.

Other solution came with the help of using automated scripts to get the images for a large number of inputs

Going through macro-puzzles

Macro puzzles are the one where we have to deal with managing our testing process

I have used this application to represent the ideas of macro puzzles

Let’s say you have to test a web application that provides some statistical data and has an export functionality. You want to focus on the export from the file type and browser perspective.

If you would try to test each browser paired with each file, a representation in the hidden image application would look like that:

But you might not have the time to test all those combinations.  Another approach where you can say you tried each browser once and each file type once, is to pair each time one browser with one file type that has not been tested.

In this case it will look more like this

Or in some cases you find something suspicious and you insist in some areas, temporally neglecting other parts:

Other more general reviews:

Recapitulating the idea and re-making the points that testing is like solving puzzles and that we need more room for creative work and analytic thinking.

So those were the slides used. The main idea was to show testing more like a fun activity and full of creativity. In the same time it wants to give ideas to others to come up with something interesting, even if its not testing related

All companies want “test automation”. Managers that never tested think they need “test automation”. Test managers who don’t like to code want someone with “test automation”. Fresh students who were just hired know “test automation”. Because of that recruiters want “test automation” for “QA”. Testers are encouraged to know “test automation”.

But when asked to think about it or describe what is that, no one wants to answer, either because it doesn’t need any explication or its like those concepts like “life”.

When people do “test automation” the following happens:
- time is spend in creating some scripts and maintaining it and eventually is not as “faster” as was ideally though it would be
- people are bored and unmotivated because the job just becomes a sub developer position at every level, especially payment; they eventually might choose to be totally change their lives and become tourist guides
- people quickly want to change their task and want to advance so they won’t do that anymore, but tell others how useful is “test automation” to do all day
- because there is not too much time to investigate the purposes and needs of some functionality, simple checks are done under the cover up of “test automation” or mocking assertions where everything passes.
- fake data is generated for managers who don’t care anyway except their own job
- parts of application code is just copied and adjusted a little bit to work as test automation
- if bugs to customer appear often, of course “we need to hire more test automation guys”

Here is how I see those concepts:

Testing is the activity performed by the person, not by the tool, including decisions, use of skills.
If I do scripts that’s not testing. That’s scripting. Testing is when I make the decision of creating a tool that will help me.
Automation can be the process of using scripts to help with testing. Populating a database with data every week/day for me to test that is automation. A script to install the application every day on my computer, that’s automation.
Occasionally the automation might check an expected result: like when I use selenium to do some web application steps and check if some data is present and that data is as expected. I call that automated checking. Its not testing. Its just while I was doing testing I though of the need of expanding my reach by using a tool.
And because of that the term “test automation” or similar associations between testing and automation doesn’t make sense. It is used improperly most of the time and its referring at some low level development.

I didn’t want to try something risky, but I noticed at least two suspicious things. By that I mean there could be some open opportunities for hacking:

1)  ”http://www.paypal.com/en/cgi-bin/webscr?cmd=_logout” – it you go to this url while logged in, your session is terminated.

Possible exploit: you can create a repeating (hidden) script (maybe on some webpage) – so that the user will believe he/she cannot login at all.

2) I am able to send money and create accounts for emails starting with “%”

Example “%john@testxcvdf.com” (You can check it for yourself – Password is “!QAZ2wsx”)

Possible exploit: the fact the char is not validated, might be considered a weak spot for hackers, by trying different alternatives.

Even if both issues are not exactly a problem, it shows that they didn’t do some type of needed testing.

(I am going to send this to their support)

Edit: Ok, I got a simple script as example for the logout issue

Steps:

1) Open this url http://www.testalways.com/exploits/1.html (reloads an iframe with the link)

2) In another tab/window of the same browser login into the Paypal account (you can create one just to test it – you don’t have to enter any private data)

Behavior:  User is disconnected quite fast without realizing what is going on

Conclusion: The value of a bug might depend on possible exploits.

This is an example of a behavior that is known, but maybe not much thinking was done in relation to security.

Another thing is the usage of same libraries for developing web applications. These bugs are identical to issues I found for a previous client.

I want to recommend this tool http://code.google.com/p/knull-shell/downloads/detail?name=knull-shellv1-beta.php for testing security issues.

With it I was able to hack into one of my client’s web server (literally the server that hosted the web application) and was able to create folders and  delete files.

The script can be used for hacking a web application in places where you can upload images or any other files.

Example of a scenario:

1) Check if the web app has a section to upload an image (usually user profile)

2)Use an image with a specific name that you can recognize and upload it

3) Check after upload and storage where is the image now located (check if the name is intact ; check if the size is the same )

4) If the image is stored with the same name or is not altered, you can try uploading the knull-shell script and the check the url

5) if the url of the stored file ends in …/some_location/knull-shellv1-beta.php and permissions are set badly, you can try to execute that script by opening that url

6) If you are successful you should have something like this:

Username: root

Password  :toor

and you should have something like this

Now you can try to see how far you can go!