Documentation

Rapid multimodal app dev tool
usability testing
Jiřı́ Šádek [email protected], Jiřı́ Červák [email protected]
January 3, 2007
This project describes a creation of usability tests for application being
developed by Filip Ornstein in IBM corporation. The main purpose of application is some kind of translation between domains like texts, sounds and
pictures. Application has a ability to deploy a user standalone application
with desired purpose. It is based on VoiceXML.
1. Objectives
Test web-based interface of application for usability and report errors in function design.
Our task was aimed on authoring and deploy features.
2. User profile
User profile wasn’t specified exactly by IBM submitter but usage of application is planned
to be wide so we were searching a persons with relative small control PC abilities, understanding navigation on Internet and advanced level of english language usage.
On this basis we make up an question form to collect information from people to find a
group of test users. Questionnaire is located at appendix part A of this report (it is in
czech language because of people asked).
3. More about application
Web-based application Ramadet – Rapid multimodal app dev tool – is being developed
by Filip Ornstein ([email protected]). It should be used to design a simple quiz applications mainly for learning. Because of using VoiceXML technology it can react on
voice input, this is actualy a reason why it can be run in Opera browser only. In actual
state it act as a two domain translator (picture to voice and vice versa). Application
screeshot can be seen on figure 1
Application consist of three following parts:
1
Ramadet is the core module which provides actions for searching in categories, learning
pronunciation and quiz testing
Authoring requires to be logged in the system and it is used to maintain categories and
items.
Deploy is used for making standalone aplication (flash, website-based) which can be run
outside the system and which provides same functions as Ramadet module.
Figure 1: Ramadet application screenshot
4. Testing
For testing purpose we created a set of tasks using main features of authoring and
deploy in application. Before test procedure to not all participants got completely lost
we made small introduction to application abilities. Than testing person try to accomplished them and add difficulty rating 1 after each one. It is very important to log user
feedback about the application. After test has been accomplished a small debate was
1
Task 1 was not rated because it’s not connected to application
2
done with tested person about the application. For set of tasks look at appendix part B.
Testing task was created to reveal problems connected to application design (look and
feel), functions grouping and locality in application (“where am I in application and how
can I get to . . . ?”).
4.1. Setup
Tests has been done on a computer at Jiri Sadek’s home with following configuration:
Operational system Windows XP Professional
Browser Opera 9.10 with Czech language support
Internet connection ADSL 3096/256
Microphone standard Genius
Desktop was set to default (not disturbing) state with created “pictures to upload”
folder and Opera icon on it.
Jiřı́ Šádek tester role was a logger
Jiřı́ Červák was a test leader - speaker to tested persons
5. Results
5.1. Introduction
We tested only 4 people because of problem mention below. We aimed specially for
normal users with not much computer skills because thats supposed to be the target
group. They were between the age 23 - 26. One of them has very good computer controlling abilities, the others use computer mainly for web browsing, e-mail and office
applications. All have at least intermediate english understanding skill. One test session
took approximately one hour.
From table 1 – showing participants difficulty rating of tasks – is clear that deploy and
image upload were the most problematic tasks for participants.
5.2. Deeper analysis
Now, lets analyse all task in more detail way. For each task are mentioned problematic
passages. For this we create a problem’s weight rating where 1 stands for “lightestweight” problem and 4 means “highest-weight” problem.
3
task no.
2
3
4
5
6
7
participants
1 2 3 4
3 2 1 2
2 3 3 4
3 4 1 2
2 2 2 1
4 2 4 2
2 2 2 1
avg. rating
2
3
2.5
1.75
3
1.75
Table 1: participants difficulty rating of tasks (1 means very easy and 4 means very
hard)
5.2.1. Task 2
Task: Log to the application with username: test-user and password: testing
Problems: (weight rating is in brackets before the statement)
(2) Search dialog without description was little confusing and two participants try to
type credentials in it.
(1) Feedback after login in ends with . . . which was thought that the process is still in
the progress. See figure 2
(1) After login nobody starts exploring at the Home link where the small help about
each menu function is. See figure 2
5.2.2. Task 3 and 4
This tasks will be discuss together because problems are connected to each other.
Task: Upload given pictures and group them in one category
Problems: (weight rating is in brackets before the statement)
(4) Everybody was looking for uploading feature in Category section due to hierarchical relation of category and item. See figure 3
(4) After succesful upload all participants thought that it has been moved to created category. If they discovered it would have not, they couldn’t find place
locating the uploaded pictures again (it was just like “ok, i will upload them
again”, then they navigated to Item → new item and “oh, here the pictures are”).
(4) An application error occured in 4th test session2 propably was caused in following
steps:
2
This is the reason we test only 4 people
4
Figure 2: Statement after login
1. new pictures were uploaded
2. duplicated pictures were deleted
3. some unwished action was taken so the browser back button was used (so
deleted pictures were shown)
4. accidentally deleted picture was edited and added to category
5. now if all pictures all this category is filtered and Item section is entered and
“Application error (Rails)” occure
(3) Form for adding or editing category has realy complicated elements for messages
(search, learn and test). Nobody catch what this messages means and because
of they was in “editing” section (and got lost in task) two of them tried to add
pictures in this form elements! See figure 4
(3) Participants were repeatly clicking on show link or category name itself
(which is not a link) in Category section to list image within this category.
(3) Almost all tested users overlooked new item link in Item section when searching for
upload dialog.
(2) In add image to category dialog Command element is not always understood correctly. And also if it was left unfilled a very small “error” message is appeared
which is insignificant and always overlooked. Another problem was that the
small picture in this dialog is not a thubnail but a excision and enlarging is unabled. Last note was about unsorted list of categories (not in alphabetical order,
which was wanted).
5
Figure 3: Categories listing
Figure 4: New category complicated form element
6
Figure 5: Screen after deploy
5.2.3. Task 6
Task: Make a standalone application of created category items for offline use.
Problems: (weight rating is in brackets before the statement)
(4) Deploy section is entered as the last unused option and nobody knows what
it stands for.
(3) App for browser Opera link has been twice left unclicked because of abbreviation
app wasn’t known to participants.
(3) After succesful deploy a long messages about zip file creation is shown. It
was propably debug statements but “normal user” participants was really scared.
See figure 5
(3) Link for Download result was often left overlooked.
7
6. Conclusion
All problems described in previous section can be generalized to following conclusions:
Technical terminology is used throught hole application. It should be replaced by common language for better understanding. Also help for main features should be
included.
Graphical interface should be design to represent relation between category and item
correctly. All users understand it as hierarchical where Item is element of Category.
All application feedback is suggested to be more understandable and in noticeable design. Current response is really bad readable and often overlooked.
Also one locality problem was mentioned above. Problem is located in New item creating part. All participants thought that uploaded image was at once added to a
category. The place where uploaded pictures rather be accesible from main menu
(not only Item → New item). After uploading image user should be informed
about category assignment.
Top menu functions are badly grouped, especially section Bug which is also not well
understand. It’s necessary to separate groups of functions in better way.
8
A. Questionnaire
1. Použı́váte osobnı́ počı́tač?
• Jak dlouho? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
• Za jakým účelem?
2
2
2
2
2
2
Prohlı́ženı́ webových stránek
E-mail
Vývoj aplikacı́
Kancelářská práce
Hry
Jiný/Dalšı́ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2. Použı́váte Internet?
• Pokud ano, jaký použı́váte webový prohlı́žeč(e)?
2
2
2
2
2
2
Mozilla Firefox
Opera
Internet Explorer
Konqueror
Galeon
Jiný . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3. Mluvı́te anglicky? Na jaké úrovni?
2 Nehovořı́m
2 Začátečnické (alespoň pasivně ruzumı́m jednoduchým textům)
2 Pokročilé (jsem schopen konverzace)
2 Výborné (státnice či delšı́ pobyt v anglicky mluvı́cı́ zemi)
9
B. Test tasks
Použijeme prohlı́žeč Opera 9+ na operačnı́m systému Windows XP. Aplikace se nacházı́
na stránkách http://ramadet.ornstein.cz/.
. . . předvedenı́ možnostı́ aplikace (ukázka)
Po předvedenı́ možnostı́ aplikace je teď na vás pomoci kamarádovi v nouzi. Jedná se o
chlapce, který má potı́že se zapamatovánı́m některých anglických slovı́ček. Konkrétně
se jedná o ty z lekce Nábytek ve vašem domě. V následujı́cı́m testu zkusı́te vytvořit kvı́z,
který mu pomůže v učenı́.
Úkolem je tedy vytvořit malou galerii nábytku. Použijte fotografie na médiu dodaném
moderátorem. Postupujte podle následujı́cı́ch bodů a vždy ohodnťte jeho obtı́žnost podle
vašeho mı́něnı́.
1. přesuňte se na stránky aplikace Ramadet (http://ramadet.ornstein.cz/)
2. přihlašte se do systému pomocı́ uživatelského jména: test-user a hesla: testing
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
3. nahrajte obrázky pro výuku chlapce
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
10
4. seskupte je do jedné kategorie
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
5. ověřte výsledek předchozı́ch kroků
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
• nahrály se obrázky opravdu v pořádku?
• jsou součástı́ jedné kategorie?
6. aplikaci, kterou jste právě vytvořili lze exportovat do aplikace použitelné bez nutnosti připojenı́ k internetu. pokuste se o to a poskytněte ji chlapci v nouzi
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
11
7. vı́te kde je vytvořená aplikace a jak ji spustit?
( velmi snadné - snadné - obtı́žné - velmi obtı́žné )
C. Pictures
12