Category Archives: measurement theory

AAI THEORY V2 – MEASURING USABILITY

eJournal: uffmm.org
ISSN 2567-6458, 6.February 2019
Email: info@uffmm.org
Author: Gerd Doeben-Henisch
Email: gerd@doeben-henisch.de

CONTEXT

An overview of the enhanced AAI theory  version 2 you can find here.  In this post we talk about the tenth chapter dealing with Measuring Usability

MEASURING  USABILITY

As has been delineated in the post “Usability and Usefulness”   statements  about the quality of the usability of some assisting actor are based on some  kinds of measurement: mapping some target (here the interactions of an executive actor with some assistive actor) into some predefined norm (e.g. ‘number of errors’, ‘time needed for completion’, …).   These remarks are here embedded in a larger perspective following   Dumas and  Fox (2008).

Overview of Usability Testing following the article of Dumas & Fox (2008), with some new AAI specific terminology
Overview of Usability Testing following the article of Dumas & Fox (2008), with some new AAI specific terminology

From the three main types of usability testing with regard to the position in the life-cycle of a system we focus here primarily on the usability testing as part of the analysis phase where the developers want to get direct feedback for the concepts embedded in an actor story. Depending from this feedback the actor story and its related models can become modified and this can result in a modified exploratory mock-up  for a new test. The challenge is not to be ‘complete’ in finding ‘disturbing’ factors during an interaction but to increase the probability to detect possible disturbing factors by facing the symbolically represented concepts of the actor story with a sample of real world actors. Experiments  point to the number of 5-10 test persons which seem to be sufficient to detect the most severe disturbing factors of the concepts.

Usability testing procedure according to Lauesen (2005), adapted to the AAI paradigm
Usability testing procedure according to Lauesen (2005), adapted to the AAI paradigm

A good description of usability testing can be found in the book Lauesen (2005), especially chapters 1 +13.  According to this one can infer the following basic schema for a usability test:

  1. One needs 5 – 10 test persons whose input-output profile (AAR) comes close to the profile (TAR) required by the actor story.
  2. One needs a  mock-up of the assistive actor; this mock-up  should  correspond ‘sufficiently well’ with the input-output profile (TAR) required by the  actor story. In the simplest case one has a ‘paper model’, whose sheets can be changed on demand.
  3. One needs a facilitator who is receiving the test person, introduces the test person into the task (orally and/ or by a short document (less than a page)), then accompanies the test without interacting further with the test person until the end of the test.  The end is either reached by completing the task or by reaching the end of a predefined duration time.
  4. After the test person has finished the test   a debriefing happens by interrogating the test person about his/ her subjective feelings about the test. Because interviews are always very fuzzy and not very reliable one should keep this interrogation simple, short, and associated with concrete points. One strategy could be to ask the test person first about the general feeling: Was it ‘very good’, ‘good’, ‘OK’, ‘undefined’, ‘not OK’, ‘bad’, ‘very bad’ (+3 … 0 … -3). If the stated feeling is announced then one can ask back which kinds of circumstances caused these feelings.
  5. During the test at least two observers are observing the behavior of the test person. The observer are using as their ‘norm’ the actor story which tells what ‘should happen in the ideal case’. If a test person is deviating from the actor story this will be noted as a ‘deviation of kind X’, and this counts as an error. Because an actor story in the mathematical format represents a graph it is simple to quantify the behavior of the test person with regard to how many nodes of a solution path have been positively passed. This gives a count for the percentage of how much has been done. Thus the observer can deliver data about at least the ‘percentage of task completion’, ‘the number (and kind) of errors by deviations’, and ‘the processing time’. The advantage of having the actor story as a  norm is that all observers will use the same ‘observation categories’.
  6. From the debriefing one gets data about the ‘good/ bad’ feelings on a scale, and some hints what could have caused the reported feelings.

STANDARDS – CIF (Common Industry Format)

There are many standards around describing different aspects of usability testing. Although standards can help in practice  from the point of research standards are not only good, they can hinder creative alternative approaches. Nevertheless I myself are looking to standards to check for some possible ‘references’.  One standard I am using very often is the  “Common Industry Format (CIF)”  for usability reporting. It is  an ISO standard (ISO/IEC 25062:2006) since  2006. CIF describes a method for reporting the findings of usability tests that collect quantitative measurements of user performance. CIF does not describe how to carry out a usability test, but it does require that the test include measurements of the application’s effectiveness and efficiency as well as a measure of the users’ satisfaction. These are the three elements that define the concept of usability.

Applied to the AAI paradigm these terms are fitting well.

Effectiveness in CIF  is targeting  the accuracy and completeness with which users achieve their goal. Because the actor story in AAI his represented as a graph where the individual paths represents a way to approach a defined goal one can measure directly the accuracy by comparing the ‘observed path’ in a test and the ‘intended ideal path’ in the actor story. In the same way one can compute the completeness by comparing the observed path and the intended ideal path of the actor story.

Efficiency in CIF covers the resources expended to achieve the goals. A simple and direct measure is the measuring of the time needed.

Users’ satisfaction in CIF means ‘freedom from discomfort’ and ‘positive attitudes towards the use of the product‘. These are ‘subjective feelings’ which cannot directly be observed. Only ‘indirect’ measures are possible based on interrogations (or interactions with certain tasks) which inherently are fuzzy and not very reliable.  One possibility how to interrogate is mentioned above.

Because the term usability in CIF is defined by the before mentioned terms of effectiveness, efficiency as well as  users’ satisfaction, which in turn can be measured in many different ways the meaning of ‘usability’ is still a bit vague.

DYNAMIC ACTORS – CHANGING CONTEXTS

With regard to the AAI paradigm one has further to mention that the possibility of adaptive, learning systems embedded in dynamic, changing  environments requires for a new type of usability testing. Because learning actors change by every exercise one should run a test several times to observe how the dynamic learning rates of an actor are developing in time. In such a dynamic framework  a system would only be  ‘badly usable‘ when the learning curves of the actors can not approach a certain threshold after a defined ‘typical learning time’. And,  moreover, there could be additional effects occurring only in a long-term usage and observation, which can not be measured in a single test.

REFERENCES

  • ISO/IEC 25062:2006(E)
  • Joseph S. Dumas and Jean E. Fox. Usability testing: Current practice
    and future directions. chapter 57, pp.1129 – 1149,  in J.A. Jacko and A. Sears, editors, The Human-Computer Interaction Handbook. Fundamentals, Evolving Technologies, and Emerging Applications. 2nd edition, 2008
  • S. Lauesen. User Interface Design. A software Engineering Perspective.
    Pearson – Addison Wesley, London et al., 2005

QUANTUM THEORY (QT). Basic elements

eJournal: uffmm.org, ISSN 2567-6458, 2.January 2019
Email: info@uffmm.org
Author: Gerd Doeben-Henisch
Email:
gerd@doeben-henisch.de

CONTEXT

This is a continuation from the post WHY QT FOR AAI? explaining the motivation why to look to quantum theory (QT) in the case of the AAI paradigm. After approaching QT from a philosophy of science perspective (see the post QUANTUM THEORY (QT). BASIC PROPERTIES) giving a ‘birds view’ of the relationship between a QT and the presupposed ‘real world’ and digging a bit into the first person view inside an observer we are here interested in the formal machinery of QT. For this we follow Grifftiths in his chapter 1.

QT BASIC ELEMENTS

MEASUREMENT

  1. The starting point of a quantum theory QT are ‘phenomena‘, which “lack any description in classical physics”, a kind of things “which human beings cannot observe directly”. To measure such phenomena one needs highly sophisticated machines, which poses the problem, that the interpretation of possible ‘measurement data’ in terms of a quantum theory depends highly on the understanding of the working of the used measurement apparatus. (cf. p.8)
  2. This problem is well known in philosophy of science: (i) one wants to built a new theory T. (ii) For this theory one needs appropriate measurement data MD. (iii) The measurement as such needs a well defined procedure including different kinds of pre-defined objects and artifacts. The description of the procedure including the artifacts (which can be machines) is a theory of its own called measurement theory T*. (iv) Thus one needs a theory T* to enable a new theory T.
  3. In the case of QT one has the special case that QT itself has to be part of the measurement theory T*, i.e. QT subset T*. But, as Griffiths points out, the measurement problem in QT is even deeper; it is not only the conceptual dependency of QT from its measurement theory T*, but in the case of QT does the measurement apparatus directly interact with the target objects of QT because the measurement apparatus is itself part of the atomic and sub-atomic world which is the target. (cf. p.8) This has led to include the measurement as ‘stochastic time development’ explicitly into the QT. (cf. p.8) In his book Griffiths follows the strategy to deal with the ‘collapse of the wave function’ within the theoretical level, because it does not take place “in the experimental physicist’s laboratory”. (cf. p.9)
  4. As a consequence of these considerations Griffiths develops the fundamental principles in the chapters 2-16 without making any reference to measurement.

PRE-KNOWLEDGE

  1. Besides the special problem of measurement in quantum mechanics there is the general problem of measurement for every kind of empirical discipline which requires a perception of the real world guided by a scientific bias called ‘scientific knowledge’! Without a theoretical pre-knowledge there is no scientific observation possible. A scientific observation needs already a pre-theory T* defining the measurement procedure as well as the pre-defined standard object as well as – eventually — an ‘appropriate’ measurement device. Furthermore, to be able to talk about some measurement data as ‘data related to an object of QT’ one needs additionally a sufficient ‘pre-knowledge’ of such an object which enables the observer to decide whether the measured data are to be classified as ‘related to the object of QT. The most convenient way to enable this is to have already a proposal for a QT as the ‘knowledge guide’ how one ‘should look’ to the measured data.

QT STATES

  1. Related to the phenomena of quantum mechanics the phenomena are in QT according to Griffiths understood as ‘particles‘ whose ‘state‘ is given by a ‘complex-valued wave function ψ(x)‘, and the collection of all possible wave functions is assumed to be a ‘complex linear vector space‘ with an ‘inner product’, known as a ‘Hilbert space‘. “Two wave functions φ(x) and ψ(x) represent ‘distinct physical states’ … if and only if they are ‘orthogonal’ in the sense that their ‘inner product is zero’. Otherwise φ(x) and ψ(x) represent incompatible states of the quantum system …” .(p.2)
  2. “A quantum property … corresponds to a subspace of the quantum Hilbert space or the projector onto this subspace.” (p.2)
  3. A sample space of mutually-exclusive possibilities is a decomposition of the identity as a sum of mutually commuting projectors. One and only one of these projectors can be a correct description of a quantum system at a given time.cf. p.3)
  4. Quantum sample spaces can be mutually incompatible. (cf. p.3)
  5. “In … quantum mechanics [a physical variable] is represented by a Hermitian operator.… a real-valued function defined on a particular sample space, or decomposition of the identity … a quantum system can be said to have a value … of a physical variable represented by the operator F if and only if the quantum wave function is in an eigenstate of F … . Two physical variables whose operators do not commute correspond to incompatible sample spaces… “.(cf. p.3)
  6. “Both classical and quantum mechanics have dynamical laws which enable one to say something about the future (or past) state of a physical system if its state is known at a particular time. … the quantum … dynamical law … is the (time-dependent) Schrödinger equation. Given some wave function ψ_0 at a time t_0 , integration of this equation leads to a unique wave function ψ_t at any other time t. At two times t and t’ these uniquely defined wave functions are related by a … time development operator T(t’ , t) on the Hilbert space. Consequently we say that integrating the Schrödinger equation leads to unitary time development.” (p.3)
  7. “Quantum mechanics also allows for a stochastic or probabilistic time development … . In order to describe this in a systematic way, one needs the concept of a quantum history … a sequence of quantum events (wave functions or sub-spaces of the Hilbert space) at successive times. A collection of mutually … exclusive histories forms a sample space or family of histories, where each history is associated with a projector on a history Hilbert space. The successive events of a history are, in general, not related to one another through the Schrödinger equation. However, the Schrödinger equation, or … the time development operators T(t’ , t), can be used to assign probabilities to the different histories belonging to a particular family.” (p.3f)

HILBERT SPACE: FINITE AND INFINITE

  1. “The wave functions for even such a simple system as a quantum particle in one dimension form an infinite-dimensional Hilbert space … [but] one does not have to learn functional analysis in order to understand the basic principles of quantum theory. The majority of the illustrations used in Chs. 2–16 are toy models with a finite-dimensional Hilbert space to which the usual rules of linear algebra apply without any qualification, and for these models there are no mathematical subtleties to add to the conceptual difficulties of quantum theory … Nevertheless, they provide many useful insights into general quantum principles.”. (p.4f)

CALCULUS AND PROBABILITY

  1. Griffiths (2003) makes considerable use of toy models with a simple discretized time dependence … To obtain … unitary time development, one only needs to solve a simple difference equation, and this can be done in closed form on the back of an envelope. (cf. p.5f)
  2. Probability theory plays an important role in discussions of the time development of quantum systems. … when using toy models the simplest version of probability theory, based on a finite discrete sample space, is perfectly adequate.” (p.6)
  3. “The basic concepts of probability theory are the same in quantum mechanics as in other branches of physics; one does not need a new “quantum probability”. What distinguishes quantum from classical physics is the issue of choosing a suitable sample space with its associated event algebra. … in any single quantum sample space the ordinary rules for probabilistic reasoning are valid. ” (p.6)

QUANTUM REASONING

  1. The important difference compared to classical mechanics is the fact that “an initial quantum state does not single out a particular framework, or sample space of stochastic histories, much less determine which history in the framework will actually occur.” (p.7) There are multiple incompatible frameworks possible and to use the ordinary rules of propositional logic presupposes to apply these to a single framework. Therefore it is important to understand how to choose an appropriate framework.(cf. p.7)

NEXT

These are the basic ingredients which Griffiths mentions in chapter 1 of his book 2013. In the following these ingredients have to be understood so far, that is becomes clear how to relate the idea of a possible history of states (cf. chapters 8ff) where the future of a successor state in a sequence of timely separated states is described by some probability.

REFERENCES

  • R.B. Griffiths. Consistent Quantum Theory. Cambridge University Press, New York, 2003