HMI Analysis for the CM:MI paradigm. Part 2. Problem and Vision

Integrating Engineering and the Human Factor (
eJournal ISSN 2567-6458, February 27-March 16, 2021,
Author: Gerd Doeben-Henisch

Last change: March 16, 2021 (minor corrections)


As described in the uffmm eJournal  the wider context of this software project is an integrated  engineering theory called Distributed Actor-Actor Interaction [DAAI] further extended to the Collective Man-Machine Intelligence [CM:MI] paradigm.  This document is part of the Case Studies section.

HMI ANALYSIS, Part 2: Problem & Vision


This text is preceded by the following texts:


Before one starts the HMI analysis  some stakeholder  — in our case are the users stakeholder as well as  users in one role —  have to present some given situation — classifiable as a ‘problem’ — to depart from and a vision as the envisioned goal to be realized.

Here we give a short description of the problem for the CM:MI paradigm and the vision, what should be gained.

Problem: Mankind on the Planet Earth

In this project  the mankind  on the planet earth is  understood as the primary problem. ‘Mankind’ is seen here  as the  life form called homo sapiens. Based on the findings of biological evolution one can state that the homo sapiens has — besides many other wonderful capabilities — at least two extraordinary capabilities:

Outside to Inside

The whole body with the brain is  able to convert continuously body-external  events into internal, neural events. And  the brain inside the body receives many events inside the body as external events too. Thus in the brain we can observe a mixup of body-external (outside 1) and body-internal events (outside 2), realized as set of billions of neural processes, highly interrelated.  Most of these neural processes are unconscious, a small part is conscious. Nevertheless  these unconscious and conscious events are  neurally interrelated. This overall conversion from outside 1 and outside 2 into neural processes  can be seen as a mapping. As we know today from biology, psychology and brain sciences this mapping is not a 1-1 mapping. The brain does all the time a kind of filtering — mostly unconscious — sorting out only those events which are judged by the brain to be important. Furthermore the brain is time-slicing all its sensory inputs, storing these time-slices (called ‘memories’), whereby these time-slices again are no 1-1 copies. The storing of time-sclices is a complex (unconscious) process with many kinds of operations like structuring, associating, abstracting, evaluating, and more. From this one can deduce that the content of an individual brain and the surrounding reality of the own body as well as the world outside the own body can be highly different. All kinds of perceived and stored neural events which can be or can become conscious are  here called conscious cognitive substrates or cognitive objects.

Inside to Outside (to Inside)

Generally it is known that the homo sapiens can produce with its body events which have some impact on the world outside the body.  One kind of such events is the production of all kinds of movements, including gestures, running, grasping with hands, painting, writing as well as sounds by his voice. What is of special interest here are forms of communications between different humans, and even more specially those communications enabled by the spoken sounds of a language as well as the written signs of a language. Spoken sounds as well as written signs are here called expressions associated with a known language. Expressions as such have no meaning (A non-speaker of a language L can hear or see expressions of the language L but he/she/x  never will understand anything). But as everyday experience shows nearly every child  starts very soon to learn which kinds of expressions belong to a language and with what kinds of shared experiences they can be associated. This learning is related to many complex neural processes which map expressions internally onto — conscious and unconscious — cognitive objects (including expressions!). This mapping builds up an internal  meaning function from expressions into cognitive objects and vice versa. Because expressions have a dual face (being internal neural structures as well as being body-outside events by conversions from the inside to body-outside) it is possible that a homo sapiens  can transmit its internal encoding of cognitive objects into expressions from his  inside to the outside and thereby another homo sapiens can perceive the produced outside expression and  can map this outside expression into an intern expression. As far as the meaning function of of the receiving homo sapiens  is sufficiently similar to the meaning function of  the sending homo sapiens there exists some probability that the receiving homo sapiens can activate from its memory cognitive objects which have some similarity with those of  the sending  homo sapiens.

Although we know today of different kinds of animals having some form of language, there is no species known which is with regard to language comparable to  the homo sapiens. This explains to a large extend why the homo sapiens population was able to cooperate in a way, which not only can include many persons but also can stretch through long periods of time and  can include highly complex cognitive objects and associated behavior.

Negative Complexity

In 2006 I introduced the term negative complexity in my writings to describe the fact that in the world surrounding an individual person there is an amount of language-encoded meaning available which is beyond the capacity of an  individual brain to be processed. Thus whatever kind of experience or knowledge is accumulated in libraries and data bases, if the negative complexity is higher and higher than this knowledge can no longer help individual persons, whole groups, whole populations in a constructive usage of all this. What happens is that the intended well structured ‘sound’ of knowledge is turned into a noisy environment which crashes all kinds of intended structures into nothing or badly deformed somethings.

Entangled Humans

From Quantum Mechanics we know the idea of entangled states. But we must not dig into quantum mechanics to find other phenomena which manifest entangled states. Look around in your everyday world. There exist many occasions where a human person is acting in a situation, but the bodily separateness is a fake. While sitting before a laptop in a room the person is communicating within an online session with other persons. And depending from the  social role and the  membership in some social institution and being part of some project this person will talk, perceive, feel, decide etc. with regard to the known rules of these social environments which are  represented as cognitive objects in its brain. Thus by knowledge, by cognition, the individual person is in its situation completely entangled with other persons which know from these roles and rules  and following thereby  in their behavior these rules too. Sitting with the body in a certain physical location somewhere on the planet does not matter in this moment. The primary reality is this cognitive space in the brains of the participating persons.

If you continue looking around in your everyday world you will probably detect that the everyday world is full of different kinds of  cognitively induced entangled states of persons. These internalized structures are functioning like protocols, like scripts, like rules in a game, telling everybody what is expected from him/her/x, and to that extend, that people adhere to such internalized protocols, the daily life has some structure, has some stability, enables planning of behavior where cooperation between different persons  is necessary. In a cognitively enabled entangled state the individual person becomes a member of something greater, becoming a super person. Entangled persons can do things which usually are not possible as long you are working as a pure individual person.[1]

Entangled Humans and Negative Complexity

Although entangled human persons can principally enable more complex events, structures,  processes, engineering, cultural work than single persons, human entanglement is still limited by the brain capacities as well as by the limits of normal communication. Increasing the amount of meaning relevant artifacts or increasing the velocity of communication events makes things even more worse. There are objective limits for human processing, which can run into negative complexity.

Future is not Waiting

The term ‘future‘ is cognitively empty: there exists nowhere an object which can  be called ‘future’. What we have is some local actual presence (the Now), which the body is turning into internal representations of some kind (becoming the Past), but something like a future does not exist, nowhere. Our knowledge about the future is radically zero.

Nevertheless, because our bodies are part of a physical world (planet, solar system, …) and our entangled scientific work has identified some regularities of this physical world which can be bused for some predictions what could happen with some probability as assumed states where our clocks are showing a different time stamp. But because there are many processes running in parallel, composed of billions of parameters which can be tuned in many directions, a really good forecast is not simple and depends from so many presuppositions.

Since the appearance of homo sapiens some hundred thousands years ago in Africa the homo sapiens became a game changer which makes all computations nearly impossible. Not in the beginning of the appearance of the homo sapiens, but in the course of time homo sapiens enlarged its number, improved its skills in more and more areas, and meanwhile we know, that homo sapiens indeed has started to crash more and more  the conditions of its own life. And principally thinking points out, that homo sapiens could even crash more than only planet earth. Every exemplar of a homo sapiens has a built-in freedom which allows every time to decide to behave in a different way (although in everyday life we are mostly following some protocols). And this built-in freedom is guided by actual knowledge, by emotions, and by available resources. The same child can become a great musician, a great mathematician, a philosopher, a great political leader, an engineer, … but giving the child no resources, depriving it from important social contexts,  giving it the wrong knowledge, it can not manifest its freedom in full richness. As human population we need the best out of all children.

Because  the processing of the planet, the solar system etc.  is going on, we are in need of good forecasts of possible futures, beyond our classical concepts of sharing knowledge. This is where our vision enters.


To find possible and reliable shapes of possible futures we have to exploit all experiences, all knowledge, all ideas, all kinds of creativity by using maximal diversity. Because present knowledge can be false — as history tells us –, we should not rule out all those ideas, which seem to be too crazy at a first glance. Real innovations are always different to what we are used to at that time. Thus the following text is a first rough outline of the vision:

  1. Find a format
  2. which allows any kinds of people
  3. for any kind of given problem
  4. with at least one vision of a possible improvement
  5. together
  6. to search and to find a path leading from the given problem (Now) to the envisioned improved state (future).
  7. For all needed communication any kind of  everyday language should be enough.
  8. As needed this everyday language should be extendable with special expressions.
  9. These considerations about possible paths into the wanted envisioned future state should continuously be supported  by appropriate automatic simulations of such a path.
  10. These simulations should include automatic evaluations based on the given envisioned state.
  11. As far as possible adaptive algorithms should be available to support the search, finding and identification of the best cases (referenced by the visions)  within human planning.


[1] One of the most common entangled state in daily life is the usage of normal language! A normal language L works only because the rules of usage of this language L are shared by all speaker-hearer of this language, and these rules are explicit cognitive structures (not necessarily conscious, mostly unconscious!).


Yes, it will happen 🙂 Here.








eJournal:, ISSN 2567-6458
Author: Gerd Doeben-Henisch

ATTENTION: The actual Version  you will find HERE.

Draft version 22.June 2018

Update 26.June 2018 (Chapter AS-AM Summary)

Update 4.July 2018 (Chapter 4 Actor Model; improving the terminology of environments with actors, actors as input-output systems, basic and real interface, a first typology of input-output systems…)

Update 17.July 2018 (Preface, Introduction new)

Update 19.July 2018 (Introduction final paragraph!, new chapters!)

Update 20.July 2018 (Disentanglement of chapter ‘Simulation & Verification’ into two independent chapters; corrections in the chapter ‘Introduction’; corrections in chapter ‘AAI Analysis’; extracting ‘Simulation’ from chapter ‘Actor Story’ to new chapter ‘Simulation’; New chapter ‘Simulation’; Rewriting of chapter ‘Looking Forward’)

Update 22.July 2018 (Rewriting the beginning of the chapter ‘Actor Story (AS)’, not completed; converting chapter ‘AS+AM Summary’ to ‘AS and AM Philosophy’, not completed)

Update 23.July 2018 (Attaching a new chapter with a Case Study illustrating an actor story (AS). This case study is still unfinished. It is a case study of  a real project!)

Update 7.August 2018 (Modifying chapter Actor Story, the introduction)

Update 8.August 2018 (Modifying chapter  AS as Text, Comic, Graph; especially section about the textual mode and the pictorial mode; first sketch for a mapping from the textual mode into the pictorial mode)

Update 9.August 2018 (Modification of the section ‘Mathematical Actor Story (MAS) in chapter 4).

Update 11.August 2018 (Improving chapter 3 ‘Actor Story; nearly complete rewriting of chapter 4 ‘AS as text, comic, graph’.)

Update 12.August 2018 (Minor corrections in the chapters 3+4)

Update 13.August 2018 (I am still catched by the chapters 3+4. In chapter  the cognitive structure of the actors has been further enhanced; in chapter 4 a complete example of a mathematical actor story could now been attached.)

Update 14.August 2018 (minor corrections to chapter 4 + 5; change-statements define for each state individual combinatorial spaces (a little bit like a quantum state); whether and how these spaces will be concretized/ realized depends completely from the participating actors)

Update 15.August 2018 (Canceled the appendix with the case study stub and replaced it with an overview for  a supporting software tool which is needed for the real usage of this theory. At the moment it is open who will write the software.)

Update 2.October 2018 (Configuring the whole book now with 3 parts: I. Theory, II. Application, III. Software. Gerd has his focus on part I, Zeynep will focus on part II and ‘somebody’ will focus on part III (in the worst case we will — nevertheless — have a minimal version :-)). For a first quick overview about everything read the ‘Preface’ and the ‘Introduction’.

Update 4.November 2018 (Rewriting the Introduction (and some minor corrections in the Preface). The idea of the rewriting was to address all the topics which will be discussed in the book and pointing out to the logical connections between them. This induces some wrong links in the following chapters, which are not yet updated. Some chapters are yet completely missing. But to improve the clearness of the focus and the logical inter-dependencies helps to elaborate the missing texts a lot. Another change is the wording of the title. Until now it is difficult to find a title which is exactly matching the content. The new proposal shows the focus ‘AAI’ but lists the keywords of the main topics within AAA analysis because these topics are usually not necessarily associated with AAI.)

ACTOR-ACTOR INTERACTION [AAI]. An Actor Centered Approach to Problem Solving. Combining Engineering and Philosophy




BACKGROUND INFORMATION 19.Dec.2018: Application domain ‘Communal Planning and e-Gaming’

BACKGROUND INFORMATION 24.Dec.2018: The AAI-paradigm and Quantum Logic

PRE-VIEW: NEW EXPANDED AAI THEORY 23.January 2019: Outline of the new expanded  AAI Paradigm. Before re-writing the main text with these ideas the new advanced AAI theory will first be tested during the summer 2019 within a lecture with student teams as well as in  several workshops outside the Frankfurt University of Applied Sciences with members of different institutions.

AAI – Actor-Actor Interaction. A Philosophy of Science View

AAI – Actor-Actor Interaction.
A Philosophy of Science View
eJournal:, ISSN 2567-6458

Gerd Doeben-Henisch



On the cover page of this blog you find a first general view on the subject matter of an integrated engineering approach for the future. Here we give a short description of the main idea of the analysis phase of systems engineering how this will be realized within the actor-actor interaction paradigm as described in this text.


Overview of the analysis phase of systems engineering as realized within an actor-actor interaction paradigm
Overview of the analysis phase of systems engineering as realized within an actor-actor interaction paradigm

As you can see in figure Nr.1 there are the following main topics within the Actor-Actor Interaction (AAI) paradigm as used in this text (Comment: The more traditional formula is known as Human-Machine Interaction (HMI)):

Triggered by a problem document D_p from the problem phase (P) of the engineering process the AAI-experts have to analyze, what are the potential requirements following from this document, all the time also communicating with the stakeholder to keep in touch with the hidden intentions of the stakeholder.

The idea is to identify at least one task (T) with at least one goal state (G) which shall be arrived after running a task.

A task is assumed to represent a sequence of states (at least a start state and a goal state) which can have more than one option in every state, not excluding repetitions.

Every task presupposes some context (C) which gives the environment for the task.

The number of tasks and their length is in principle not limited, but their can be certain constraints (CS) given which have to be fulfilled required by the stakeholder or by some other important rules/ laws. Such constraints will probably limit the number of tasks as well as their length.

Actor Story

Every task as a sequence of states can be viewed as a story which describes a process. A story is a text (TXT) which is static and hides the implicit meaning in the brains of the participating actors. Only if an actor has some (learned) understanding of the used language then the actor is able to translate the perceptions of the process in an appropriate text and vice versa the text into corresponding perceptions or equivalently ‘thoughts’ representing the perceptions.

In this text it is assumed that a story is describing only the observable behavior of the participating actors, not their possible internal states (IS). For to describe the internal states (IS) it is further assumed that one describes the internal states in a new text called actor model (AM). The usual story is called an actor story (AS). Thus the actor story (AS) is the environment for the actor models (AM).

In this text three main modes of actor stories are distinguished:

  1. An actor story written in some everyday language L_0 called AS_L0 .
  2. A translation of the everyday language L_0 into a mathematical language L_math which can represent graphs, called AS_Lmath.
  3. A translation of the hidden meaning which resides in the brains of the AAI-experts into a pictorial language L_pict (like a comic strip), called AS_Lpict.

To make the relationship between the graph-version AS_Lmath and the pictorial version AS_Lpict visible one needs an explicit mapping Int from one version into the other one, like: Int : AS_Lmath <—> AS_Lpict. This mapping Int works like a lexicon from one language into another one.

From a philosophy of science point of view one has to consider that the different kinds of actor stories have a meaning which is rooted in the intended processes assumed to be necessary for the realization of the different tasks. The processes as such are dynamic, but the stories as such are static. Thus a stakeholder (SH) or an AAI-expert who wants to get some understanding of the intended processes has to rely on his internal brain simulations associated with the meaning of these stories. Because every actor has its own internal simulation which can not be perceived from the other actors there is some probability that the simulations of the different actors can be different. This can cause misunderstandings, errors, and frustrations.(Comment: This problem has been discussed in [DHW07])

One remedy to minimize such errors is the construction of automata (AT) derived from the math mode AS_Lmath of the actor stories. Because the math mode represents a graph one can derive Der from this version directly (and automatically) the description of an automaton which can completely simulate the actor story, thus one can assume Der(AS_Lmath) = AT_AS_Lmath.

But, from the point of view of Philosophy of science this derived automaton AT_AS_Lmath is still only a static text. This text describes the potential behavior of an automaton AT. Taking a real computer (COMP) one can feed this real computer with the description of the automaton AT AT_AS_Lmath and make the real computer behave like the described automaton. If we did this then we have a real simulation (SIM) of the theoretical behavior of the theoretical automaton AT realized by the real computer COMP. Thus we have SIM = COMP(AT_AS_Lmath). (Comment: These ideas have been discussed in [EDH11].)

Such a real simulation is dynamic and visible for everybody. All participating actors can see the same simulation and if there is some deviation from the intention of the stakeholder then this can become perceivable for everybody immediately.

Actor Model

As mentioned above the actor story (AS) describes only the observable behavior of some actor, but not possible internal states (IS) which could be responsible for the observable behavior.

If necessary it is possible to define for every actor an individual actor model; indeed one can define more than one model to explore the possibilities of different internal structures to enable a certain behavior.

The general pattern of actor models follows in this text the concept of input-output systems (IOSYS), which are in principle able to learn. What the term ‘learning’ designates concretely will be explained in later sections. The same holds of the term ‘intelligent’ and ‘intelligence’.

The basic assumptions about input-output systems used here reads a follows:

Def: Input-Output System (IOSYS)

IOSYS(x) iff x=< I, O, IS, phi>
phi : I x IS —> IS x O
I := Input
O := Output
IS := Internal

As in the case of the actor story (AS) the primary descriptions of actor models (AM) are static texts. To make the hidden meanings of these descriptions ‘explicit’, ‘visible’ one has again to convert the static texts into descriptions of automata, which can be feed into real computers which in turn then simulate the behavior of these theoretical automata as a real process.

Combining the real simulation of an actor story with the real simulations of all the participating actors described in the actor models can show a dynamic, impressive process which is full visible to all collaborating stakeholders and AAI-experts.


Having all actor stories and actor models at hand, ideally implemented as real simulations, one has to test the interaction of the elaborated actors with real actors, which are intended to work within these explorative stories and models. This is done by actor tests (former: usability tests) where (i) real actors are confronted with real tasks and have to perform in the intended way; (ii) real actors are interviewed with questionnaires about their subjective feelings during their task completion.

Every such test will yield some new insights how to change the settings a bit to gain eventually some improvements. Repeating these cycles of designing, testing, and modifying can generate a finite set of test-results T where possibly one subset is the ‘best’ compared to all the others. This can give some security that this design is probably the ‘relative best design’ with regards to T.

Further Readings:

  1. Analysis
  2. Simulation
  3. Testing
  4. User Modeling
  5. User Modeling and AI

For a newer version of the AAi-text see HERE..


[DHW07] G. Doeben-Henisch and M. Wagner. Validation within safety critical systems engineering from a computation semiotics point of view.
Proceedings of the IEEE Africon2007 Conference, pages Pages: 1 – 7, 2007.
[EDH11] Louwrence Erasmus and Gerd Doeben-Henisch. A theory of the
system engineering process. In ISEM 2011 International Conference. IEEE, 2011.


For a toy-example to these concepts please see the post AAI – Actor-Actor Interaction. A Toy-Example, No.1