In this position paper we try to extend the discussion about the human augmentation of the mind through reasoning. Memory seems favored but both of them play a major role in the mind, we want to equilibrate it. Those two components of our cognitive system are actually quite intertwined with sometimes similar properties and goals. Both are failing, in many ways, that is why we are interested in it, a slight augmentation to those abilities would have huge impact. Therefore, we made a first attempt to propose an approach of a system capable of augmenting the reasoning of his user. First by sensing his context and detect a kind of cognitive bias for many domains through wearable devices and natural language processing. Then we propose a radical method to debiasing that could be used in different scenarios thanks to augmented reality. Finally, those propositions are a beginning of a much needed bigger work that confronts many challenges discussed at the end.
In the field of human augmentation related to the mind, most of the current research focus on the memory and few address reasoning [16, 20], despite being another high intellect function of the mind. Therefore, the aim of this position paper is to extend this domain by proposing some way to augment reasoning thanks to context awareness and augmented reality.
We will first recall that reasoning is not that far from memory. In a second part we will see how detecting some cognitive bias could be done with ubiquitous technologies, followed by a third part proposing a way to debiasing with augmented cognitive environments. Lastly, we will discuss some challenge and potential ethical issues related to this approach.
From Memory to Reasoning
In fact, extending the discussion to augment the mind by reasoning is not outlandish because memory and reasoning have many similitudes and seems to share characteristics. Some researches already points out that relation between those two central components of the human cognition [10,11]. From a psychological perspective, reasoning researchers could learn from memory researchers and vice versa. In a sense, it also means that some previous studies to enhance memory (such as ) have probably already made some positive effects on reasoning. Lastly, the reason behind our research on this subject is mostly similar with those for memory: those cognitive components are both failing and need to be fixed in order to provide a better life for the user, to make it more enjoyable. In the case of memory, we face many issues of omission or alteration, in the case of reasoning, there are also many cognitive biases.
Cognitive biases are particularly interesting due to their famous analogy with optical illusions [1, 19]. When we are facing an optical illusion, even if we are aware of it, even if we know the trick and understand exactly why it happens, most of the time it is still there, we are still experiencing it (see Figure 1). Most of the time, this is similar for cognitive biases. Considering the following Linda problem from Tversky and Kahneman :
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is more probable?
1. Linda is a bank teller. 2. Linda is a bank teller and is active in the feminist movement.
Even if it is impossible for the second solution to be more probable, most of us chose it. Even when we know why this is impossible (the probability of two events occurring together is always less or equal to the probability of either one occurring alone), it remains a small part of us that do not want to accept it because it does not represent the reality. There are many more cognitive biases, some researches have already made distinctive taxonomy . Even if we will not address all cognitive bias in this paper, it can represent a premise for a much bigger work, including more and more cognitive bias to handle. We will try to address for instance, the problem firstly presented by Levesque :
Jack is looking at Anne but Anne is looking at George. Jack is married but George is not. Is a married person looking at an unmarried person?
1. Yes 2. No 3. Cannot be determined
Most of the people would answer the third solution (Cannot be determined), but the correct one is the first solution (Yes). People tend to make the easiest inference only from the given information and are not going to infer correctly by doing the full disjunctive reasoning. For instance, we could know that a married person is looking at an unmarried person by testing whether Anne is married or not. Here, there would be a way to make a computing system helping us to resolve this kind of problem. At least, if it is still presented in a natural language. The first step would be to detect it, therefore we propose to a user to wear any kind of smart eyewear beneficing a camera, even the cheapest one  would be good enough. Then, the user has to take a picture by her/himself or the user would adopt a strategy similar to Narrative Clip (a wearable camera) taking pictures every few seconds, but in our case the pictures would have to be processed in real time. We can also suppose the user will take time thinking and/or at least reading the problem, then we could use a solution to detect a reading activity (though it will be very short) and take a picture at that time. Or we could even infer that the user is facing a problem from a cognitive state  and get a picture at that moment, with the risk that the user will be pretty confident about the answer even if s/he is wrong, therefore it would make it harder to detect it. The best would be to continuously record and analyzed the user point of view, but it is hardly practicable right now for obvious reasons related to the current technology with battery limitations, or even heat, setting aside the privacy. A first user experimentation would be achieved to determine the best solution. Then, whatever the chosen solution to get the picture with the presented problem inside, the picture will be analyzed with an optical character recognition (OCR) to pick up the text from the image . The text would feed a natural language processing technology such as Attempto  that is controlled to manipulate formal logic and would be adequate to analyze the reasoning presented in the example to make correct inference.
The visual input could be switched by an audio input, it would require a speech recognition technology  instead of OCR to get the text from audio sources, but with the same issue than the picture to choose when to record it. Both of theses domain are becoming very efficient partly due to progress in deep learning .
We would now have a system capable of detecting any kind of problem with deductive reasoning from the user perspective. Whatever the domain, even though presented in different ways, the system would detect the problem and find the solution, even in a more abstract presentation. Actually if the reasoning system is based on Attempto, it would work only with natural language in English, but this could be handled efficiently in respect to another subdomain of NLP: machine translation .
Despite recent progress to make such computing process on mobiles, all of the system could rely on a server side technology and the eyewear would be just an interface, in a similar way to some previous work .
Finally, even if we build a system to detect it, we still have the question of the output. What to do about it and how to restitute it, for instance we could propose directly the correct answer of the problem, directly through the eyewear if there is a display, or it could be notified to a smart watch or mobile phone. However, it would be perhaps more appropriate to just warn the user of potential cases, and tell them to go deeper into their reasoning exploration by providing only key concepts, similar to a tutor. By this way, for instance, certain statistical cognitive bias related to the law of big numbers would be significantly better addressed .
Both ways would augment the human reasoning, the first one would be maybe more efficient, quicker, because the answer is directly provided. The second way would be perhaps slower, but the user would maybe rely less on the technology, because s/he still has to do the exploration and find the solution with cues by her/himself. Maybe s/he could develop a habit to do it, create a sort of faculty to detect it naturally, it would make a sort of discreet training for some reasoning. Obviously it would be interesting then for the system to be aware of the user, making less notification in the future, at least for the cognitive bias that seems to disappear, the assessing would be supposed by fewer and fewer cues asked with a solution verification. The system would then focus more on other potential biases. However, people tend to minimize their cognitive load and process the minimum of information , thus taking part of the computing system to take care about all the reasoning by processing the maximal information could increase the overall reasoning of the user without increasing the cognitive workload. Therefore, in the next part, we chose to debiasing by doing so and the system outcome would be more related to persuasive technology.
Augmented Cognitive Environments
Despite all the research about reasoning, except some specific cases, it seems that we still have trouble making people to reason better . In fact, sometimes, if we try to learn how to avoid a cognitive bias, the effect is the opposite, that reinforces it. For example, if someone just learn to be a better reasoner, s/he could be overconfident and will be very sure about his thought, therefore if a similar bias had remained it will be reinforced.
The actual working solutions found to debiasing take a lot of time to be effective, such as real courses about the domain. As pointed by Guillaume Beaulac during the Cognitive Sciences Institute Summer School 16’, another radical and perhaps controversial solution for debiasing would be modifying the environment, it has already proven to be effective in different ways. For example, in the famous Marshmallow test, in which a child will be placed alone in a room with a marshmallow. It is told to the child that if s/he does not eat the candy for a certain period of time, then s/he will obtain a second one. There are few things that some clever children do to resist the temptation: they turn away, they put their hand in front of their eyes... Whatever their actions, the aim is to hide the marshmallow, to make it less visible in their environment.
Because using the right cognitive tools in the right time and place for debiasing is difficult, therefore if we can rely on the environment to do it, that would make the reasoning easier and better. This method is already widely use in many aspects, such as design . For instance, when we enter in a car, and start the car, if we do not have fastened the belt, a signal light up, it does not oblige us to do it, it is an authorizing strategy. Therefore, something that usually happens in the mind of the person is now externalizing onto the environment.
It is actually something that we can even retrieve in the animal kingdom, for instance with the squirrel who rely on specific objects to retrieve food. This strategy to adapt our needs is widely used by the humans in everyday life, when we put a clock forward of five minutes in order to avoid being late... We all use similar strategy by using something in the environment to help us think about other things, and therefore make us better reasoner. This method is also used for some famous issues. For instance, during recruiting process, people with a foreigner name have less chance to get a job. Therefore we can directly remove the name from the process, making it anonymized, at least for the first steps concerning curriculum vitae and motivation letters.
Augmented Reality is almost poised to be widely used in the future, whatever the way the user will experience it. It could be through a handheld device (such as smartphones), peripheral see-through display (such as Google Glass), with stereoscopy (such as Microsoft Hololens) or even light field (such as Avegant or the ongoing Magic Leap device). Whatever the technology used, it is adequate to modify and augment the environment. With Hololens for instance, capable of sensing the physical environment, it allows the system to easily augment cognitive environment by digital entities whose user can rely on, making her/him a better reasoner. It can also be used to hide some visual aspect of our environment, for instance concerning the previous problem about a married person looking at an unmarried person, just by hiding the third solution (cannot be determined) would oblige the user to reason properly by testing the different solution instead of directly choose the given information. It could be the same in the case of the hiring process, to remove the name from the motivation letters. However, the solution of hiding is particularly computationally expensive and would need excessive computer vision technology. Also, paradoxically, hiding things, sounds more like a “reduced reality” than “augmented reality”... In fact even the married/unmarried problem could be addressed differently by adding something to the environment. For instance, what about changing the statement by adding just one sentences, such as: “Anne could be either married or unmarried.” It would obviously make the user think about the different possibilities. We tend to think about augmented reality for augmented cognitive environment because in the later, we would make some obvious change, but in fact, again, as mentioned before, a slight change to the statement of the problem could be done with a notification to a simpler device such as a smartphone or a smartwatch.
Challenges and Discussions
Applying human mind augmentation directly to general reasoning is difficult because it is underlying any domains, that is also why we think that similar to memory, this area deserves more attention, researches and studies. The beneficial outcome of such augmentation would be important.
Therefore the main challenge is to make it practical for any everyday reasoning. For this first attempt, we try to help with a solution to recognize some biases through natural language but in real life many cognitive biases do not come from a very formal way, they are highly integrated into the flow of activities. Therefore the system would have to rely much more on ubiquitous computing, using other sensors to detect cognitive biases not from natural language. This challenge very close to intelligence amplification  seems very difficult, that is why we approach it and then try to bring specialists into the discussion.
We set aside any cognitive architectures to approach our problem, but this has to be explored, particularly because when we think of a system to detect when a human is going to need some help or whatever as a support for reasoning, it means that the system should be able to reason as a human too. This solution is quite hard, because of the obvious need of an artificial intelligence capable to reason exactly as humans do.
The system would also face some reluctance concerning privacy issues. Moreover, ethically speaking, the boundary between modifying the environment and direct manipulation is very thin or blur. This kind of manipulation does not have to be with hiding or adding obvious things, a simple suggestion or slight modification could trigger a change in the reasoning . We are also asking ourselves if modifying the environment to the users could imply some sort of forced reasoning. But forced reasoning sounds like an oxymoron, could we even talk about “reasoning” if we are forced? In extreme, it seems to annihilate reasoning at least for the human part, making us just a tool of the computing system.
Finally, to return to the analogy about optical illusion, do we really need to augment this, because, it seems that cognitive bias and heuristics had a positive effect on our evolution and survival ability , and everything has a cost, even debiasing . We must at least ensure further and deep exploration of possible side effects from such systems.
We would like to thank all the publications support and staff, who wrote and provided helpful comments on previous versions of this document. As well authors gratefully acknowledge the grant from Natural Sciences and Engineering Research Council (NSERC) of Canada and the company Black Artick.
Dan Ariely. 2008. Predictably Irrational: The Hidden Forces That Shape Our Decisions. HarperCollins New York.
Hal R Arkes. 1991. Costs and Benefits of Judgment Errors: Implications for Debiasing. Psychological Bulletin 110, 3: 486–498.
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv Preprint arXiv:1409.0473,
Damien Brun, Susan M. Ferreira, Charles Gouin- Vallerand, Sébastien George. A Mobile Platform for Controlling and Interacting with a Do-It-Yourself Smart Eyewear. 2017. International Journal of Pervasive Computing and Communications 13, 1: 41–61.
Tilman Dingler. 2016. Cognition-Aware Systems As Mobile Personal Assistants. In Proc. of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, 1035–1040. UbiComp ’16. ACM, New York, NY, USA.
Geoffrey T. Fong, David H. Krantz, Richard E. Nisbett. 1986. The Effects of Statistical Training on Thinking about Everyday Problems. Cognitive Psychology 18, 3: 253–292.
Norbert E. Fuchs, Kaarel Kaljurand, Gerold Schneider. 2006. Attempto Controlled English Meets the Challenges of Knowledge Representation, Reasoning, Interoperability and User Interfaces. In FLAIRS Conference, 12:664–669.
Ian Goodfellow, Yoshua Bengio, Aaron Courville. 2016. Deep Learning. MIT Press.
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton. Speech Recognition with Deep Recurrent Neural Networks. 2013. In Acoustics, Speech and Signal Processing (Icassp), 2013 Ieee International Conference on, 6645–6649.
Brett K. Hayes, Evan Heit, Caren M. Rotello. 2014. Memory, Reasoning, and Categorization: Parallels and Common Mechanisms. Frontiers in Psychology 5.
Evan Heit, Caren M. Rotello, Brett K. Hayes. 2012. Relations Between Memory and Reasoning. Psychology of Learning and Motivation-Advances in Research and Theory 57: 57.
Hosub Lee, Cameron Upright, Steven Eliuk, and Alfred Kobsa. 2016. Personalized Object Recognition for Augmenting Human Memory. In Proc. of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, 1054–1061. UbiComp ’16. ACM, New York, NY, USA.
Hector J. Levesque. 1988. Logic and the Complexity of Reasoning. Journal of Philosophical Logic 17, 4: 355–389.
Luyao Liu. 2016. Applying Intelligence Amplification in Decision Making. Master Thesis, University of Twente, Enschede, Netherlands.
Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books (AZ).
Sajid H. Sadi. 2012. ReflectOns : Mental Prostheses for Self-Reflection. Ph.D Dissertation. Massachusetts Institute of Technology (MIT), Cambridge, MA.
Keith E. Stanovich, Maggie E. Toplak, Richard F. West. 2008. The Development of Rational Thought: A Taxonomy of Heuristics and Biases. Advances in Child Development and Behavior 36: 251–285.
Amos Tversky, Daniel Kahneman. 1983. Extensional versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment. Psychological Review 90, 4: 293.
Lance Workman, Will Reader. 2014. Evolutionary Psychology. Cambridge University Press.
Cassandra Xia, Pattie Maes. 2013. The Design of Artifacts for Augmenting Intellect. In Proc. of the 4th Augmented Human International Conference, 154–161. AH ’13. ACM, New York, NY, USA.
Qixiang Ye, David Doermann. 2015. Text Detection and Recognition in Imagery: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 7: 1480–1500.
Damien Brun - Université TÉLUQ Montréal, QC, Canada firstname.lastname@example.org Charles Gouin-Vallerand - Université TÉLUQ Montréal, QC, Canada email@example.com Sébastien George UBL, Université du Maine, EA 4023, LIUM, 72085 Le Mans, France firstname.lastname@example.org