Investigation II: Machine Assisted Beliefs

tldr: Prototype a tinyML-enabled electronic object that uses an otherworldly practice for fortune telling, soothsaying, or augury. Examine how these rituals — for example, reading the tea leaves, palm reading, horoscopes, numerology, or dowsing — might offer ways to move from prediction to open-ended interpretation of data, machine learning, and device intelligences. Reflect on the value of ambiguity and these otherworldly metaphors in unsettling expectations of or beliefs in predictive machine outcomes.

“Predicting the future is no longer about the mystical reading of natural and celestial phenomena. Today it is all about data.” - James Auger and Jimmy Loizeau, 2015, Real Prediction Machines

Our tools for data science have in many ways enhanced our ability as a species to glean insight into the future. At what point does a belief in the objectivity of big data and the patterns that it reveals become a leap of faith? David Benqué, 2018. almanac computer

About this Investigation

Investigations are a series of small exercises designed to explore a conceptual space and culminates with a made artefact. The format is short rapid explorations of a theme, idea or theory. Because of being remote for the first two weeks, our first investigation will be a really rapid one week exercise.

In the last module, we explored an animistic frame on devices and explored how we might create unexpected, unsettling or haunting behaviors in everyday devices by preparing speculative technologies. In this module, we’ll shift slightly to expand our scope to consider experiences (and anxieties) in everyday encounters with critical technologies. We’ll continue this ask questions about ubiquitous computing, by creating new and alternative technologies through supernatural, haunted, and otherworldly frames.

In this investigation, we’re going to explore Tiny Machine Learning (tinyML) as a technical approach and design material for everyday intelligences. ML, AI and other processes have become more and more entwined with everyday interactions and domestic rituals. It often comes with the myth or belief in what those processes can do; that sensors, data and machine learning have the power to reliably predict outcomes and describe everyday situations. Using speculative approaches, we’ll question some of these assumptions about ML, data and intelligences. We’ll examine how ambiguity can be a resource for open-ended interpretation of ML and data driven proceses. Specifically, we’ll examine fortune telling, divination, and augury as metaphorical lens on machine prediction and interpretation. We’ll use these otherworldly practices to inform and materialize alternative electronic intelligences.

“[Ambiguity] has the added advantage … of enabling designers to go beyond the limits of their technologies. From this point of view, ambiguity provides a context that allows the use of inaccurate sensors, inexact mappings, and low-resolution displays because it encourages users to supplement them with their own interpretations and beliefs” …. “By impelling people to interpret situations for themselves, it encourages them to start grappling conceptually with systems and their contexts, and thus to establish deeper and more personal relations with the meanings offered by those systems.”
- Gaver, W. W., Beaver, J., & Benford, S. (2003, April). Ambiguity as a resource for design.

Learning goals

This investigation is designed to deepen your skills with technical prototyping and conceptual exploration. The module is designed to give you more latitude to explore making and designing technological artefacts. To do this, it will build on the skills introduced in module one, and add a few new approaches to materializing strange and alternative ubiquitous, physical and tangible computing. As part of this exercise, you will:

Conceptual

• Develop your understanding of theory, concepts and ideas relating to the supernatural or haunted, specifically rituals and practices of forecasting, augury, and soothsaying.
• Examine and explore ambiguity as a resource for designed interactions
• Investigate existing technologies, creative projects, critical designs, and precedents which operate intersection of hardware and speculation
• Explore the intersection of machine learning, prediction and ambiguity
• Critically reflect on the domain and develop a voice with critical making and speculation

Technical

• Be able to describe the fundamentals of working with tinyML;
• Be able to produce a working interaction using tinyML using IMUs, environmental sensors or gesture sensors as inputs for microcontrollers
• Be able to produce a working interaction with advanced microcontroller outputs, specifically screens, MP3 boards and thermal printers
• Be able to implement the steps in a machine learning / tinyML training process;
• Become familiar with the possibilities of APIS and online generative AI in image and text generation
• Be able to train and deploy a simple tiny Machine Learning model using Edge Impulse

Schedule

Deliverables and Deadlines

Case Study

Do some research and find an existing creative work that relates to this module. Review examples from Spooky Tech first. Identify and report on an example (research project, artistic installation, design project, etc.) related to the themes of the module: animism, enchantment, agency, intelligences, etc. The focus here is on discovering a product or project that you didn’t previously know about and represents an interesting approach, method or strategy that can inspire our work in this module. Read the full description.

Think Piece

On Forecasting, Predictions and Intelligences: To help prepare for your project, research and report on a topic directly related to the themes of the module: machine learning and prediction, how we interpret, behave and believe in machine intelligences, the role of ambiguity in design, and otherworldly frames for forecasting, augury and soothsaying. Prepare 500-750 words (max) in a short essay on a topic of interest that elaborates a critical point of view, raises a question, or provoke conversation. Read the full description.

Warm Up

To help prepare for your project, develop a chart, map, or system diagram that imagines and illustrates the possible logics of your interactive experience. This exercise asks you to speculate on your end interaction, how it might be encounters and how people would experience it and talk about it. Complete this in no more than 2 hours. Read the full brief.

Creative Project

Drawing inspiration from Real Prediction Machines and Automato.farm’s Believe it Yourself (BIY) toolkit, prototype a tinyML-enabled electronic object that uses an otherworldly practice for fortune telling, soothsaying, or augury. Examine how these rituals — for example, reading the tea leaves, palm reading, horoscopes, numerology, or dowsing — might offer ways to move from prediction to open-ended interpretation of data, machine learning, and device intelligences. Reflect on the value of ambiguity and these otherworldly metaphors in unsettling expectations of or beliefs in predictive machine outcomes. Read the full brief.

Readings and Prepare Aheads

Review For Class

Resources

Below is a list of additional online material that relates to the module and provides a starting point for your explorations. This is by no means exhaustive i.e. you should read/research beyond it.

Projects

• Curious Rituals by Nicolas Nova, Nancy Kwon, Katie Miyake, Walton Chiu. https://curiousrituals.nearfuturelaboratory.com/

• AltaR 3000. Vytautas Jankauskas, Robin Luo, Jiawen Yao, Gottfried Heider. https://vjnks.com/works/altar-3000/

• BELIEVE IT YOURSELF. BIY. automato.farm (2018) http://automato.farm/portfolio/believe_it_yourself/

• Slow Games - Garnet Hertz, Will Odom and others (2015-current). http://www.conceptlab.com/slowgame/ see also: http://www.ishback.com/slowgames/index.html & https://vimeo.com/109974160

• Morse Things. Doenja Oogjes, Ron Wakkary + others. http://iat-eds.dcr.sfu.ca/?eds_project=morse-things See also: https://doenjaoogjes.com/portfolio/morse-things/ https://www.youtube.com/watch?v=rPcobGuSVsc

• Real Prediction Machines. https://auger-loizeau.com/real-prediction-machines.html

• GhostWriter. Arvind Sanjeev. https://arvindsanjeev.com/ghostwriter.html see also https://vimeo.com/781361655 and https://www.creativeapplications.net/objects/ghostwriter-gpt3-powered-typewriter/

• Mind the Uuh. https://benedikt-gross.de/projects/mind-the-uuh/

• Tarot of Things. Haider Ali Akmal. https://haiderali.co/Tarot-of-Things

• almanac computer. David Benqué, 2018 https://davidbenque.com/projects/almanac/

• Unroll. Meijie Hu. https://meijie-hu.com/Oueksmorphism-1

• SneezeLove. https://studioforcreativeinquiry.org/project/sneeze-love

• Experiments with Google’s TensorFlow Lite for Microcontrollers projects - TensorFlow Lite for Microcontrollers

• Haunted Machine’s

Article and Reafings

• Sconce, J. (2000). Haunted media: Electronic presence from telegraphy to television. Duke University Press. https://www.dukeupress.edu/haunted-media (See Chapters 1 and 2)

• Davis, E. (2015). TechGnosis: Myth, magic, and mysticism in the age of information. North Atlantic Books.

• The Powerful Role of Magical Beliefs in Our Everyday Thinking. Gustav Kuhn. MIT Press Article

• Gaver, William W., Jacob Beaver, and Steve Benford. “Ambiguity as a resource for design.” In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 233-240. 2003.

• Wakkary, Ron, William Odom, Sabrina Hauser, Garnet Hertz, and Henry Lin. “A short guide to material speculation: Actual artifacts for critical inquiry.” Interactions 23, no. 2 (2016): 44-48 https://dl.acm.org/doi/10.1145/2889278

• Wakkary, R., Odom, W., Hauser, S., Hertz, G., & Lin, H. (2015, August). Material speculation: Actual artifacts for critical inquiry. In Proceedings of The Fifth Decennial Aarhus Conference on Critical Alternatives (pp. 97-108).

• Helmes, J., Taylor, A. S., Cao, X., Höök, K., Schmitt, P., & Villar, N. (2010, January). Rudiments 1, 2 & 3: design speculations on autonomy. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction (pp. 145-152).

• Odom, William T., Abigail J. Sellen, Richard Banks, David S. Kirk, Tim Regan, Mark Selby, Jodi L. Forlizzi, and John Zimmerman. “Designing for slowness, anticipation and re-visitation: a long term field study of the photobox.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1961-1970. 2014.

• Oogjes, D., Wakkary, R., Lin, H., & Alemi, O. (2020, July). Fragile! Handle with Care: The Morse Things. In Proc DIS 2020

• Wakkary, R., Oogjes, D., Hauser, S., Lin, H., Cao, C., Ma, L., & Duel, T. (2017, June). Morse things: A design inquiry into the gap between things and us. In Proc DIS 2017

• Audrey Desjardins, Heidi R. Biggs, Cayla Key, and Jeremy E. Viny. 2020. IoT Data in the Home: Observing Entanglements and Drawing New Encounters. In CHI ’20 :https://doi.org/10.1145/3313831.3376342

• Jacobs, R., Benford, S., Luger, E., & Howarth, C. (2016, June). The Prediction Machine: Performing Scientific and Artistic Process. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (pp. 497-508).

• Gaver, W., Blythe, M., Boucher, A., Jarvis, N., Bowers, J., & Wright, P. (2010, April). The prayer companion: openness and specificity, materiality and spirituality. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 2055-2064).

• Caitlin Lustig and Daniela Rosner. “From Explainability to Ineffability? ML Tarot and the Possibility of Inspiriting Design”. Conference on Designing Interactive Systems. ACM, 2022. [pdf]

• Churchill, Elizabeth F. “From data divination to data-aware design.” interactions 19, no. 5 (2012): 10-13.

• Kieran Browne and Ben Swift. 2018. The Other Side: Algorithm as Ritual in Artificial Intelligence. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA ‘18). Association for Computing Machinery, New York, NY, USA, Paper alt11, 1–9. https://doi.org/10.1145/3170427.3188404

• Reddy, A. (2022). Artificial everyday creativity: creative leaps with AI through critical making. Digital Creativity, 1-19.

• Lev Manovich. Artificial Aesthetics: A Critical Guide to AI, Media and Design. http://manovich.net/index.php/projects/artificial-aesthetics-book

• Warden, P., & Situnayake, D. (2019). Tinyml: Machine learning with tensorflow lite on arduino and ultra-low-power microcontrollers. O’Reilly Media.
• Gian Marco Iodice, Ronan Naughton. TinyML Cookbook. (2022). Packt Publishing