Marcus Lyon is a British artist whose works are held in prestigious institutions worldwide, including the Smithsonian Air & Space Museum, the Museum of Fine Arts Houston, and the Arts Council Collection of Great Britain. He marries together science and sound with traditional arts forms, through the themes of globalisation, identity and dance.
Calvium first worked with Marcus on his acclaimed Somos Brasil (“We Are Brazil”) project, and more recently with i.Detroit: A Human Atlas of an American City, which will see a book and an exhibition launch this year. A project that started in 2018, i.Detroit shows the diversity in the city by plotting the ancestral DNA of 100 extraordinary Detroiters who are pushing forward social change and telling their stories through photography and sound.
In this article I want to take a look at how i.Detroit has been brought to life and the careful consideration that was taken when Marcus went about telling the story of the city and its people. I’ll then focus on some of the work that Calvium has undertaken to design and develop an app that seamlessly integrates with this body of work.
Before i.Detroit came Somos Brasil
i.Detroit would not be possible without Marcus’ experience with Somos Brasil, during which he and his team painstakingly mapped the ancestral DNA of more than 100 Brazilians, photographing them and recording their stories for six months.
Marcus and Matt Hill of Rethink Audio approached Calvium with the idea of creating an immersive app that features the portraits of the Brazilians they interviewed, whilst adding a layer of interactivity through soundscapes and image recognition technology. Instead of simply browsing through the photos, users can also listen to the oral histories of every Brazilian featured, allowing them to move past the flat surface of each portrait and interact more deeply with the photos.
After Somos Brasil, came WE: deutschland, a similar project which featured 51 Germans working predominantly in the diversity and migration space. When commissioners saw what Marcus had done with Somos Brasil and then WE: Deutschland, they wanted to do the same with Detroit.
From Brazil to Detroit: Mapping the Human Landscape
During the launch exhibit of Somos Brasil Portraits and its supporting book in 2017, Marcus told us that he had been gripped with an existential crisis about what his next big challenge would be after working on this project for several years.
It was the insight of his friend and trusted advisor, Tony Elliot, that helped Marcus lay down the plan for a series of projects similar to Somos Brasil that would “explore the truth at the heart of our societies through photography, anthropology and science.”
This was the beginning of the Human Atlas project—a series of projects that would map, collate, interview, design, and quantify groups of inspirational people all over the world, much in the way that Somos Brasil had.
“We were making human atlases. Like cartographers we were trying to give meaning to landscape, human landscape,” he explains. “We were trying to map the miracle of human endeavor, the endeavor to build more powerful communities through inspired leadership and commitment to a more hopeful future.”
i.Detroit: A Human Atlas of an American City
Today, Marcus is embarking on the next leg of his Human Atlas journey by telling the stories of Detroiters and giving voices to those who are underrepresented or misrepresented in what was once known as America’s Motor City. The i.Detroit project, commissioned by the Kresge Foundation and in partnership with the Charles H. Wright Museum in Detroit, is a deep-dive exploration of the human capital of the ‘Motor City’.
Similar to Somos Brasil, Marcus analysed the DNA of 100 Detroiters of all ages, races, and backgrounds (selected by a Detroit nominating committee). He then traced their generational migration to the city and interviewed each one to record their personal stories.
From the outset, Marcus understood how Detroiters would not take too kindly to outsiders who record locals’ stories only to pick those that will support their own preconceived notions. He endeavoured, therefore, to let the locals speak for the city as they see it themselves and not let his personal views colour the project.
To Listen, to Honour and to tell the Actual Story
As the third initiative under his Human Atlas project, Marcus now knows that listening is key to its success. The nomination process alone took six months to pick the names of Detroiters who will represent the soul of the city. He and his team made sure they listened to the voices of several communities before making any assumptions to provide proper representation of more than 4,000 years of making an impact in the city.
“The art of the portraitist is to make the camera’s gaze the right type: kind, enquiring, thoughtful and connected. For me this is not a dark art, there are no tricks or clever devices that can outplay good old-fashioned empathy and listening,” Marcus writes in his paper The Origins of Detroit. “And this is the key to the Human Atlas: we are there to listen, to honor and to tell the actual story. Our intentions are only to do that. In the tsunami of misinformation and the constant noise of 24/7 social media, our choice is to slow down… to witness more deeply… and truly touch another life.”
Creating an Interactive Exhibition and Book
The entire i.Detroit project involved collating 1,202 pieces of information over the course of two years, including the ancestral DNA of 100 individuals, 91 places of origin, and maps of migration journeys out of Africa that date back thousands of years.
Marcus and his team formed a team and partnered with companies and individuals who helped bring his vision to life and create the components that would make up the i.Detroit project.
Since the journey of i.Detroit started in 2018, Marcus has been working closely with his team in coming up with the final list of 100 participants, creating their portraits, and recording their stories.
For this, Marcus credits Camilla Pastorelli (long-time producer and colleague) and Joe Briggs-Price (studio manager, photography, and sound assistant)—both of whom played huge roles in creating the final project.
After two years, all the data they collated was brought together to form a fabric of identity that tells the story of Detroit, of human excellence and leadership.
According to Marcus, the book would be no more than a compilation of information without the help of essayists Kenyatta D. Berry, Professor Howard Bossen, Ken Coleman, and JoAnn Watson.
He also named Jim Sutherland, Joe Briggs-Price, Camila Pastorelli, L’Renee Hollins, Rosey Trickett, and Jacob Wheeler for their exceptional design work and copy editing that did justice to the histories of these 100 extraordinary individuals.
Since Detroit is the birthplace of Motown, Marcus’ team selected 100 inheritance tracks from the individuals they interviewed—a legacy they say they wish to pass on to the next generation.
A 45rpm 7″ single will accompany the limited-edition book, which was made possible through his collaboration with Brian Eno and Derrick May for the bespoke soundtracks based on the recordings of Efe Bes’ drums, Marcus Elliot’s saxophone, and the voice of Giizhigad.
The interactive app will enhance the visitor experience of the i.Detroit exhibition and accompanying book. It adds a fresh sensory dimension to the works, through sound. The Somos Brazil app had an emotional resonance, so it was important that this wasn’t lost when developing an app for i.Detroit. The sound design for I.Detroit is aimed at moving to the next level: ambient sounds will be recorded separately and the sound should be interesting, layered and emotional. The technology will work to connect the audience to the person in the photograph.
Using image recognition, when a visitor uses their smartphone to scan one of the photographs, a piece of pre-mixed audio plays.
Deeper dive into the app
So, when it comes to hearing the voices and the music of the i.Detroit project, it’s all about the app. The Calvium team has created an audio experience that seamlessly integrates with the visual elements of the book and the exhibition. For instance, when a user scans a portrait the voice of the subject will play and their story in their own words can be heard. Similarly, scanning the front cover or back cover of the book will trigger music tracks. In addition, the accompanying I.Detroit playlist can be accessed when scanning the i.Detroit logo in the book.
That all sounds very simple, and for the user experience it is and should be. However, for the development team it took a good deal of painstaking experimentation to create such a smooth experience. The team had used an external provider’s image recognition technology in the initial Somos Brazil App, but as the tech had since been bought by Google it was no longer available. This meant that Calvium had to search for a different image recognition solution that would work and after much testing, settled on using Firebase’s (Google’s app development platform) machine learning capability.
Firebase has a general purpose image recognition library/feature for Android and iOS that focuses on recognising common things (house, dog, person, plant, etc …) but it didn’t have the ability to recognise a high number of complex markers (a marker in this context is a particular image that should be recognised and not a class of things for example). In addition, we were building the app using the React Native framework and that added a layer of complexity that we needed to overcome. So after some preliminary research we determined that it should be possible to train the models to recognise the high number of specific markers required (more than 100 in the app).
To train the Machine Learning models we started with the original JPEGs and from those we generated a data set that gave us a decent starting point. It provided us with good results for most of the i.Detroit images, but not all. One thing that became apparent during testing is that training the models with real images taken with the phone cameras gives more accurate results, but of course it also means that the work to generate the data set could not be totally automated. Hence, we approached this activity as a mix of both techniques – using real photos of specific portraits when the automated data set was giving results below our desired accuracy.
We also found that when a person frames a portrait through their camera they centre on the face and don’t necessarily choose to frame the whole image. As such, the accuracy of the image recognition is much better when we build the model using just the face and not the whole image. Thus, we now present just the subject’s face and a small bit of contextual detail around them when we feed the machine learning model. To achieve this, when scanning we apply face detection to the scanned image so that we can crop the desired area around the face and apply image recognition only to that portion.
Our work is extensive and although there’s much complexity in that which we do to build the app, for the user the experience is simple and beautiful.
An Agent of Change
In an interview with the Detroit News, Marcus said he “wants to inspire others to do more in their communities, neighborhoods and families” by featuring a powerful cohort of change agents and introducing them to a much broader audience. His aspiration is to inspire the next generation of activists and leaders not to fashion their lives in the shadow of those who came before but in honour of them, seeing the work and lives of those whose stories he has told as the real heroes of our time.
The I.Detroit book launches this August and collectors will be able to use the app to listen to each subject’s soundscape while looking at their portraits and understanding their ancestral DNA maps alongside essays and maps of this extraordinary city – a real, modern, interactive atlas of an extraordinary group of people.
Book available to pre order at – http://www.marcuslyon.com/artworks/idetroit#!/cart