The Space of surveillance

From the concept of pixels to optical sensors for digital cameras: computer vision owes much to space research. Now that cameras can be integrated into every device, how do they affect our movements and our society? You can find it out at the exhibition When They See Us, held in Bologna from 17 to 28 September 2024, organised by the cultural association Sineglossa, which is responsible for the editorial direction of Mangrovia. Here are the details.

Behind the lens, a space sensor: the most common video cameras use the CMOS active-pixel sensor, developed in the early 1990s at NASA’s Jet Propulsion Laboratory¹.

Smaller cameras were needed for installation on spacecraft for extra-orbital missions: Eric Fossum’s team solved the problem by configuring the components of the new optical sensor on a single chip, which also included amplifiers to boost the output electrical signal. Less material space, lower energy consumption, and more efficient image processing. Since then, digital photo and video cameras have become increasingly smaller and have been incorporated into mobile phones, computers, smart glasses, cars, and installed at road intersections and public places.

Many small eyes, through which we look and are looked at. As devices shrink in size, does the space of surveillance increase?

The difference between human and computer vision

The back of our eyeballs is covered by a thin membrane, the retina, where cells that receive light (photoreceptors) are located. They convert light waves into electrical signals, which travel through the optic nerve to the brain. It is only within the brain that the image is processed, “composed” in 3D, through a psychical and multisensory process—both analog and digital—where light becomes information and meaning. Can computers replicate this process? To answer this question, computer vision was born in the last century, the interdisciplinary field that deals with artificial vision.

We are still far from achieving “high-level” vision: a computer system cannot (yet?) reconstruct and analyse the entire 3D context in which an image is placed. However, significant progress has been made in “early vision,” which focuses on the acquisition, pre-processing, and encoding of visual information without interpreting the context. In artificial vision systems, the optical sensor reproduces the function of biological photoreceptors: it captures light, converts it into electricity, and sends the signal to the electronic brain-computer.

Every digital image, much like in pointillist paintings, is broken down into dots, pixels, a term coined by engineer Frederic Billingsley at the Jet Propulsion Laboratory to describe the portions of images received from spacecraft scans². Whether arranged on a grid or depicting geometric figures through mathematical equations, the pixels in the image reach the computer as a set of numbers. So how does the electronic computer compose the image and find the object it is looking for?

Through specific algorithms that tell it “what to look for”: algorithms previously trained through increasingly sophisticated machine learning techniques (deep learning), which fall under the broad umbrella of Artificial Intelligence. Whether it learns through supervised models, associating certain pre-classified characteristics with a label, or whether it learns “on its own”, finding similarities among unclassified data, the artificial vision system will extract from the image the elements or portions of interest, compare them to what it has learned, and perform the task it has been programmed for. Recognising our face, for example. Or telling us what it is looking at.

When machines watch us

«The Surveillance Speaker has been active since 2018: the speaker describes what a rotating camera is observing. Six years ago, the technology didn’t allow for many details to be seen, like the type of clothes people were wearing, and there was still a noticeable difference between artificial and natural voices. Nowadays, things are changed. It’s a work that I update weekly; there’s always something new in the technologies I’m using». Dries Depoorter is the Belgian artist who, with The Follower, showed the world how it is possible to cross-reference geolocated photos posted on Instagram with open surveillance camera data in public spaces to reconstruct an individual’s movements. «When I studied Media Arts in Ghent, I discovered that you can access data and images from open cameras, meaning those connected to the Internet with a standard or no password», he recalls. «This discovery sparked many of my projects on surveillance».

The Surveillance Speaker is one of three works by Depoorter on display at the When They See Us – Quando le macchine ci guardano (When Machines Watch Us) exhibition, running from 17 to 28 September 2024 at the Biblioteca Salaborsa in Bologna. It’s the first event of The Next Real, a series on art, AI, and society curated by the cultural association Sineglossa. «I haven’t spent much time in Italy, so I’m looking forward to coming», says Depoorter. The exhibition also features Border Birds (2022-2024) and Jaywalking (2015-2024): the former, created with his sister Bieke, captures images of birds crossing borders between Mexico and the United States, Morocco and Spain, Greece and Turkey, France and England, while the latter directly involves those watching the live video streams. Jaywalking consists of a series of screens showing live-streamed footage from surveillance webcams installed at street intersections in various countries. Pedestrians crossing outside the lines can be reported to the nearest police station with the push of a button, triggering an automatic sequence: first, a screenshot is taken, and then an email is sent to the police. All automatic. How many times has the button been pressed? «I monitored it after a few years,» Depoorter explains. «There are variations between countries and depending on the exhibition setups, but especially recently, the button has been pressed most of the time».

Why? The possible explanation is left to those who observe and interact at the exhibition: «I try to keep my work as simple as possible», the artist concludes. «I don’t want to offer additional explanations». The exhibition will also feature The Glass Room Misinformation Edition project, curated by the NGO Tactical Tech, and is promoted by the non-profit associations The Good Lobby and the Hermes Center for Digital Rights, in collaboration with info.nodes, all dedicated to promoting digital rights.

«Fifteen years ago, when we raised the issue of mass recognition in public spaces, we were told that this was a concern only in authoritarian regimes, like China, and wouldn’t affect Europe», recalls Davide Del Monte, Executive Director of the Hermes Center and founder of info.nodes. «In reality, the development of security policies is leading to a normalisation of the militarisation of public spaces, also through the use of these technologies. Migrants at borders are often used as guinea pigs to test their effectiveness».

The European AI regulation, the AI Act, prohibits the use of biometric recognition in public spaces, but with various exceptions related to criminal activities. «The prevention of certain crimes, such as drug trafficking, could open the door to facial recognition cameras in any park», Del Monte explains. «This exhibition is one of several activities aimed at citizens, alongside training for journalists and NGOs working on migration, as well as institutional advocacy». And how do you react when you feel watched?

---

1. To discover more on CMOS active-pixel sensor, see Fossum E. R. (1993), Active pixel sensors: are CCDs dinosaurs?, in SPIE Proceedings Vol. 1900: Charge-Coupled Devices and Solid State Optical Sensors III, vol. 1900, International Society for Optics and Photonics, 12 July 1993, pp. 2–14 ↩︎
2. For more, see Billingsley F.C. (1967), Processing Ranger and Mariner Photography, in Computerized Imaging Techniques, Proceedings of SPIE, Vol. 0010, International Society for Optics and Photonics, Jan. 1967 (Aug. 1965, San Francisco), pp. XV-1–19. ↩︎