Spotlight CES: Spectricity unveils first multispectral image sensor for smartphones

You wouldn't necessarily think there's many more improvements to be made on the cameras found in some of the latest smartphones.

Thanks to the likes of Google's Pixel and the evolution of Samsung's Galaxy offering, amateur smartphone snaps have verged ever closer to the cinematographic art form, driven by AI that 'cheats' and does some of the creative work for you. 

Last year, in tech's photographic and creative image realm, it was generative AI that grabbed much of the limelight.

The exploits of OpenAI's Dall-E 2, Stable Diffusion and other generative AI models are predicted to further democratise the use of creative image making tools, combining real photographs with text-to-image systems to sculpt imaginative art scenes from scratch.

The results are undeniably creative ⁠— tell Stable Diffusion to draw a Brazilian street soccer player in Tokyo and sure enough there he is, about to showcase some nifty footwork at a Tokyo crosswalk and clad in Brazil's green and gold.

Photo credit: Stable Diffusion

As text-to-image generative AI continues to develop, there's a glaring blot that's been left in the smartphone industry's printing room.

Capturing colours accurately remains a challenge even for the latest smartphone cameras, and that means many modern photos fail to portray some objects or skin tones accurately.

As these images can form the basis for generative AI models, it's easy to see how a disparity in AI depictions of different skin tones could arise.

Difficulties also face industries that want to use smart imagery to build new products to fit specific colour palettes, in personal cosmetics for example or certain e-commerce style offerings.

One option to help tackle this is the use of multispectral image sensors, already used by government agencies from NASA to US Army research lab to distinguish land surface features imperceptible to the human eye.

The main challenge until now has been shrinking the multispectral camera to fit inside minaturised chip-based form factors, so they can be embedded into smartphones and other consumer devices, says Spectricity, which announced the forthcoming launch of its product in this category at CES today.

Spectricity's S1 spectral sensor will supposedly become the first "miniaturised" and "mass-manufacturable" mobile unit that captures full visible and near-infrared range in similar fidelity to video, reducing the errors of automatic white balancing channels, which are limited to blends from conventional RGB coding. The use of this spectral technology in tandem with a smartphone's main shooting lens should enable it to pick up on the true white balance in every hue, rendering colour at far greater fidelity.

Jonathan Borremans, chief technology officer at Spectricity, explained: "Essentially, even your high-end smartphone camera is colour blind. The S1 uses richer spectral signature data, resulting in much higher colour fidelity in your pictures. 

"In order to represent colours accurately, one must estimate what the white point of the scene is. What does that mean? Objects, colours, skin look different in different lighting conditions. 

"For example in the cold light of your bathroom you may look very different as compared to during a beautiful sunset. To accurately represent colours, cameras estimate the lighting conditions of the scene, and then correct the colours to compensate for this. This is called automatic white balancing (AWB): the algorithm tries to make white truly white (and other colours truly what they should be), independent of the lighting conditions. However, if the AWB algorithms make a mistake in estimating the lighting conditions, they basically screw up the white point, and the colours. Normal cameras have hardly any information, and basically have to guess the lighting condition, or white point."

Photo of S1 multispectral smartphone imaging chip, courtesy of Spectricity

Hobbyists have been trying similar innovations in a personal capacity. Last June, DIY-sensor enthusiast Robin Kearey claimed to have upscaled an old Google Pixel 3A to add ultraviolet and infrared spectral imaging.

In an interview with hardware hacking news source Hackaday, Kearey explained how multispectral cameras could detect wavelengths that aren't visible. While pricey, these cameras can tell the difference between two paint pigments, both of the same hue, that were used at different times in history and therefore used distinct materials, to cite Kearey's main example.

Using such imaging in smartphone cameras could remove some of the colour discrepancies for generative AI, with wavelength markers used to flag up otherwise indeterminable pigmentation. Spectricity's Borremans tells Tech.eu that the S1 spectral chip will measure skin's blood volume, oil factors, blood oxygen and moisture. In healthcare, he predicts more advanced spectral smartphone sensors could allow clinicians to detect skin cancers such as melanoma, due to specific spectral signatures linked to the disease.

"Through the spectral signature we can detect fake skin, because skin has a specific spectral signature." Borremans said. "So, not only can we detect the correct skin tone, but also detect fake masks. This could help in security authentication use cases to complement other computer vision techniques. Also in general, it could authenticate counterfeit materials in some use cases."

"Even more advanced, it can help existing imaging based AI algorithms to detect melanoma to become more specific and sensitive thanks to the spectral signature of melanoma. All these applications are currently used in laboratories or doctor's practices, but Spectricity's sensor now brings this potential to consumer markets."

Image of Spectricity founders Vincent Mouret (CEO, left) and Jonathan Borremans (right)

Spectricity anticipates shipments of smartphones containing its minaturised spectral camera will begin from 2024, and it is currently evaluating the sensor's efficacy together with leading smartphone makers.

Next year's launch on flagship smartphone cameras will conclude an R&D campaign that began in 2018 from a lab sponsored by Imec, the Belgian research institute. Following that, certain low-to-mid range smartphones could be sporting Spectricity's S1 sensor from 2025.

Borremans claims to have around 100 patents and publications to his name for specific innovations pertaining to imaging, circuits, systems and medtech. Following a €14 million series B raise announced August 2021, Spectricity has raised north of €23 million to date, from a slate of VCs, including Capricorn, Atlantic Bridge and imec's Xpand.