Robotic technology has been used in various sectors, from manufacturing to security and even in operating theatres. Despite this, the application of robotic systems is limited in unknown environments due to the severe risk an unknown environment presents. To aid a robotic system, building up a map of its surroundings is key to understanding its environment. There is a clear desire for robotic systems to safely map and understand various environments across a range of sectors, including Nuclear decommissioning, construction, site monitoring and even areas such as welding and social care. One of the major limitations is the lack of an integrated sensor system that works in tandem.

Callisto aims to address this need, combining advancing stereo camera technology to provide a 3D map of the scene and Raman spectroscopy to identify the chemical compensation of target substances within that scene.

Titania + Raman probe

Figure 1: Callisto system. Titania stereo camera integrated with RP1000 Standoff Raman probe

The system is the result of a collaboration between IS-Instruments and i3D robotics. It combines Raman spectroscopy with 3D vision to allow a user to both 3D map a scene and characterise chemical species within it, all in one integrated package. This technology emerged from a successful nuclear decommissioning project, where mapping locations and characterising samples were crucial. This system can enter environments that are dangerous to humans both saving time for the decommissioning and potentially saving lives.

Callisto brings together the Titania stereo camera from i3D robotics and the HES2000 Raman spectrometer from IS-Instruments.


First, a scene is mapped in 3D using a method known a SLAM (Simultaneous Localisation And Mapping). This works by tracking a camera in a scene by tracking important features in the images. The map is created by using a 3D system, which could be a stereo camera or LIDAR, however stereo camera systems are particularly suited to this task as they both generate 3D and provide an image that can be used for tracking.

As the stereo system moves around in the scene, the 3D data is added to a large map. This method works in real-time and can be very useful for robot systems to monitor their surroundings for path planning and collision avoidance. We use our proprietary stereo matching algorithm (I3DRSGM) to provide clear and accurate 3D for the mapping process.

Figure 2: Map being created using Callisto mapping software

Figure 2 shows a map being created. The white line represents the path of the camera and the green dots are key points along the path where new features were tracked. The 3D map is being created as the camera moves about in the scene.

Built on top of this SLAM method is an extra layer for tagging and storing other types of data in a scene. This extra layer allows for Raman data to be stored in the mapped environment. The Callisto system consists of a stereo camera and a Raman probe in one complete system which allows for tracking of the probe location in 3D space.

After mapping a scene, the user can then use the 1m focal length Raman probe to identify a sample they would like to scan. The probe would then be moved into the correct position for scanning the sample, all while being tracked in 3D space.

For the Raman measurement, a laser exits the Raman probe and interrogates the target. This light interacts with the sample, transferring a small amount of energy to the lattice, and scattering away with a slightly shifted wavelength. This is known as Raman scattering.

Callisto mounted on a robotic arm

Figure 3: Callisto system mounted on UR10

The probe then collects the photons from the target, filters out the elastic scattered photons and transmits the Raman scattered photons to ISI’s HES2000 spectrometer. From this a characteristic Raman spectrum is generated, which is then passed to ISI’s machine learning algorithm. This identifies the sample by using a complex neural network that has been trained on a large chemical database. ( Liu, J. (2017) Deep convolutional neural networks for Raman spectrum recognition: a unified solution.) The top 5 most likely matches are then displayed to the user.

The HES2000 spectrometer used is a spatial heterodyne Raman spectrometer. It does not have a slit, unlike traditional dispersive spectrometers, which gives it a higher throughput. The result of this is a Raman system that is particularly good at making stand-off measurements. Having the capability to make stand-off measurements is ideally suited for robotic deployment solutions, where access is often limited or inhospitable, which, in turn, benefits from real-time 3D mapping solutions.

When taking a measurement of a sample, an interferogram is returned and analysed by ISI’s processing and Fourier transform routines. Figure 4 shows a typical resulting spectrum.

Figure 4: Prototype Callisto system mounted on a robotic ROV


Figure 5: A example Raman spectrum of paracetamol

From this spectrum, Callisto can identify the sample using machine learning which is tagged in a 3D map. Quick identification of different types of samples is easy as each sample is colour coded in the map.

Figure 6: Process for viewing spectral data in 3D map


The system has been demonstrated for the Nuclear Decommissioning sector where the identification and characterisation of radioactive material and other hazardous substances are an essential requirement. Hazardous materials (acids, alkalis, organic solvents) may be derived from multiple sources and may be present in multiple locations. They may be stored in containers, tanks or silos, which have been in operation for several decades and the composition of the waste and the condition of the storage vessels is often unclear or unknown.

Currently, hazardous substances are identified by the visual inspection of an experienced operator, who must then collect samples to be analysed. This is an expensive and potentially hazardous process. Therefore, in-situ identification of unknown samples which may be hazardous or indicative of corrosion without the need for direct human interaction is a clear advantage.

Callisto was used to identify several chemicals of specific interest to the nuclear industry. This included a Magnox slurry simulant material (Versamag), uranyl nitrate, kerosene, tri-butyl phosphate (TBP). Mixed samples of TBP and kerosene were also examined to demonstrate the detection of multiple materials simultaneously. The system mapped the target area and then targeted the Raman spectrometer at the suspicious samples.

We know Callisto is an exciting development, combining advanced 3D vision systems with the latest Raman technology and artificial intelligence. It can be adapted for a range of environments and functions with additional capability added. This might include additional sensors or varying the robotic or visualisation platform.

There are clear applications in a variety of sectors including:

• Nuclear Decommissioning

• Additive Manufacturing and Welding

• Construction

• Foundation/Process Industries

• Medical

• Security

• Social Care

While this system focuses on bringing together spectroscopy and 3D mapping, the system is built in a modular design so other data sources could be added, such as thermal imaging or radiation sensors.