Fluorescence microscopy has been a genuinely disruptive innovation for examining live specimens. It has enabled the invention of numerous imaging techniques and methods for observing the behaviour of biological objects of interest in a way that mirrors their spatial and temporal scales as closely as possible. Fluorescence microscopes have become sophisticated devices combining all these photonic, chemical and biological innovations.

Image acquisition software is the component that enables interaction and coordination between these innovations. It allows users to define parameters for their image acquisition sequences based on their samples, research, experience and creativity. It automates image capture, enabling researchers to get as much information as possible out of their specimens.

Best industrial practice for sophisticated devices consisting of several actuators requiring coordination is to use a distributed control system implemented on specific electronics such as a PLC. While these control systems and their supporting electronics have different names depending on their industry or complexity, they all have one thing in common – they are used exclusively for their dedicated task.

This is not the case in microscopy. Commercially available image acquisition software controls the microscope from a standard computer. This raises issues, since even the most powerful personal computer is still a multi-tasking device intended for office automation applications. And it is not designed to control the execution of an image acquisition sequence on a system as complex as a fluorescence microscope.

ONE BRAIN. TWO FUNCTIONALITIES

All imaging software for microscopes must manage two very different functionalities.

The first, which we will refer to as the “User Interface”, involves interaction with the user. This is the visible part of the software – its interface must enable the researcher to trigger microscopy system functionalities, acquire system status feedback, and of course, receive acquired data for viewing or manipulation.

The second functionality, which we will refer to as “Device Control”, involves execution of the acquisition sequence and communication with each of the microscope’s third-party devices. Communication with third-party devices is performed by “low-level” protocols, ranging from simple electrical TTL-type signals with a high or low voltage level depending on the required state, to more elaborate protocols such as USB, enabling several set point values to be encapsulated in a single frame (e.g. stage x and y coordinates).

It is entirely appropriate to deploy the User Interface functionality on a computer or touchscreen device. Indeed, the trio consisting of a computer mouse/keyboard/display is still the most user-friendly means of entering parameters or manipulating images, even when their volume or complexity increases.

In contrast, the Device Control functionality is not suitable for execution on a computer and benefits from being implemented on dedicated electronics. When the imaging software executes the acquisition sequence on a computer, communication by low-level protocols, which are directly managed by the microscope’s third-party devices, is not possible. The computer operating system (virtually always Windows for microscopes) requires software drivers to be used to communicate with each device external to the computer.

So, what exactly are these software drivers that often create issues for microscopes? They could be described as “translators” enabling software commands run in Windows to be converted into low-level language understood by target third-party devices. Each of these is a computer program composed of tens to thousands of lines of code. Developing the driver for a microscopy third-party device is therefore an onerous task for manufacturers as it requires time and expertise, which of course cost money.

Drivers are also a major source of bugs, as unfortunately, programming quality aside, the more lines of code there are to execute, the more bugs there will inevitably be. Finally, drivers considerably slow the image acquisition framerate, simply because the hundreds or thousands of lines of code included in them use up execution time. While most microscope users are unaware of this latency, anyone who has ever triggered third-party devices with a specialist circuit board will be aware of the scale of this phenomenon. Unfortunately, it is not possible to solve these problems by increasing computer processor performance or memory capacity (RAM).

LEFT BRAIN: DEVICE CONTROL

Where all other factors are constant, the framerate for standard 4D (XYZT) or 5D (+Channels) multidimensional acquisition is three times faster if the Device Control functionality is implemented on dedicated electronics, regardless of the commercial control software used.

That’s right – at equivalent exposure time and system configuration, it is possible to collect three times more images per second with optimised control of any of the advanced microscopes produced by top manufacturers Leica, Nikon, Olympus and Zeiss, equipped with a scientific camera made by Andor, Hamamatsu, PCO or Photometrics.
This provides researchers examining biological events occurring in live specimens with abundant additional information. Drawing a parallel with spatial super-resolution, it could be described as “temporal super-resolution”!

This factor of 3 is in fact an approximate figure. Compared to the least efficient image acquisition software, the difference may be even greater (x7).
This performance factor is affected by two parameters:
1/ Exposure time. The performance factor gains in significance the shorter this is. Conversely, it loses significance if the exposure time is long (>250ms), since the impact of latency is proportionally lower.
2/ Acquisition sequence complexity. The higher this is, the more other systems are outperformed. In general, the more third-party devices there are to control in a microscopy system, the more optimised control considerably improves the framerate.

The other main advantage of executing sequences outside a computer is that users are free of Windows’ multi-task management. A computer’s operating system manages at least fifty processes vital to its functioning (these can be identified and counted in task manager) and imaging software is just one of these many processes. This adds even more latency to the microscope control system due to the computer sharing its resources among the various processes.

More importantly, multi-task management by the computer causes significant problems in terms of the reliability of scientific results due to random variations in the timing of communication signals sent to the microscope. Such temporal variation is generally referred to as “jitter”. This phenomenon prevents the repeatability of acquisition timings, reducing and in some cases even compromising the consistency of results analysis. Bypassing an operating system is the key to eliminating jitter.

RIGHT BRAIN: USER INTERFACE

Now let’s consider the User Interface functionality. In conventional imaging software, this is interlinked to a greater or lesser extent with the Device Control functionality.

This type of design requires compromises to be made in terms of the potential offered by these two functionalities. In particular, any decisions that may be made to ensure the tool is intuitive and user-friendly must not impinge on its reliability for performing image acquisition sequences and executing third-party device drivers. As in a children’s balance game, the more interlinked the software’s functions are, the more difficult it is to modify or add a part without potentially creating a problem on the other side.

This is a particularly important issue, since fluorescence microscopy techniques are continually developing and are now combined to extract increasing amounts of information from specimens. The software architecture needs to be designed to cater for this, otherwise each new addition creates structural instability.

However, users opening microscope imaging software for the first time are immediately struck by the multitude of windows they need to open, close and organise, all packed with information, as well as the overwhelming array of menus and submenus.
Even experienced researchers sometimes find themselves struggling to manage imaging software, while they should be focusing their energy on optimising the parameters of their acquisition sequences based on the specific characteristics of their specimens.

This disconnect is compounded by the fact that people are now accustomed to using attractive interfaces on their personal devices that enable them to do things simply at the touch of a screen, even rendering user instructions obsolete. Well-designed, user-friendly professional software saves its users productive time, which is particularly valuable for microscope imaging software used by highly qualified researchers.

 ARTIFICIAL INTELLIGENCE WAITING IN THE WINGS

With the application of artificial intelligence to microscopy, acquisition software will emerge as the brain of microscopes with even more force. It will be interesting to see what form it takes.

How important is imaging software to you? How does it impact your work? Your comment is welcome!