My Research

Background

When we look at objects, we observe from a single, cyclopian view. However, the brain is stitching information from two different sensors at two different locations in space to create that view.

How does the brain know how to focus on an object, and know that the object in the left eye is the same object in the right eye? This is the correspondence problem, meaning the challenge of finding what parts of each image correspond to the same object in space.

Unfortunately, the human brain is incredibly complicated and difficult to study. Fortunately, we are not the only animals who can see.

Insects- especially fruit flies (Drosophila melanogaster)- are a common first destination when approaching complex biological processes due to their relatively simple neural circuitry and their suitability for lab testing. Yet, due to their small size and the complexity of visual processing, more than half of Drosophila melanogaster‘s neurons are dedicated to vision.

Insects with compound eyes have columnar segments called ommatidia that act as photoreceptors. Clusters of neurons downstream from the receptors process that information to, for example, observe optic flow or primary patterns. In our case, it’s depth perception.

Above is a representation of a sectional view of a compound eye. To the right is from a bioinspired camera based on the working principles of compound eyes. The object image is not split into individual pixels, but is cast onto each ommatidium with the object in the fields-of-view. Then, the corresponding strength of the signal sent by the neurons in the eye correspond to the light intensity at each ommatidium. When compounded, the resulting image lacks fidelity but can capture the movement of an object well.

In the arthropod phylum, mantises are particularly adept at tracking prey and can determine when its prey is close enough to strike. This, in spite of their simple neural circuitry. It seems that solving the correspondence problem outright would take more computational power than is available- yet the mantis still behaves as if it can process depth information. What is going on?

Understanding how mantises are able to use their binocular vision to track prey can inform our understanding of stereopsis and sensor fusion.

Previous work

My collaborator, Dr. Jenny Read and her colleague, Dr. Vivek Nityananda, conducted experiments to understand the behaviors of the mantis. From these observed behaviors, Dr. Read developed a system that sought to mimic the mantis and its ability to track and strike prey once it has gotten within reach.

The figure on the right is from her paper introducing this model. Each eye observes a ‘scene’ in the form of an image, which feeds into a Fully Connected Neural Network (FCNN) that relates the distance disparity and projected location on each eye.

The result is a model that can gauge relative distance from an object in cases where one eye alone would fail. Monocular depth perception is weak to larger objects appearing closer than they actually are due to the size the object subtends upon the retina. In comparison, binocular depth perception does not rely on only the size of the object, but rather how objects at different distances shift different amounts from the perspective of each eye.

In the figure above, Dr. Read simulated her model and plotted the successes and failures based on the range of target distances and the angular error between the chosen target and the target that was actually the closest. The simulation used spheres of varying sizes at different azimuthal locations and tasked the model to output the angle of the closest object.

The model’s ability to discern the closest object is significantly hampered when the objects are similar distances apart from the observer. This sharp cutoff observed at 10 cm is likely due to the size of the targets themselves.

This is all well and good, but these results are from a simulation. How can we verify that this model would be applicable outside of the simulation?