The method of Predictive Markers or how we step closer to the white paper.

Sketchar -

We are on our way to the complete absence of markers on paper, where any stroke a user makes is considered as a marker. In the meantime, we have achieved a better detection and attachment, for example, if you close three of the four markers, our algorithm will guess where they should be, saving and holding the virtual image in place. And even if some of the markers go beyond the frame — the image will also remain in place! We call this method “prediction of markers.”

How does it work?

We detect the position of the marker in the frame and when the marker is lost our algorithm assumes its location by matching with the location on the previous frame. For example, if in the previous frame the left lower marker was found on the coordinate 100 x 100, then the nearest marker found on this frame is the lower left.
We successfully tested three cases: when one, two or three markers were lost. If all four markers lost — we cannot predict so far. So, for now, the algorithm should detect at least one marker.

The more markers are detected, the more accurate the prediction:

  1. When three markers found, it is allowed to move the camera in a rather broad range, as well as shift relative to the sheet and scale.

First, we detect one marker, then bound it to the marker from the previous frame (for example, we realize that this is the lower left mark). We look at how the marker has moved from the previous frame to the current and just relocate the entire sketch. We believe that the motion is linear and make the prediction with a very little accuracy, but it is enough if the camera moves a bit.

  1. When two markers are detected, the displacement will be noticeable for any movement of the camera.
  2. And if only one marker is found — a displacement of the camera by a couple of centimeters leads to noticeable problems.

This method is our small technological victory, which will significantly improve the user experience.

Update the iOS version of SketchAR to check out the difference!