EBSynth

On a video of 2 minutes paper (link), Károly Zsolnai-Fehér presented a new approch to do a style transfer on a video starting only with one transfered image.

Until now, it was required to transform each frame with style transfer. This creates flickering beacuse the transformation starts with random noise. Several approaches were done by using RNN in order to keep temporal consistency but the result was not very good.

The new approach can be easily implemented on any project by using the software available for free at ebsynth.com.

In this notebook, we will use it to tranfer a part of a video downloaded from youtube. Attention, the video requires to have few movements between frame and no hidden artifact. You should refer to the tutorial on their website to understand it. In this exercice, I'll use 2 videos of a village taken from a drone.

In [2]:
import cv2
import numpy as np
import glob
import os

Create Frames

The video is a complete report about a small village in France. I'll keep only 2 distinct parts with an aerial view taken from a drone. Each frame has to get a numbering and no other info.

In [21]:
vidcap = cv2.VideoCapture('video.mp4')
success,image = vidcap.read()
count = 0
nframes = 1
success = True
while success:
    success, image = vidcap.read()
    if 483 < count < 625 or 1510 < count < 1673:
        cv2.imwrite("input/{:06d}.jpeg".format(nframes), image)
        nframes += 1
    if count > 1673:
        break
    count += 1

Make Style Transfert

Now I have all the frame of the 2 parts of the video

  • 1 to 141 : First part of the video
  • 142 to 303 : Second part of the video

For the Style Transfer, I used a website instead of building the code. I ran it on the first image of the clip and save the rendered image as a keyframe for the tool (link)

As there is 2 parts in the video, I have to transfer 2 images. They are placed in the folder keys. Attention, the name must match the number of the original image. In that case, the frame 0 and 142 are converted

Construct Transfered video

Now we have the video in one folder and the keyframes transfered in another folder. We can now do the rendering of the transfered images

This step is quite long, it tooks around 1h to render the 300 images

Rebuild video

Now we can rebuild the 2 videos (original and transformed)

In [3]:
img_array = []
for filename in glob.glob('input/*.jpeg'):
    img = cv2.imread(filename)
    height, width, layers = img.shape
    size = (width,height)
    img_array.append(img)
 
out = cv2.VideoWriter('in_video.avi', cv2.VideoWriter_fourcc(*'DIVX'), 25, size)
 
for i in range(len(img_array)):
    out.write(img_array[i])
out.release()
In [4]:
img_array = []
for filename in glob.glob('output/*.png'):
    img = cv2.imread(filename)
    height, width, layers = img.shape
    size = (width,height)
    img_array.append(img)
 
out = cv2.VideoWriter('out_video.avi', cv2.VideoWriter_fourcc(*'DIVX'), 25, size)
 
for i in range(len(img_array)):
    out.write(img_array[i])
out.release()

The result can be seen on the script page of the website. The result is quite consistent along the video. We can see a small defect on the root of one house because the original video had deinterlacing. However the consistency remains impressive

Conclusion

In conclusion, we discovered on this notebook, how to apply style transfer to a video by having only one example of the style transfer. What was not tried is to apply mask. In this software, you can include mask to not transform the background. This will be very usefull in cinema thanks to green background technic but could not be used on this specific video. However, the principle is exactly the same as keyframes.

The result is impressive with a software free and easy to use. The temporal consistency is maintain and there is safe bet to say that this method will continue to evolve ,and may be used in animation with latest technologies