Transitioning to Filmmaking Day 6: Using AI for Dubbing, Closed Caption, Subtitles & Transcription

Today was a pretty exciting and full day of learning all about dubbing, transcribing, close captioning and subtitling. If you don't know the difference between closed caption and subtitles, here is a great article to read.

But learning about all of that was not the exciting part. What made the day exciting was playing with AI and learning how to use artificial intelligence to automate all of it.

I started by learning about HeyGen's latest tool release, Video Translator. It's a very simple tool that allows you to upload a video, pick a language and then it will actually not only translate the audio to that language, but it will also lip sync the actor to that language you chose and it will use the actor's voice from the original video for the new language, including intonations.

The HeyGen "Video Translate" app allows you to upload a video, then pick a language to dub it to.

Watch the video below. It's a crazy thing to watch. It starts out in English (the original version), followed by the Hindi translation and finally the Spanish translation.

I then took this video into Adobe Premiere Pro and used the "transcribe" AI powered tool built into Adobe Premiere Pro to transcribe all three videos separately (each of them in their own sequence).

Video Transcription Generated by AI in Premiere Pro

Then, once each of the videos were transcribed, I generated closed caption 708 for each of those videos (again, using AI).

Creating Closed Captions in Premiere Pro is very easy!

Finally, I combined all three videos into one sequence.

Premiere Pro sequence with CC track and all dubbed videos.

Then exported that sequence and closed caption content as an .mp4 file with a sidecard .srt file.

Premiere Pro export settings for closed caption.

Here are the final exported files in my Finder window.

Exported .mp4 file with sidecar .srt CC file.

Then I uploaded both the .mp4 file and the .srt file into YouTube.

The CC Editor in YouTube was auto populated by the .srt file.

Here is the final video, dubbed and with Closed Captions. Make sure you click on the "CC" button to turn on Closed Captions or you won't see them.

I didn't perform QA on the final version, as I was only focusing on the process for this workflow, so you may find some issues with the video above with the transcription and / or dubbing. But the technology is improving at a rapid pace.

The future looks so bright for us filmmakers, with all this technology helping us do things that, at one time, only big studios could only afford to do.

Other things I worked on

• Attended the Hollywood Post Production Workflows webinar put on by Adobe with the editors of the movie Everything Everywhere All At Once. It was great seeing their process and how they used Premiere Pro to edit a major motion picture.

• Had a fantastic meeting with Lawrence Jordan, a film and TV editor with over 40 years of experience, who is also the founder of Master the Workflows. He will end up being on my podcast to chat all about which NLE we should pick and focus on.

• My short, The Adversarial, was screened at the Bay Area Cinema and I was invited (over zoom) to chat about it.

Until the next entry!