Creating a look-a-like digital deepfake avatar

Hier vind je honderden artikelen geschreven door Jarno Duursma. Meer weten? Abonneer je dan op zijn AI-nieuwsbrief. 27.000 mensen gingen je voor.
Mis niks en blijf op de hoogte van inzichten en actualiteiten. Abonneer je hier.

Digital deepfake avatar

For people who don’t yet have a clear picture of what I do professionally – I research technological trends and write books and blogs about them and give presentations. For some serious time now, I have been deepening my knowledge of artificial intelligence and related developments. This is a  very interesting area. For the last couple of years, this has also included the development of synthetic media, machines with imagination, deepfakes and GAN technology.

In 2017, I wrote the book ‘The Digital Butler’. In 2019, I compiled the report: ‘Machines with imagination, an artificial reality’ and also wrote a report: Deepfake technology: The Infocalypse. For English visitors, unfortunately, the book and reports are in Dutch only.

Short summary

TL;DR: the short summary: I’ve created a deepfake avatar of myself because of my interest in synthetic media and new technology. And because I want to investigate how I can digitize myself. And I want to know what the advantages and disadvantages are. For me, for companies and for society as a whole.

PS: a small update:

Just for the context: by doing this research I am not saying that deepfake avatars should be used everywhere. That they should, where possible, replace human interaction. On the contrary: I am an advocate of human contact. Rather, I am investigating where, with the help of this technology, we can make our lives easier. For instance, how information becomes available to people for whom it is now accessible. And how we can prevent some professionals from having to tell others the same thing 25 times a day.

What are synthetic media?

Synthetic media are all types of media, such as photos, videos, written text or audio clips, which have been created or modified by artificially intelligent (AI) software. This type of AI-software can create realistic media or modify existing media in such a way that completely new media are created.

If you are interested, you can read my blog about the future of synthetic media here. This podcast, in Dutch is compact and clear.

Synthesia

When I was researching synthetic media for my report, I came across the company Synthesia.  I was impressed by the digital avatars on their website.

They offer various deepfake avatar templates on their website which enable you to make a video within a short period of time. They are very accessible and the quality is convincing. You can create your own avatar by typing text in the input field. The software creates a video. The text is converted into a video of a digital presenter. The facial expression and lip movements in the video coordinate with the words spoken. It’s magical to see. The appearance of the avatar is very convincing in particular. However, the synthetic voice can still be distinguished from a real, warm human voice.

You can use these kinds of avatars to generate company presentations, create instructional videos for a new software tool or for internal communication. Creating a video is as easy as writing an e-mail.

And in my article on synthetic media I write the following:

“Wherever we obtain information in the future, there might be deepfake avatars. They tell us the departure times of the bus, train or plane. And they can read us the latest news 24 hours a day, both online and on television. Breaking news can be delivered by these voice robots. At any time of the day, 24/7, they read us the weather forecast, take our orders from fast food restaurants, and even do a short intake in some places. It doesn’t matter where you are as a consumer: on a laptop in your chair at home, on your smartphone, or at the location itself”.

Synthesia Avatar

I contacted the director of Syntesia, the company that creates these digital avatars, Victor. We are both excited about the future of synthetic media. We quickly agreed to work together on this topic.

I was particularly interested in my own digital deepfake clone. Templates are of course useful for everyday applications, but as a speaker I am particularly interested in digitising myself. I eventually want to be able to give a lecture in several places at the same time.

It was with great interest that I read about the green screen avatar which you can use to make a digital copy of yourself.

And how does that work?

For the green screen avatar, it is important that you make a recording in a studio with a green screen. 😉 Luckily, my good friend Ismael Lotz has a mobile studio with a green screen which was put up in my office for an afternoon.

To obtain a correct recording file, you need to read a script in front of the camera. In order to achieve a good result, it is important to have a high definition recording with professional lighting. As a speaker, it is vital to read out the script flawlessly during the recording, keep your head straight and (within these constraints) come across as enthusiastic. That is pretty difficult. If you are planning to make a green screen recording, I can highly recommend Ismael to you. And another tip: work with an auto cue – that makes the result much better.

When the recordings were finished, the file was sent to Synthesia. And then, frankly, the long waiting started. This software is so incredibly popular, that there is a six-week queue. The recording has to be processed and synchronised with the software. (costs $1.000). (The associated subscription is a few hundred euros per month, depending on how you are going to use the software. )

Avatar Finished

In early December, I received a message in my inbox that the avatar was ready. And I must say, I was genuinely surprised by the good result. Amazing to see. A convincing representation, clone, copy of myself.

And since then, I’ve been experimenting quite actively regarding what works and what doesn’t.

Since I have selected the corporate plan, I also have the option of uploading my own voice, which is then synchronised with the avatar. So, I went to my studio the same day to use my professional equipment to make a voice recording which could be used for the video of my Avatar. Long story short: I was satisfied with the final result, but I wasn’t overwhelmed by it.

After some experimentation, it turns out that the synchronisation of the lips and facial expression works better with written text. So, I have opted for the synthetic voice for now. Another advantage is that it also saves a great deal of time; typing is faster than having to make a voice recording.

So, once again, I have chosen a synthetic voice for this first video. You can choose from several voices, by the way. It just depends on the atmosphere you want to create in the video. And it is also possible to add text in other languages. The website says it supports 39 languages. It is important that you know how to write these languages correctly though. The software does not offer a translation software machine. (For the Dutch readers of this blog: there is indeed a synthetic voice in Dutch as well. I’m going to try it out soon.)

And once the text has been written, you can also choose different backgrounds, music and positioning -right, left, centred- in the video. (For your (dashboard with your) own avatar you do need a corporate Synthesia subscription. The price depends on a variety of parameters. So you’ re going to have to discuss this with them.)

Result: my objective opinion

The result of the final video is quite convincing. In terms of outward appearance there is hardly any difference between myself (flesh and blood) and the digital clone copy. As for the synthetic voice, it’s good, it does the job very well, but to be 100% honest, it’s not yet human enough, not yet warm and varied enough.

The voice is most convincing in the English language. Occasionally the accent is inaccurate, but that’s the only thing.

Some small remarks (Victor, if you’ re reading this). A small downside is that the software is not yet able to save your project temporarily. So, once you start, you have to finish the video. Also, the software doesn’t offer a preview function for you to check if you’re on the right track. Also, it is not yet possible to clone your own voice. This will probably become a feature in the future.

Conclusion

This is an incredibly interesting technological development. A new wave of creative digital innovation is coming. Machines with imagination. It feels like when I was listening to House music on Dutch radio for the first time in 1992. The excitement!

It’s relatively easy to make a video with your own digital copy. Your digital other half.

The ability to make an infinite number of videos by simply writing text alone is amazing. The resemblance to my human self is incredible.

I see myself making a presentation for a client with this avatar. A small footnote: that could be something like a 10-minute lecture; a kind of TEDx lecture. Are you interested in this? Let me know. I can also offer a 1-minute teaser to companies that book me as a speaker, which they can send to (potential) visitors to their event or conference.

I do think that listening to a synthetic voice for more than 15 minutes is going to be a bit monotonous. It’s more pleasant to watch and listen to a speaker in the flesh. In the next experiment, I will upload my voice and synchronise it with the avatar.

Deepfake avatar video’s

For many forms of communication, video will become the norm. This digital deepfake process will save me and many others having to manage and rent a studio, actors, camera, microphone, et cetera. If the purpose of the video is to transfer information, which it often is, then this is an ideal tool.

Do you know of any other options? Let me know. Or do you have an idea for a cool video, or would you never do something like this? Let me know via the contact form.

Ps: A question I get a lot these days: “Jarno, Everything digitizes, even now presenters: don’t you find that creepy?” Actually not: I’m looking for a way to digitize myself. However, I am certainly not blind to the dark side. Last year I wrote an extensive report about the risks and dangers of deepfake technology.

You can subscribe to my newsletter ‘Trending in Tech’ (in Dutch) here.

Connect with me on LinkedIn here and on Twitter here