modular mod idea: real-time talking

[Apos]

Essentially, this mod would work standalone, but it would also be designed to be bundled into any other multiplayer game.

Essentially, the idea is to make use of both the facial expressions and lip-syncing aspect of the new models in HL2, as well as any other special expressive animation.

The mod would allow people to gather in small conferance maps and talk face to face. This could be useful for any online mod: a chance to talk in a neat new way before or after a game. Teams could work out play by play strategy for the future, discuss what happened in the last match, or just mill around demonstrating techniques and ideas.

And the key would be that it would use the audio feature: players could talk into their mikes at home, and their characters would speak out what they are saying (perfectly lipsynched thanks to the new Source Engine). They could even control their own facial expressions to convey emotion to other players.

In fact, I kind of hope Valve is already planning this: it would be incredible not just from a practical value, but just as an experiment in online community. No 3d gameworld has this potential for vocal and physical expression in avatars. HL2 sounds like it will make this not only possible, but even pretty easy to implement.

Any other ideas or interests?

 

[neva]

"perfectly lipsynched thanks to the new Source Engine"

How can that work ?? It sounds impos. to me..

BUT it sounds really cool.. :cheese:

 

[Zerimski]

There's been no mention of voice comms being perfectly lip synched, as doing that kind of processing and animating on the fly would take an immense amount of power. However, the engine can lip synch perfectly with .wav files.

 

[Apos]

There's no fundamental difference: it's still just a waveform. If it can dynamically render .wav files on the fly (as they seem to be saying: they said you could just drop in a .wav file and the engine would do the rest: that sure doesn't sound like the engine pre-renders all possible mouth animation before each level), then it can easily be made to dynamically render any audio waveform on the fly.

 

[SilentKilla]

sounds awesome, and you can actully do this? Or your just sayin it would be cool. And how would we show facial expressions? By pressing buttons? lol

 

[Apos]

Whether it can be done depends on where the code is, and what they publically release. If the facial animation and sounds require game engine code, and that's still closed, it could be hard. But if the relevant parts are available in the SDK, then it could actually be incredibly easy to do. We'll just have to wait and see.

 

[Pajama Sam]

yea, im sure its possible bc Valve will give the community every scrap of code they can w/o ruining the entire engine. Desert Crisis has something like this now, where the mouth opens depending on what you are saying, although it is volume based

 

[Apos]

Mostly it's the facial animation that matters: the lip synching is just secondary.

 

[SidewinderX]

it would probery kill a hard drive.

first the player has to record the spoken audio as a .wav file, then it has to be played to show the lipsyning.

 

[D|2AGON]

I agree with Zerimski on this...I dont think it is possible to lip sync with a mic, HL2 just is n'sync (i hate those guys) with the previously recorded wav files.
This is my take on it although I could be wrong.

 

[Apos]

No.

HL already has voice-talk in multiplayer, and it works fine even on my crappy dial-up and ancient system.
HL2 already has real-time waveform parsing for the lip-synching (they use .wav files, but the waveform is all that matters, not the particular file format it's in).
The only question is whether these things can be made to work together.

Same thing with the animations. It's not impossible or even all that taxing on systems to do it. The question is simply whether the particular way they've implemented it in the engine will permit these sorts of tweaks.

 

[Sirius]

Sounds like a great idea Apos, but without further knowledge of how the engine deals with lip-syncing, we'll have no definite answers.

I would say that at this point, no, it won't be possible. Here's why. Each player is supposed to have 40 "muscles" in the face to give them the desired expressions. Now let's say you have a small server, but keep in mind there could be many more, of 10 people. That's 400 "muscles" that have to be transferred via packets to all the clients in order for them to see the expressions. Seems like a bit much for bandwidth usage. Not to mention any voice recorded in a mic has to be saved in the .wav format on the clients' hard drive, then transferred to the server for distributing. Again, that's going to eat loads of bandwidth.

The idea has merit, but it doesn't look practical at this moment. However, once we find out more details on the interaction between voice and muscle movement we may be able to find a way to implement such a feature.

 

[2ltben]

If we have the proper tools this can be done. If you say otherwise, then may I advise you to look at Natural Selection.

 

[Sirius]

Natural Selection may include voice comms, but not real time lip-sync abilities. That's the issue here, not whether you can create a "holding" room for players to chat.

 

[Apos]

Simple solution, and how I'm sure it's already set up: the actual "muscles" stuff would be rendered CLIENT side (like all the no-gameplay affecting physical effects). What has to be sent are just a general code for particular elements per character: the client does the rest. So, at best, you have only 10-20 more client signals coming at any one time.

As far as the .wav file, I really doubt this is an issue. In the specs, it says that the engine analyzes the waveform. .wav is simply the particular file format they've been using. The real problem would be if the waveform analysis pre-done, or is done when the level is loaded or something: anything but real-time.

 

[KiNG]

i think i can build on this idea..

what if they did lip synic and only epople in your sphere of infulence(30 metters could hear u and the farther u go away the less u hear them and server admins could make it like it is now like the could make it hear all enemies included or just team talk.
oh and by the way if i played cs then u would know if u looked at models that the lip synic pretty well, when u yell the mouth opoens wide and stays that way for the entire time oyour yelling, and it work pretty good, take a look next time your in a room with a friend

 

[Apos]

Yes, I definately would want to have the sound fade with distance, like any other sound in the game. Definately want two people to be able to go off away from others and have an isolated conversation.

 

[SpoBo]

I think it's already included in the full game .. knowing valve they'll show off with their new engine by including voice communication ... I saw the movie from the GMan and I think it was generated on the fly so ... And in HL now .. the succes of voice communication is SO huge they'll HAVE to include it ..

 

[ComradeBadger]

madnees, although it sounds cool, it will be forgotten soon enuff

 

[2ltben]

Originally posted by Sirius
Natural Selection may include voice comms, but not real time lip-sync abilities. That's the issue here, not whether you can create a "holding" room for players to chat.

I was referring to the RTS aspect of NS:cheese:

 

[smilez]

This is an excellent idea.. even if the lip syncing doesnt work to start with.

Hearing enemies would be good because then people could whisper if they suspect someone is near... it adds a new element to the game

:bounce:

 

[Fisheye]

I think HL2 will support voice communication like they did with the original game. Besides, if you communicate, in for example Counter-Strike, the lips move to.

-FishY

 

[bonanzaguy]

yeah. i was just about to say that in CS the mouths move somewhat along with the text that people say over the mic (which is one of the funniest things to watch)

and we also know that Valve created a way to lipsinc the characters mouths with text they input (read that in an article somewhere)

so combining the two wouldnt be a terribly difficult task i would think

 

[derby]

A lot of people are making wild claims that they have no ability to back up.

HL2 already has real-time waveform parsing

This has never been said by Valve or anyone who has seen the game. Half-life 2 has waveform parsing. We don't know if it needs the whole wavform in advance to determine mouth movements. It's highly unlikely that it will work in real time. Play a DivX movie that has the sound 1/10th of a second out of synch. It now looks like the film has been dubbed. The lip synch process is complicated, the chances are it will take a litte bit of prcessing time especially when the game is doing a lot of other computations.

What is likely is that the engine will only need to see each syllablel of a wav before it can calculate the mouth position. You could say yor wavemform, the game would start processesing it then as soon as each bit was done it wold be sent to the server. A player wouldn't need to hear his own speach, and no-one else would know it was delayed by 1/2 a second.

Each player is supposed to have 40 "muscles" in the face to give them the desired expressions. Now let's say you have a small server, but keep in mind there could be many more, of 10 people. That's 400 "muscles" that have to be transferred via packets to all the clients in order for them to see the expressions. Seems like a bit much for bandwidth usage. Not to mention any voice recorded in a mic has to be saved in the .wav format on the clients' hard drive, then transferred to the server for distributing.

There was a conversation with someone from valve regarding extra bandwidth required for the new physics. He said that as much as possible is done server side, each person's client PC (the one they are playing on) calculates where each one of the 50 pieces you smash something into will land. The same technique would be used here. There are a limited number of general expressions to use, the server only needs to know that someone is frowning, the clients can move the 40 muscles. And the last part? Do you know anything about computers, why would the .wav file need to be saved on the harddrive? It's likely to be less than a megabyte, it can be cached in main memory before being sent to the server. This works for all the games right now that use voice communication for online games.

In summary, the mod seems possible given the current information we have about the source engine. I have made assmptions here, but they are reasonable and I can back them up.