ࡱ> `! E4 JDm>F .W xX{tTE]Unt:N#Mr !vHxnp$kF (a80d"ayfGP!K.̙?9RW_U\M#L2q.9$40Uz%f0vr0݁ b":C$hYRES30R)Kt]Cd.׊%RQTQt1P!Q`@w&1IbfrtxWuzkbg$ɯ{Suܨo.Flbz6R%=C>w>:Y;/~\;*ܭ0M;&SW?Ȅnk>I2nЗ94(ΕFx_ 7F5m0߉}# ~ ?w7gy&*kx_zs!Ìޅk6 _| 8s~>h=lC5i:]5DKEY6 B+=SR|.W>o ~/BM+Q'|Rt xsj| N5@Ƽ+tNhW`7 wQ3U[RN[Z#Ļ=Zr&bfL;bZ ]; 8o읪sv$rc" *.u^AE:.뚈.Rb:RpLK(tv?-i>Z$xXKV" 5 !ҹA8Bm(puMx6Z O oOL+s۵"7wk G9jsYo\-swװȍ~o &_sBkw٥ 1C| 8;Nh&| xcF{uNh |n?Dlaos M:3-tJjtA\wxBtc@}5 tNh{6x1 ]B/X7絸^ /4_CW  MZ|w7{Up69᫁Opq]O/$~ǿCһճZjgr`d%uTF/#L-1~=fotqKrZMdmS͸ڀ]xz0Ht\L$-;d%)?a (c,d :۲\XR)=W[t?)h,,=KWYV:U\ R!RgC3JΨ` u Ylaz;svYN{-GةF63M`9lʼVi|1@2ϣh?s<ȼTd.*Y%V'm 6 7` M,F-HMVGfI!4&˘\-hz6ldN$mG=Iȗjel/enϐ*o.{ WQNegdjvT9r BO(XZn߉>CrM^dze31HݜFms3veX*r_Gx,tdX SM\^ro}пcW&r3zey*tg z'C/[q7|IvxG'e:=p"ʽ *{* JϐqW w ߻[aW;f:OGتFV]19r؋hZcb\A5^:yw*\媳rq.$=N{6:,pSg%},so%Kܕ=Lq{wyWnro8ՌZqP潩lΫA<eڂ{ּ 3s7{ٔ-XyJ$Dl\՟ ݭ >u3+S׳&64&*(  8 $ex4.html/ 0DTimes New Roman$b$b0b0TbTb 0DMS LineDrawman$b$b0b0TbTb 01 DArialeDrawman$b$b0b0TbTb 0"@HH    @n?" dd@  @@``  3   .%  ,*(  0&'.'$%""    ,2$E4 JDm zS ~1? z3Oa@g4 d d`b 0Xb:ppp@ g4KdKd`b 0Xbppp@  <4BdBd$b$bP  <4!d!d$b$b4?P-$November 4th, 1997 ICAD '97O=$November 1st, 1996dActiveMovie Audio and Video Capture Design Preview ,Microsoft ConfidentialDirectAnimation; A behavioral approach to multimedia Ken Greenebaum kgreene@microsoft.com Internet Client and collaboration Division, Multimedia Microsoft CorporationH Z$$$N$$? S 1Materials onlineThis presentation HTML version of paper with working DA examples Links to DA materials DA is included free in IE4.0 http://www.research.microsoft.com/~kgreene/ New approach to MultimediaWe are doing something different Contrast conventional approaches Then cover the new concepts Emphasize audio examples for ICAD+ Traditional Media Libraries$Been there, done that SGI s GL, AL, IL, VL, CL Microsoft s D3D. Dsound, Ddraw Monolithic Powerful, but difficult to integrate Imperative Examples:Z8 %8 %>%F,DirectAnimation mMedia Integration Audio, Video, 2D, 3D Implicit Timeline and Synchronization Description based Rich behaviors6GG- DA Behaviors~Dynamic Value wrt time time or other behaviors Behaviors belong to fundamental types Fundamental types have dynamic parametersn4Primitive BehaviorsdBuilding blocks System constant Examples: Audio : Silence Image : EmptyImage Color : Red Number : 1$*;*;C 8DA Behavior TypesPrimary Media Types: Sound, Image, 3D geometry Supporting behaviors: Number, String, Color, Point, Vector, Transform, Camera, Microphone, Font, Line, Spline!HYY;Behaviors are abstractSound type actually represents: Sampled PCM sound Networked streaming sound Parametrically synthesized sound MIDI Soon: Dynamically composed music$ s sFEx: primative, static behavior 0 800 Hz sine wave rawSound = sinSynth.rate(800);$, GEx: Dynamic behavioroSiren based on previous example rawSound = sinSynth.rate(800); siren = rawSound.rate(div(add(2, sin(time)), 2);$ P P>  !X$Ex: Recursive MixDemo =Media integrationCombine behaviors for richness Primitive behavior yield time varying behaviors Time varying combine to form hybrid behaviors Hybrid behaviors combine to form animations$@ Single engine controls all media! ,SExtreme richness Implicit synchronization Examples: Steam locomotive Sea Shore demo$4 4 IAudio CapabilitiesMedia importation via DirectShow Many native PCM audio formats wav, aifc, au, etc. Many audio CODECs supported G711, uLaw, MPEG, voxware, H.* coming via NetShow Users may add formats via Dshow filtersl!3(!3(  J. Audio Capabilities cont.Mixing Audio of different types (MIDI and PCM) Auto format conversion of Audio of different formats Multiple instantiation Import a sound once then play it simultaneously at different positions or rates Recursive mixing of soundsH]k]kL Audio Capabilities cont.eSpatial Audio Embed sounds in 3D geometry 3D transform sound Render sound via 3D oriented microphones$XXT"Present Audio ParametersAll Sounds (PCM, Streaming, SinSynth, MIDI) support: Gain Rate Phase Pan Loop (filters/synthesizers coming which will have more elaborate controls)65F5FpU#Sound Behavior ConstructorsMix SoundSource(soundBvr) Union, transform (on sound geometries) SoundGeometry.RenderSound(microphones) Coming: numberBvr.SoundSynth(timeInterval)       /DA Engine Exposed via comUAvailable from wide range of languages: JavaScript VBScript Java Visual Basic C, C++ $(.(.*30TU*  Future workAudio Parametric synthesis Dynamic music synthesis Additional network streaming of audio/movies Engine Enhanced retained mode control More sophisticated optimizationHZ?Z? ConclusionDescription based behavioral approach to multimedia Easy to construct media rich, synchronized animation High degree of integration Web based or stand alone animationsM!We are hiring! ) Audio AnecdotesCookbook in the spirit of Graphics Gems Soliciting Articles now! Topics and details at http://www.sworks.com/cookbook/solicitation.html $W2W2b!/'23567:<>AEHKNO P!Q"R#S$V%W&Y'PbI  ` ̙33` 3` 3333f` 999MMM` f` f3` 3>?" dX@0z?KX\@ H5dddH5KX`H5dd\d ?" dX(F 5d!!5X(CC5dddPR    @ ` ` p>> sk(    T vgֳgֳ ?:` v ? Slide Title  #  T!vgֳgֳ ?6 v ;Body Text Second Level Third Level Fourth Level Fifth Level     <  Td!vgֳgֳ ?D`d v ?*  T!vgֳgֳ ?Dd  v A*  T$"vgֳgֳ ?DK d v A*`  Hd޽h @? ? raa  panel 0 @O(    TT6jJjJ ?O  v y* F!!CCdd  T6jJjJ ?  v {* F!!CCdd  Z7jJjJ ?O  v y* F!!CCdd  Zt7jJjJ ?  v {* F!!CCdd  T7UU?8 ' lPage *Z  )&&99LLp  01 ?$1  v#  T48gֳgֳ ? 7 v ;Body Text Second Level Third Level Fourth Level Fifth Level     <H  0޽h/.? ? a(. 0 ( X    T4jJjJ ?O  v ]* F!!CCdd  T45jJjJ ?  v _* F!!CCdd  Z5jJjJ ?O  v ]* F!!CCdd  Z5jJjJ ?  v _* F!!CCddj  BA?ZU $d  <1?9 d  <1? 9H  0޽h/.? ? a( bZ ( @.    fgֳgֳ ?60P  H  0޽h ? raa  ( -@ l  C :`   l  C 64   H  0޽h ? raa h $ ( ֳ     fgֳgֳ ?:`      fdgֳgֳ ?6^   H  0޽h ? raa  (  l  C :`   l  C D6   H  0޽h ? raa  ( zez l  C d:`   l  C 6   H  0޽h ? raa  ( ` l  C $:`   l  C 6  H  0޽h ? raa  `( H,$ l  C :`   l  C 6   H  0޽h ? raa  @$( ` $l $ C d+:`   l $ C 46{   H $ 0޽h ? raa   4( ` 4l 4 C D*:`   l 4 C *6   H 4 0޽h ? raa ! (  l  C $):`   l  C )6  H  0޽h ? raa  ( ` l  C ':`   l  C (6  H  0޽h ? raa ' (  l  C %:`   l  C $&6  H  0޽h ? raa  D( ` Dl D C #:`   l D C D$6G  H D 0޽h ? raa  d(  dl d C ":=   l d C $#6   H d 0޽h ? raa  `( -@ l  C  :`   l  C D!6   H  0޽h ? raa " @( ` l  C T :`   l  C  6   H  0޽h ? raa #  (  l  C  :`   l  C 4 6G  H  0޽h ? raa % ( ` l  C :`   l  C T6S  H  0޽h ? raa & ( ` l  C :`   l  C 64   H  0޽h ? raa  ( ` l  C 4:`   l  C 6   H  0޽h ? raa $ (  l  C :`   l  C 67   H  0޽h ? raa #a $(     fgֳgֳ ?:`      fgֳgֳ ?6^  v H  0޽h ? raa  `( ` l  C v:`   l  C 6~  H  0޽h ? raa    ( ` l  C 3:`  v l  C 46  v H  0޽h ? raa 0 0e(     H1 ?#1      f$gֳgֳ ? 7   Hi, I am Ken Greenebaum. I am very excited to be presenting this paper today, talking to you about description based media and the DirectAnimation system which implements it. When I joined Microsoft two years ago we were a small team in research working on a prototype. A lot has changed since then. Today the first version of the DirectAnimation system has released and is literally in the hands of Millions of users.,IH  0޽h/. ? a( 0 $,(  $ $  H1 ?#1   d $ C  7   In Directnimation we are doing something rather different. Before I introduce the new concepts lets examine conventional media libraries. Since I don t have too long to speak, and have a lot of concepts to cover I am going to forego some of the demos I would have liked to have shown. The demos are online, and if you go to my web page I have provided links. I encourage you to try them yourselves and use your browser s show source option to examine the implementation.,)H $ 0޽h/. ? a( 0 kc(       H1 ?#1   3   fTgֳgֳ ? 7   =In conclusion DirectAnimation offers a novel description based animation system. Behaviors provide a natural way for people to describe and construct rich dynamic animation. The description approach allows for interaction and dynamics while greatly reducing media download times over movies or even other animation systems. Direct animation allows very sophisticated animations to be incorporated into web content. It may also be used in stand alone C based applications. DirectAnimation is available now, free of charge, and is already installed on millions of machines.,UH  0޽h/. ? a(} 0 =5( -@ R  3 #1   ;  C  7   Behaviors are the key to the dynamics of direct animation. Behaviors are values which are defined with respect to time or other behaviors. This provides implicit synchronization and great richness. Sound is one of the fundamental media types and is not treated as a second class citizen as it is in other multimedia systems. Sound parameters currently include rate, phase, gain and pan. Each of these parameters could be dynamic behaviors themselves.H  0޽h/. ? a(  0 | t  (  R  3 #1   z  C  7   DirectAnimation offers a very high degree of media integration between audio, video, 2D, and 3D graphics primitives. These media types are manipulated in similar ways and are often interchangeable. All dynamic behaviors are synchronized via a common timeline. Behaviors are often defined with respect to time or in terms of other behaviors. In direct animation the desired results are described to the engine whose job it is to perform the necessary tasks. For example orbit a 3D space ship with an embedded sound around a planet at 1000 meters per second. The user doesn t need to make calculations or worry about frame rate but rather describes the desired results. The engine having complete knowledge of the non side effecting model is free to make profound optimizations automatically such as level of detail for the spaceship model, culling the sound when it doesn t audibly appear in the mix, and selectively redrawing only the portion of the background or planet overwritten by the spaceship model.H  0޽h/. ? a( 0 p&(  R  3 #1     C d 7   4 Primitive Behaviors are constant, meaning they are not time varying. They may be used alone, or more often form the building blocks for more complicated composite behaviors. A number of system constants exist for each media type. We have silent sounds, empty images, colors and numbersH  0޽h/. ? a(x 0 80(  R  3 #1   6  C  7   |I have provided links on my web page to a number of resources that may be interesting: these power point slides, an html version of this paper which includes actual working DirectAnimation examples if you are running IE4, and links to DirectAnimation documentation. The DirectAnimation engine is included in the IE4 base install and is available for free. If you are running IE4 already then you have DA and probably didn t even know it! l>0H  0޽h/. ? a(  0 |(  R  3 #1     C  7   "I have worked in industry for a while and helped create some of the libraries on this slide. They represent some of the most popular and powerful ways to manipulate individual capabilities of graphics, audio and video. However I can also tell you first hand that they are very difficult to integrate and use together even for the people who developed them. These libraries are also mainly Imperative. The programmer must provide step by step instructions to get a desired result. Usually this takes place in a frame loop.H  0޽h/. ? a( 0 ]UP0( ` 0R 0 3 #1   [ 0 C  7   Sound, image, and 3D geometry represent the primary media types DirectAnimation works with. Many other types exist to support these primary types. For instance number behaviors are used to make the audio rate, phase, and pan parameters dynamic. Transform behaviors can cause sounds embedded in geometry to move in 3 space, maybe along a 3D spline. And Microphone behaviors may be oriented in space and used to render 3D spatial sound.,@ZH 0 0޽h/. ? a(} 0 =50<( ` <R < 3 #1   ; < C + 7   DirectAnimation media types are abstract and represent a group of related capabilities. For instance the DirectAnimation Sound behavior currently represents sampled sound, streaming networked sound, parametricaly synthesized sound, and MIDI. What is significant is that any of these sounds may be used interchangeably. They all support the common rate, gain, phase and pan sound parameters.6ZNH < 0޽h/. ? a( 0 WOL( ` LR L 3 #1   U L C % 7   Media integration through use of hybrid behaviors is key to the DirectAnimation philosophy. Animations are constructed from hybrid behaviors, those are constructed from time varying behaviors, and those in turn are made up of primitive behaviors. In this way simple building blocks are put together to form increasingly more sophisticated results. This corresponds to the way people naturally describe complex phenomena. Consider animating a spider who climbs up and down a pendulum which is in turn in motion. You could mathematically determine where in space the spider should be rendered, as would be required using conventional tools. Or you could create the simple up and down the pendulum behavior which locates the spider with respect to time, then create a new behavior which swings the pendulum. We could easily continue this indefinitely placing the pendulum in an airplane flying around a planet orbiting a star. 6@ FH L 0޽h/. ? a(R  0   l ( ` lR l 3 #1     l C # 7   hTaking the concept of media integration one step further we can remember James Hahn s Timbre Trees work. Basically using one engine to control all aspects of an animation. Consider animating a steam train. In DirectAnimation you could easily create a behavior modeling the physics of the train: its power, mass, etc. This behavior could in turn control the elements of the animation. The train itself, the steam, the piston and drive shaft, and of course the chug chug sound. Then as the train accelerates from a stop the rate of the choo choo sound would automatically be proportional to the speed of the train and be synchronized to the puff of steam, and reciprocating piston. As the train hit an incline the timbre of chug sound would change corresponding to the change in load. Animations such as these are straight forward to create. Last year I demonstrated a beach animation where a weather model controlled the sound of the wind and the surf. The animated Palm trees swayed to the degree dictated by the wind. The wind vane spun in the direction of the wind.,6 H l 0޽h/. ? a( 0 kc( ` R  3 #1   i  C T 7   The DirectAnimation api is exposed via com. This allows it to be used by a very wide range of languages and applications. The scripting languages are especially convenient allowing DirectAnimations to be embedded directly in html files via the script tag. If you visit the web version of my paper online you will see working DirectAnimation embedded directly in the text in much the same way as other web pages have static images.>ZH  0޽h/. ? a( 0 H( ` R  3 #1     C d( 7   VHere we use rawSound, the 800Hz sine wave behavior from the previous example, as the basis for a siren. We ask for rawSound s rate to vary from one half to one and a half times it s nominal 800Hz frequency every second. If we had more time we could next embed the siren in the geometry of a moving police car and spatialize the resulting moving siren via 3D positioned microphones.> `  :H  0޽h/. ? a( 0 p@( ` R  3 #1     C d" 7   NdNow that we have reviewed the basic concepts, we can explore some of DirectAnimation s current sound capabilities. I will talk a little bit about some upcoming audio features at the end of the talk. DIrectAnimation uses the DirectShow filter graph technology for all audio format importation. In this way DirectAnimation has access to the wide variety of file formats and CODECs supported by DirectShow. However it also allows DirectAnimation to be extensible. DirectAnimation is able to use any registered DirectShow filter automatically. End users may write or purchase Dshow filters. Or filters may automatically be downloaded by web based animations via the codebase mechanism.Du H3  7U  H  0޽h/. ? a(! 0 pP( -@ R  3 #1     C t 7   ^JI a going to sneak a couple of slides in! This one is self explanatory...H  0޽h/. ? a(  0 PD( j R  3 #1   v  C 9 7   R In case you missed my announcement Monday night. Audio Anecdotes is the working title for a book project I am involved with. Basically it is the audio cookbook spanning the audio field that I have always wanted but hasn t existed. The work is very much inspired by Andrew Glassner s Graphics Gems series. Please contact me if you would like to contribute articles or help in another fashion. Please pass this information on to your colleagues and students! Do we have time for demos? Thank You. Any questions? , H  0޽h/. ? a( 0 VN(  R  3 #1   T  C ) 7   This is an example of a primitive, static behavior called rawSound. rawSound is comprised of the sinSynth primative with a rate of 800 yeilding a 800 hertz sine behavior. We could have made the rate parameter time varying by describing it in terms of time. For instance a rate of time would create a sine ramp which would start at DC and would increase in frequency by one hertz a second. P:  H  0޽h/. ? a( 0 P^(  R  3 #1     C $  7   l6The audio mixer is central to DirectAnimation s audio features. Audio of different types such as MIDI and sampled PCM sound are handled. In addition format conversion is automatically performed allowing sounds recorded or synthesized at different sample rates and formats to be correctly mixed. Multiple instantiation allows a sound to be played back many times simultaneously at different positions and rates.lH  0޽h/. ? a(|  0 <40( [ R  3 #1   :  C   7   As mentioned earlier DirectAnimation allows sounds to be embedded in geometry, transformed spacialy, then rendered with respect to 3D microphones.,60H  0޽h/. ? a(  0 M(  R  3 #1     C  7   [GWe are now working on very exciting extensions to the current system. On the audio front we are introducing a powerful parametric synthesis capability which will allow all sorts of sounds such as collisions, scratching, turbulence, etc. to be synthesized. In this way the appropriate sound is played based on the actual materials, speed and duration of the interaction, and the user does not have to wait for large sound files to download across the network. Enhanced support for streaming audio will further slash download times. Work on the engine includes more advanced control of retained mode systems. This will allow much finer grained control of audio and video, ultimately allowing sample accurate control of parameters and start and stop times. More sophisticated optimizations should further increase animation performance beyond navely written code. Already automatic use of dirty rectangle techniques, static pruning, and others result in very favorable performance. However very carefully tuned single purpose applications such as doom may always outperform generalized systems.H  0޽h/. ? a(" 0 NF( ` R  3 #1   L  C t  7   All sound behaviors, no matter if they are actually sampled sound, streaming, the sine wave synthesizer, or MIDI, support gain, rate, phase and pan parameters. These parameters accumulate so that you can take a primitive sound, apply again of one half to it forming a new behavior, then gain one half that behavior yielding a new behavior with one quarter the gain of the original. Of course any of these parameters may be number behaviors themselves allowing the parameter to dynamically change. The number behavior itself could be an arbitrarily complex programmatic statement. Say modulating the volume of a sound based on the distance between two objects in the animation. Phase controls the local time offset of the sound. A simple chorus behavior might mix a primitive sound with itself phased a couple of milliseconds. The parametric synthesizers and filters we are now working on will of course still yield sound behaviors, but they will accept a richer set of parameters.H  0޽h/. ? a(# 0 c( j R  3 #1     C 4 7   qThere are a number of additional ways to construct sound behaviors. The most common is mix, which takes two sound behaviors, and creates the sum of them, a new behavior. More elaborate is SoundSource which turns a sound behavior into a geometry which can then be unioned to other geometry and geometricaly transformed. The resulting geometry may then be spatially rendered back into a sound behavior using a microphone. One fun capability I am working on would evaluate a number Behavior over a time interval synthesizing a sound of that duration. Since number behaviors may be arbitrarily complicated this is an extremely flexible way to add arbitrary synthesis. A similar mechanism could be used in 2 dimensions to create a synthetic texture, or three dimensions to create a geometry.> ? H  0޽h/. ? a($ 0 (( ` R  3 #1     C & 7   6Lets demo recursive mixing since some folk Sunday were concerned about mixing and layering sounds on a PC. This PC is actually my anemic 70 megahertz laptop. Every time I hit the mouse button another instance of the sound is mixed in. As you can hear I can mix quite a few. Lets look at the html for this web page. It includes the jscript calling DirectAnimation via com. Most of what you see is boiler plate. These three lines actually provide the recursive sound mix. >NmH  0޽h/. ? a(r03; .*AnE ip3t'z)P6ngHIK/f1C~OMqO:P=bQXg@ ZEP`BS"U\^b`Vl GBb"dLW pY(0t0(  8 $ex4.html/ 0DTimes New Roman$b$b0b0TbTb 0DMS LOh+'0 ,8 P\    Bluegray PowerPoint Templatet TemplatePowPresentations Group'Bluegray background, Gold text/bulletsluelueKen Greenebaumo110Microsoft PowerPoint 4.0 te@X"'X@`ƻ@(3Я@G GoM   <& &&#TNPPp0D z & TNPP &&TNPP    &&--&&- $--- $--ZZ- $ZZ- $- $- $- $;;- $;;hh- $hh- $- $- $- $II- $IIvv- $vv- $&&&- & $&&-&& &&-&&&&- $--- $--ZZ- $ZZ- $- $- $- $;;- $;;hh- $hh- $- $- $- $II- $IIvv- $vv- $&- --&&&G& wwgw - Times New Romanwgw - ."2 RNovember 4th, 1997     .&o&  .2 ICAD '97  .&a+&  . 2 1 .--}-- Times New Romanwgw - .2 DirectAnimation./#5  #. z.2 DirectAnimation./#5  #. . 2 b;. z. 2 `;.Times New Romanwgw - .<2 #A behavioral approach to multimedia       . z.<2 #A behavioral approach to multimedia       .Times New Romanwgw - .2 lKen Greenebaum% &(. z.2 jKen Greenebaum% &(.Times New Romanwgw - .2 kgreene@,. z.2 kgreene@,. .2 X microsoft&  . z.2 V microsoft&  . . 2 .com %. z. 2 .com %.Times New Romanwgw - .92 !Internet Client and collaboration #     . z.92 !Internet Client and collaboration #     . .%2 <Division, Multimedia#    - ( . z.%2 :Division, Multimedia#    - ( . .'2 pMicrosoft Corporation- # . z.'2 nMicrosoft Corporation- # .--"Systemw3f  -&TNPP &Root EntrydO)(V@PicturesCurrent User.SummaryInformation(}@HH    @n?" dd@  @@`` *"  3   .%  ,*(  0&'.'$%""    ,2$E4 JDm S ~1? z3Oa@g4)d)d 0Nppp@ g4KdKd 0.ppp@  <4BdBdP  <4!d!d4?P-$November 4th, 1997 ICAD '97O=$Nov &_bKen Greenebaumember 1st, 1996ICAD'97 2Ken Greenebaum, Microsoft&#DirectAnimation; A behavioral approach to multimedia Ken Greenebaum kgreene@microsoft.com Internet Client and collaboration Division, Multimedia Microsoft CorporationH Z$$$N$$? S 1Materials onlineThis presentation HTML version of paper with working DA examples Links to DA materials DA is included free in IE4.0 http://www.research.microsoft.com/~kgreene/ New approach to MultimediaWe are doing something different Contrast conventional approaches Then cover the new concepts Emphasize audio examples for ICAD+ Traditional Media Libraries$Been there, done that SGI s GL, AL, IL, VL, CL Microsoft s D3D. Dsound, Ddraw Monolithic Powerful, but difficult to integrate Imperative Examples:Z8 %8 %>%F,DirectAnimation Media Integration Audio, Video, 2D, 3D manipulated in similar ways/interchangeable Synchronization via common timeline Description based Up to the engine to decide how to render Non-side effecting Frees the engine to optimizelA6)A6)- DA BehaviorsThe key! Dynamic Value wrt time time or other behaviors Behaviors belong to fundamental types Sound is a fundamental type, not a second class citizen Fundamental types have dynamic parameters Rate, phase, gain and pan (each could bea bvr itself)H^8*6^8*6, 4Primitive BehaviorsBuilding blocks Constant, not time varying System constant Examples: Audio : Silence Image : EmptyImage Color : Red Number : 1$E;E;^ 8DA Behavior TypesPrimary Media Types: Sound, Image, 3D geometry Supporting behaviors: Number, String, Color, Point, Vector, Transform, Camera, Microphone, Font, Line, Spline! Transform sounds embedded in Geometry through space and spatially render using MicrophoneH\;Behaviors are abstractSound type actually represents: Sampled PCM sound Networked streaming audio Parametrically synthesized sound MIDI Soon: Dynamically composed music/more synthesis$  FEx: Primative, Static Behavior 0 800 Hz sine wave rawSound = sinSynth.rate(800);$, GEx: Dynamic behaviorSine ramp(DC then one hertz/second) ramp = sinSynth.rate(time); Siren based on existing behavior rawSound = sinSynth.rate(800); siren = rawSound.rate(div(add(2, sin(time)), 2); // rate will vary from 1/2 to 1 1  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|~/2 nominal 800 Hz sineH$!$!P* . YX$Ex: Recursive MixDemo =Media integration(Combine behaviors for richness Primitive behavior yield time varying behaviors Time varying combine to form hybrid behaviors Hybrid behaviors combine to form animations Spider climbing up swinging pendulum& Build step by step instead of defining where the spider is absolutelyH&F&F@PowerPoint Document(bDocumentSummaryInformation8ineDrawman$b$b0b0TbTb 01 DArialeDrawman$b$b0b0TbTb 0"@HH    @n?" dd@  @@`` *" 3   .%  ,*(  0&'.'$%""    ,2$E4 JDm zS ~1? z3Oa@g4)d)d`b 0XbNppp@ g4KdKd`b 0Xb ppp@  <4BdBd$b$bP  <4!d!d$b$b4?P-$November 4th, 1997 ICAD '97O=$November 1st, 1996dActiveMovie Audio and Video Capture Design Preview ,Microsoft Confidential&#DirectAnimation; A behavioral approach to multimedia Ken Greenebaum kgreene@microsoft.com Internet Client and collaboration Division, Multimedia Microsoft CorporationH Z$$$N$$? S 1Materials onlineThis presentation HTML version of paper with working DA examples Links to DA materials DA is included free in IE4.0 http://www.research.microsoft.com/~kgreene/ New approach to MultimediaWe are doing something different Contrast conventional approaches Then cover the new concepts Emphasize audio examples for ICAD+ Traditional Media Libraries$Been there, done that SGI s GL, AL, IL, VL, CL Microsoft s D3D. Dsound, Ddraw Monolithic Powerful, but difficult to integrate Imperative Examples:Z8 %8 %>%F,DirectAnimation Media Integration Audio, Video, 2D, 3D manipulated in similar ways/interchangeable Synchronization via common timeline Description based Up to the engine to decide how to render Non-side effecting Frees the engine to optimizelA6)A6)- DA BehaviorsThe key! Dynamic Value wrt time time or other behaviors Behaviors belong to fundamental types Sound is a fundamental type, not a second class citizen Fundamental types have dynamic parameters Rate, phase, gain and pan (each could bea bvr itself)H^8*6^8*6, 4Primitive BehaviorsBuilding blocks Constant, not time varying System constant Examples: Audio : Silence Image : EmptyImage Color : Red Number : 1$E;E;^ 8DA Behavior TypesPrimary Media Types: Sound, Image, 3D geometry Supporting behaviors: Number, String, Color, Point, Vector, Transform, Camera, Microphone, Font, Line, Spline! Transform sounds embedded in Geometry through space and spatially render using MicrophoneH\;Behaviors are abstractSound type actually represents: Sampled PCM sound Networked streaming audio Parametrically synthesized sound MIDI Soon: Dynamically composed music/more synthesis$  FEx: Primative, Static Behavior 0 800 Hz sine wave rawSound = sinSynth.rate(800);$, GEx: Dynamic behaviorSine ramp(DC then one hertz/second) ramp = sinSynth.rate(time); Siren based on existing behavior rawSound = sinSynth.rate(800); siren = rawSound.rate(div(add(2, sin(time)), 2); // rate will vary from 1/2 to 1 1/2 nominal 800 Hz sineH$!$!P* . YX$Ex: Recursive MixDemo =Media integration(Combine behaviors for richness Primitive behavior yield time varying behaviors Time varying combine to form hybrid behaviors Hybrid behaviors combine to form animations Spider climbing up swinging pendulum& Build step by step instead of defining where the spider is absolutelyH&F&F@ Single engine controls all media!!,James Hahn s Timbre tree Extreme richness Implicit synchronization Examples: Steam locomotive (steam, chug sound, location) Sea Shore demo$M>M>IAudio CapabilitiesMedia importation via DirectShow Many native PCM audio formats wav, aifc, au, etc. Many audio CODECs supported G711, uLaw, MPEG, voxware, H.* coming via NetShow Users may add formats via Dshow filters Filters may be codebasedl!3A!3A  J. Audio Capabilities cont.Mixing Audio of different types (MIDI and PCM) Auto format conversion of Audio of different formats Multiple instantiation Import a sound once then play it simultaneously at different positions or rates Recursive mixing of soundsH]k]kL Audio Capabilities cont.zSpatial Audio Embed sounds in 3D geometry Animate Sound Geometry via 3D transforms Render Sound Geometry using Microphones$mmT"Present Audio ParametersAll Sounds (PCM, Streaming, SinSynth, MIDI) support: Gain, Rate, Phase, Pan, Loop Accumulate Parameters may be number behaviors themselves (filters/synthesizers coming with more elaborate controls)J5t58;U#Sound Behavior ConstructorsMix SoundSource(soundBvr) Union, transform (on sound geometries) SoundGeometry.RenderSound(microphones) Coming: numberBvr.SoundSynth(timeInterval) Could also be used in 2D to create textures$,,       ./DA Engine Exposed via comvAvailable from wide range of languages: JavaScript, VBSript DA in html file using script tag Java Visual Basic C, C++ H(!(!*490uv*  Future workAudio Parametric synthesis Dynamic music synthesis Additional network streaming of audio/movies Engine Enhanced retained mode control More sophisticated optimizationHZ?Z? ConclusionDescription based behavioral approach to multimedia Easy to construct media rich, synchronized animation High degree of integration Web based or stand alone animationsM!We are hiring! ) Audio AnecdotesCookbook in the spirit of Graphics Gems Soliciting Articles now! Topics and details at http://www.sworks.com/cookbook/solicitation.html $W2W2b!/'23567:<>AEHKNO P!Q"R#S$V%W&Y'Pb  ( zez l  C d:`   l  C 6   H  0޽h ? raa  ( ` l  C $:`   l  C 6   H  0޽h ? raa  `( H,$ l  C :`   l  C 6^   H  0޽h ? raa  @$( ` $l $ C d+:`   l $ C 46   H $ 0޽h ? raa  ( ` l  C ':`   l  C (6   H  0޽h ? raa ' (  l  C %:`   l  C $&6  H  0޽h ? raa  D( ` Dl D C #:`   l D C D$6   H D 0޽h ? raa  d(  dl d C ":=   l d C $#6B   H d 0޽h ? raa  `( -@ l  C  :`   l  C D!6   H  0޽h ? raa #  (  l  C  :`   l  C 4 6G  H  0޽h ? raa % ( ` l  C :`   l  C T6)   H  0޽h ? raa & ( ` l  C :`   l  C 6w   H  0޽h ? raa  ( ` l  C 4:`   l  C 6   H  0޽h ? raarh , e/04E8%=$@&G!I(Le*T E,%.X" p1Yb$0(  8 $ex4.html/ 0DTimes New Roman0ܳܳ 0DMS LineDrawman0ܳܳ 01 DArialeDrawman0ܳܳ 0" Single engine controls all media!!,James Hahn s Timbre tree Extreme richness Implicit synchronization Examples: Steam locomotive (steam, chug sound, location) Sea Shore demo$M>M>IAudio CapabilitiesMedia՜.+,D՜.+,   *On-screen Showtb+j Times New Roman MS LineDrawArialpanelDirectAnimation; A behavioral approach to multimedia Ken Greenebaum kgreene@microsoft.com Internet Client and collaboration Division, Multimedia Microsoft CorporationMaterials onlineNew approach to MultimediaTraditional Media LibrariesDirectAnimation DA BehaviorsPrimitive BehaviorsDA Behavior TypesBehaviors are abstractEx: Primative, Static BehaviorEx: Dynamic behaviorEx: Recursive MixMedia integration!Single engine controls all mediaAudio Capabilities. Audio Capabilities cont.Audio Capabilities cont.Present Audio ParametersSound Behavior ConstructorsDA Engine Exposed via com Future work ConclusionWe are hiring!Audio Anecdotes  Fonts UsedDesign Template Slide Titles(RZ _PID_GUID _PID_HLINKSAN{ED5EF75E-5005-11D1-AA36-00C04FC2F60F}AL ex4.html importation via DirectShow Many native PCM audio formats wav, aifc, au, etc. Many audio CODECs supported G711, uLaw, MPEG, voxware, H.* coming via NetShow Users may add formats via Dshow filters Filters may be codebasedl!3A!3A  J. Audio Capabilities cont.Mixing Audio of different types (MIDI and PCM) Auto format conversion of Audio of different formats Multiple instantiation Import a sound once then play it simultaneously at different positions or rates Recursive mixing of soundsH]k]kL Audio Capabilities cont.zSpatial Audio Embed sounds in 3D geometry Animate Sound Geometry via 3D transforms Render Sound Geometry using Microphones$mmT"Present Audio ParametersAll Sounds (PCM, Streaming, SinSynth, MIDI) support: Gain, Rate, Phase, Pan, Loop Accumulate Parameters may be number behaviors themselves (filters/synthesizers coming with more elaborate controls)J5t58;U#Sound Behavior ConstructorsMix SoundSource(soundBvr) Union, transform (on sound geometries) SoundGeometry.RenderSound(microphones) Coming: numberBvr.SoundSynth(timeInterval) Could also be used in 2D to create textures$,,       ./DA Engine Exposed via comvAvailable from wide range of languages: JavaScript, VBSript DA in html file using script tag Java Visual Basic C, C++ H(!(!*490uv*  Future workAudio Parametric synthesis Dynamic music synthesis Additional network streaming of audio/movies Engine Enhanced retained mode control More sophisticated optimizationHZ?Z? ConclusionDescription based behavioral approach to multimedia Easy to construct media rich, synchronized animation High degree of integration Web based or stand alone animationsM!We are hiring! ) Audio AnecdotesCookbook in the spirit of Graphics Gems Soliciting Articles now! Topics and details at http://www.sworks.com/cookbook/solicitation.html $W2W2b!/'23567:<>AEHKNO P!Q"R#S$V%W&Y'Pry2pU2bY