ARTS 2610: Designing Musical Games

Arts Department, Rensselaer Polytechnic Institute

Monday/Thursday, 10:00-11:50am, VAST Lab, SAGE

Instructor: Rob Hamilton, West Hall 305/6

e: hamilr4 [at] rpi [dot] edu

w: http://homepages.rpi.edu/~hamilr4/arts2961

Office Hours: TBD

COURSE SYLLABUS - Fall, 2023

PREREQUISITE

None

COURSE DESCRIPTION

One of the most exciting areas of music technology development is happening in the realm of gaming and interactive virtual space. Music and Sound Design play crucial roles in the design of gaming environments, narratives and flow. And as designers create ever more innovative game experiences featuring rich graphics, fast multiplayer networking and next-generation controllers, new techniques for creating immersive music and sound for games to complement and showcase these advances are not only possible but necessary.

This Studio class will explore cutting edge techniques for building interactive sound and music systems for games and 2D/3D rendered environments. To better understand the link between virtual space and sound, students will learn the basics of designing sound and composing music for interactive game spaces by designing and implementing rich musical games within the Unity and Unreal gaming engines. Coursework will require the ability and desire to code game logic and design game environments. Techniques for integrating sound and music within games including game-centric middleware tools like FMOD and WWISE, interactive sound synthesis and computer networking using Open Sound Control may be explored.

Working in teams or on their own, students will design their own music-rich game experience, compose music, design sound material, and implement their own playable interactive musical game experiences.

CATALOG DESCRIPTION

Students will explore the artistic role of music and sound in gaming by building their own interactive sound and music-rich games and 2D/3D rendered environments. Within the context of their own creative game projects, students will learn the basics of designing sound and composing music for interactive game spaces. Using workflow programming languages and software tools, students will program basic gaming interactions, link them to interactive audio software, and create a musical gaming experience.

LEARNING OUTCOMES

    Students who successfully complete this course will demonstrate...
  1. an understanding and appreciation of game design and musical creation through an awareness of the many disciplines underlying the field including: software design and programming, interaction design, listening skills, musical theory, musical acoustics, digital audio theory, and digital signal processing.
  2. basic technical facility in the areas of game development, audio recording, editing, sound synthesis, software development and post production.
  3. creativity and resourcefulness through the creation of musical game environments and composition of your own sonic projects

EVALUATION

Evaluation is based on the following:

CLASS PARTICIPATION

You will be required to present some of your assignments to the class, to show your work within the software environment you used to create it, and to engage the class in discussion of your work. When you are not presenting your own work, you need to be attentive to whoever is presenting, and to engage them in discussion of their work. Failure to participate in class will lower your grade.

ATTENDANCE

You must attend class to succeed in this course.

  1. Since much of the class is focused on listening to and discussing work in class, attendance is mandatory.
  2. ** More then two unexcused absences will affect your grade, detracting 1/2 grade each additional 2 unexcused absences. **
  3. Absences can only be excused by a letter from a medical doctor or from the Dean of Students' office.
  4. Late arrivals are very disruptive - continued late arrival will affect your grade.
  5. It should go without saying but no use of mobile devices or personal computers during class time (except for as required by the coursework itself) is acceptable. Continued violations will be treated as an unexcused absence.

STATEMENT REGARDING ACADEMIC INTEGRITY

Collaboration between students in this course is strongly encouraged. Likewise, students are encouraged—indeed, to some extent required—to exchange ideas, opinions and information . You are also encouraged to help each other in the lab and with performance, production, and presentation of composition projects.

Plagiarism of any kind is in direct violation of University policy on Academic Dishonesty as defined in the Rensselaer Handbook, and penalties for plagiarism can be severe. In this class you will be expected to attribute due credit to the originator of any ideas, words, sounds, or music which you incorporate substantially into your own work. This applies particularly to citation of sources for sonic "samples" included in your compositions.

The use of automated technical aids (e.g. ChatBots, AI code plugins) is not strictly forbidden but if used MUST be documented in great detail and discussed with the Professor prior to use in a graded project/assignment.

Submission of any assignment that is in violation of this policy may result in a grade of F for the assignment in question. Violation of this policy will be reported, as defined in the Rensselaer Handbook

DISABILITY SERVICES FOR STUDENTS

Students requiring assistance are encouraged to contact Disability Services: http://doso.rpi.edu/dss to discuss any special accommodations or needs for this course.

OFFICE HOURS

Office Hours for Fall, 2023 will be TBD

TEXTBOOKS / REPOSITORIES / FOUNDATIONAL MATERIAL

Designing Sound, by Andy Farnell

Musimathics (Vol. I), by Gareth Loy (bad link)

Musimathics (Vol, II), by Gareth Loy (bad link)

Theory and Techniques of Electronic Music, by Miller Puckette


COURSE SCHEDULE

The proposed course topics and schedule will be as follows (take note of project due dates!). Based on class progress and interests, this schedule is subject to change. Special topics, guest lectures, supplemental reading, listening and additional assignments to be announced.

Week 1:
Monday, 8/28

CLASS CANCELLED

Thursday, 8/31

Introduction to Designing Musical Games 2023 - Syllabus review and Class Overview

Welcome to DMG Questionaire: Goo-drive link

Tool Overview:
- Audacity: http://www.audacityteam.org
- Unity: https://unity3d.com/get-unity/download
- Unreal engine: https://www.unrealengine.com/
- Pure Data (PD): https://puredata.info/downloads/pure-data
- ChucK: https://chuck.cs.princeton.edu/
- WWise: https://www.audiokinetic.com/

Week 2:
Tuesday, 9/5

Lecture: Designing Musical Games :: Gaming Musical Design
The role of sound and music in games; filmic influences; spectra of interactivity

- Destiny Trailer: https://www.youtube.com/watch?v=VFh5ArG46_M
- "Hope for the Future" https://www.youtube.com/watch?v=163_C5UVU-I
- Destiny 2 musical lore: https://gamerant.com/destiny-2-paul-mccartney-hope-for-the-future-song-lore/
- Marty O'Donnell talks Destiny/McCartney: https://www.youtube.com/watch?v=xsmRNGrh7Vk

- Journey Trailer: https://www.youtube.com/watch?v=2FZkTXs3Td8
- Journey Grammy Nomination: Forbes
- Austin Wintory Interview (BAFTA): https://www.youtube.com/watch?v=WcuMLbKTgwc

- Smule Magic Piano: https://apps.apple.com/us/app/magic-piano-by-smule/id421254504
- Smule Musical Scoring paper: Social Composition: Musical Data Systems for Expressive Mobile Music

- Zatoichi: The Blind Swordsman: https://www.imdb.com/title/tt0363226/

- Q3OSC (Quake 3) paper: Q3OSC: or How I Learned to Stop Worrying and Love the Game
- Nous Sommes Tous Fernando: https://ccrma.stanford.edu/~rob/portfolio/fernando/

- In C (Sirikata version): https://ccrma.stanford.edu/~rob/portfolio/inc/
- In C (Terry Riley): https://www.youtube.com/watch?v=yNi0bukYRnA

- OSCCraft (Minecraft + Open Sound Control): https://github.com/robertkhamilton/osccraft

- Carillon article: http://blog.leapmotion.com/twist-gears-massive-vr-music-engine-carillon/
- Carillon: https://www.youtube.com/watch?v=N6Me8B7Wst8

Thursday, 9/7

HOUSEKEEPING: box.rpi.edu file submission folder
- You all received an invitation to a shared "DMG Assignment Dropbox" Box folder. - zip up all your files into one .zip archive
- use the following naming convention: "a" + assignment # + "_" + rcs username + ".zip" (e.g. "a1_hamilr4.zip")
- if files are too crazy large, you can break your submission into multiple .zip archives. For example if you have 3 large .zip files, each would be titled as: "a1_hamilr4_1_of_3.zip", etc.


Basics of Sound File Editing with Audacity
- download Audacity: http://www.audacityteam.org

SUGGESTED READING: Chapter 7 "Digital Signals" in Designing Sound by Andy Farnell

MEDIA: I Just Want A Bit Part In Your Life Drive Link (.wav)

Assignment #1: In Class/At Home: Sound Design/Editing Challenge - Bit Part
- DUE DATE, THURSDAY, SEPTEMBER 14, 9:59:59 am EST (before class begins)

Your challlenge, should you choose to accept it, is to create a set of sounds for your next AAA game project using only the Bit Part single source sample, and Audacity. FOR THIS ASSIGNMENT YOU MUST USE AUDACITY AND NOT ANY OTHER DAW/EDITOR/SOFTWARE

    - Using Audacity, create FIVE the following audio cues/sound events listed below.
    - Each file should be exported as lossless .wav files at 44100Hz, 32 bit, Stereo or Mono
    - For each sound event you create, write a 2-3 sentence description of the processes you used to create it.
    - Create more than five for Extra Credit (each with a written description)
      Wind
      Laser beam
      Jet Engine
      Rolling ball
      Large object impact
      Small object impact
      Space-ship flying
      Player Death
      Player Spawn
      Game Victory
      Game Failure

Week 3:
Monday, 9/11

Introduction to Unity

- https://unity.com/
- IDE, simple navigation, basic GameObjects, Components
- C# Scripting


Thursday, 9/14
The Art and Science of Foley

aka: MOR-TAAL--KOMMMM-BAAAAT-day

Video : Digital Foley: "Mortal Kombat Foley: The Sound of Violence"
Video: "The Hunger Games" & "Frozen" Foley Artists Turn the Sound of Junk into Miracles

Assignment #1 Review: Super-Mega Lazers

Unity:

- Basic C# Scripting Overview
- AudioSource, AudioListener, 2D vs 3D audio settings
- Roll-A-Ball tutorial: https://learn.unity.com/project/roll-a-ball
Week 4:
Monday, 9/18
Unity Overview Continued:

- Roll-A-Ball tutorial: https://learn.unity.com/project/roll-a-ball
Unity Audio Continued:

- (Video/Tutorial): Audio Setup Tutorial
- AudioListener - the "ears" of your Player
- Audio and Visual Perspectives: Camera vs AudioListener placement
- AudioSource and AudioClip classes
- AudioMixer: hierarchies of Mixer Groups
- Mixer plugins: Attenuation + effects, order of operations/signal flow
- Parent/Child relationships
- Randomizing AudioSources
- C# AudioScript playback: Play vs. PlayOneShot

* In Class Haxoring: Roll-A-Ball Audio Madness

Script/Asset Share folder: Goo-drive

Thursday, 9/21
AudioMixer: Exposed Parameters and SetFloat() (tutorial video")

- Unity Asset Store and Package Manager: accessing Starter Assets
- - Third-Person Character Controller assets
- - First-Person Character Controller assets

Limbo Walkthrough: https://www.youtube.com/watch?v=1ie19_GXAAw
WWise Limbo Project: WWise Limbo project
GDC Talk: Martin Stig Andersen: INSIDE: A Game that Listens: https://www.youtube.com/watch?v=Dnd74MQMQ-E


Assignment #2: Sample-based game audio scene (- DUE DATE, THURSDAY, SEPTEMBER 28, 9:59:59 am EST (before class begins))

Using your favorite samples, either from our in-class recording challenge or new samples you make yourself, populate your Roll-A-Ball project OR either the First or Third-Person Starter Assets scenes with sound. You can modify and add functionality or assets to the scene as needed/desired to make the game do what you'd like it to.

The success of your scene will revolve around whether or not your scene is sonically-coherent, i.e. all the sounds exist and belong in the same space. Coherence can be helped by creative use of reverberation in game, by careful editing of sounds in Audacity, or simply through artistic choices you make in your role as sound-designer.
In addition to the project itself, create a detailed "call-sheet" list of all the sounds you will use/are using in the project, including variations. List file-names, extensions, durations, and any notes that would be helpful or useful to your programming team*. Please include a short paragraph or description of the overall-zeitgeist of your scene, and why we should care.
Please include examples of the following Unity audio processes in your scene:

    * Simple Audio Source playback (multiple)
    * Mixer Groups
    * Basic C# scripting of audio playback events
    * 3D sound/distance-based attenuation
    * Varying pitch, amplitude, etc.
    * Sounds triggered through collision
    * An awesome looping stereo background "musical" track
    * EXTRA-CREDIT: Make your background track a multi-channel "stem"-based playback example
    * EXTRA-CREDIT: include MIDI file control and/or Microphone input

Please upload your entire project to the Box dropbox folder using the same naming conventions as Assignment #1, i.e:
HOUSEKEEPING: box.rpi.edu file submission folder
- You all received an invitation to a shared "DMG Assignment Dropbox" Box folder. - zip up all your files into one .zip archive
- use the following naming convention: "a" + assignment # + "_" + rcs username + ".zip" (e.g. "a1_hamilr4.zip")
- if files are too crazy large, you can break your submission into multiple .zip archives. For example if you have 3 large .zip files, each would be titled as: "a1_hamilr4_1_of_3.zip", etc.


* the programming team is a lie...
Week 5:
Monday, 9/25

AudioMixer: Exposed Parameters and SetFloat() (tutorial video")


Reverb Zones
AudioMixer bussing (continued): Sends and Receives, Ducking (Sidechaining), Mixer Parameterization
Advanced Mixing

Audio as Data: Microphone input - thread...
Microphone input

Thursday, 9/28
Procedural Audio: an Overview and (gentle) Introduction

Spore and Pure Data: Usage of PD in Spore and Darkspore
Kent Jolly, Aaron McLeran @ GDC 2008: https://www.gdcvault.com/play/323/Procedural-Music-in
Will Wright & Brian Eno - Spore Creature Demo
Generative Music: a talk (Eno)
The Music of 'Spore Hero' (Wii, Winifred Phillips)

The Sound of 'No Man's Sky' (GDC 2017, Paul Weir)
Behind the Sound of 'No Man's Sky'

Proteus
Panoramical

FractOSC
FractOSC walkthrough: https://youtu.be/v__SzSFkrOQ?t=211
LibPD: http://libpd.cc/

Appolonian Gasket by Ignacio Pecino: video, paper
Singularity by Ignacio Pecino: video, demo (Firefox, Safari)

Hacking Musical Games (for fun and profit)

q3apd by Julian Oliver: site
Week 6:
Monday, 10/2

Generative Music Systems: Leonard Paul's music from "Sim-Cell": youtube
Martin Brinkmann patches: http://www.martin-brinkmann.de/pd-patches.html
RJDJ Abstractions: https://github.com/rjdj/rjlib

GG Music (Soundcraft/Starcraft II)
Paper: Soundcraft: Transducing StarCraft 2 (2013)
ChucK: https://chuck.cs.princeton.edu/

Introduction to Pure Data: [objects], [number-boxes], [messages], [dac~], [*~], [osc~]

Pure Data (PD): https://puredata.info/downloads/pure-data

Introduction to Open Sound Control: OSC-from-Unity with OSCsharp (C# classes), integration with Unity - https://github.com/valyard/OSCsharp
Bi-directional OSC PD Patch: simple_pd_receiver.pd - https://ccrma.stanford.edu/~rob/rpi/arts2963/simple_pd_receiver.pd

Thursday, 10/5

OSC with PD and Unity (continued)
Introduction to Unreal Engine 5

Blueprint Workflow Scripting Language, IDE basics, default projects
Week 7:
Monday, 10/9
Thursday, 10/12

Unreal Engine 5 basics (continued)

Blueprint Workflow Scripting Language (continued)
Audio in Unreal Engine 5.3
UE5: Legacy Audio system: Audio Mixer, Sound Classes, and Sound Mixes
Sound Wave Assets: importing, properties
Audio Listener - Blueprint for moving from Camera to third-person character
Week 8:
Monday, 10/16

Unreal Engine 5: Blueprints (continued)
arrays, audio files in blueprints
Sound Cues: https://docs.unrealengine.com/5.3/en-US/sound-cues-in-unreal-engine/

MetaSounds Introduction
- Epic tutorial: Quick Start: Bomb and Wind
- Epic tutorial: Procedural Music

Tutorial project download: .zip (big)

Thursday, 10/19
MetaSounds
- WavePlayer sample playback
- Random, Trigger Repeat, variables, parameter widgets
- Wave asset collections, Random Get, weights/probability
- Loop start, Loop duration


Set of introductory tutorials: Epic community tutorials
Week 9:
Monday, 10/23

MetaSounds Continued...
- Metasounds UE5 Reference: https://docs.unrealengine.com/5.3/en-US/metasounds-reference-guide-in-unreal-engine/
- Adding audio file cues in Reaper
- Reading audio file cues in Metasounds: https://www.youtube.com/watch?v=ZHPZoNFNlY8
--- or (simpler): https://www.youtube.com/watch?v=R4OvC9ekYso
- Sending Metasound data back to Blueprint: https://www.youtube.com/watch?v=G_eK2LRJBDo

Footsteps via animation Blueprint (anim notify): https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/Sequences/Notifies/


Assignment #3 Proposal Document (Due Thursday, 10/26, 11:59:59 PM AoE [anywhere on earth] )
* Submit a written project proposal for your Assignment #3 Unreal Engine audio project (see below). This proposal should describe the scene you'll be creating, the types of sound assets and sound generation techniques you plan on using, and should show me that you've given thought to the overall direction and focus of your upcoming assignent.
* There is no "minimum" or "maximum" length to your design document submission; if you can succinctly explain your project in one sentence, all the better. However it is your responsibility to clearly explain your proposed project and the technologies you will leverage to create it to me.
* Email your proposal to me by the due date/time.


Assignment #3: Unreal audio scene (Due Thursday, 11/2, 11:59:59 PM AoE [anywhere on earth] )

This project represents your chance to showcase your understanding of the basic underlying principles of game sound and music within the Unreal Engine. Populate a simple Unreal scene using sound files downloaded from Soundsnap or some other online repository - or sound assets of your own creation (even better!). You can create an Unreal scene from scratch or use one of the many available Unreal project templates (displayed when you create a new Unreal project).

As always, the success of your scene will revolve around whether or not your scene feels coherent, i.e. all the sounds exist in the same space. Coherence can be helped by creative use of reverberation in game, by careful editing of sounds in Audacity, or simply through artistic choices you make in your role as sound-designer. In addition to the project itself, create a detailed "call-sheet" list of all the sounds/assets/processes you will use/are using in the project, including variations. List file-names, extensions, durations, and any notes that would be helpful or useful to your programming team*. Please include a short paragraph or description of the overall-zeitgeist of your scene, and why we should care.

Please include examples of the following Unreal audio processes in your scene:

    * Footsteps (for 1st and 3rd person scenes), or movement sounds (i.e. vehicle sounds)
    * Simple Sound Assets
    * Sound Assets that continuously play regardless of player distance
    * Randomized Sound Cues or Metasounds
    * Sound Assets/Cues/Metasounds with proximity volume attenuation
    * Sound Assets/Cues/Metasounds triggered by Box or Sphere trigger volumes
    * Sound Assets/Cues/Metasounds triggered by a keypress
    * Sound Cues/Metasounds with one or more parameters driven by in-game action or events

*** As Unreal Projects can be quite large, we're not going to upload entire projects this time. Instead, I'm requesting you record a short video, YouTube style, showcasing all the elements in your project. Please include either a voice-over or text captions so that I know what's going on.
*** Name your project "a3_your-rcs-id.mov" or whatever extension/video type


Thursday, 10/26
NO CLASS
Week 10:
Monday, 10/30
Fortnite modular synths: https://cdm.link/2023/10/fortnite-gets-a-modular-synth/"
2023 Epic UnrealFest audio overview: https://www.youtube.com/watch?v=rOZrE-MkhM8
Introduction to Open Sound Control with Unreal

Thursday, 11/2
Week 11:
Monday, 11/6
Virtual Reality Musical Instrument design: Coretet demo and overview

Thursday, 11/9
Introduction to Wwise: Wwise 101 course
Week 12:
Monday, 11/13
Wwise in Unreal Engine 5

Thursday, 11/16
Wwise in Unity: 301 tutorial series
pipe-burst + fire alarm + acrid smelling steam...
Week 13:
Monday, 11/20
Embedding Madness:
- LibPD
- Enzien Audio: Heavy/hvcc
- Chunity
- Chunreal
- Max/MSP's RNBO

Monday 11/27- Final Project Proposals Due:

Submit a detailed written 1-page description of your proposed final project for this class.

Your project may encompass any or all of the sound and music compositional/music/sonic techniques and technologies we have investigated during the course of this class including fixed media sound design and composition with Audacity or another DAW, workflow interactive programming systems with Pure Data, ChucK or some other audio programming environmnet, Unity, Unreal, Metasounds and/or WWise/FMod. Your proposal should address the integration (technical) and the mapping (artistic) of your data and control systems from game engine to audio layer, and vice versa.

Your proposal should be clear, concise and well written and should include both creative ideas and descriptions you wish to engage alongside technical tools and concepts you wish to employ.

FINAL PROJECT OVERVIEW

Due: Saturday, 12/16

Your final project for Designing Musical Games will be, surprisingly, a musical-game, built using the tools and techniques we've investigated during this semester. That means your game can be built using game engines such as Unreal Engine or Unity, using the audio capabilities of the engines themselves or using an external audio environment such as Pure Data, ChucK or WWise. Sounds for your system should be your own, with music and samples edited in Audacity or your favorite DAW.

If you use stock music or sounds downloaded from the internet they should not make up the majority of your project; if you do download assets, please credit them in your project write-up.

For your final project, you will be expected to submit the following:
* A written overview of your project - this should be a well crafted overview document describing the technical and artistic choices you made in your project and how you made use of sound and interaction. Also any instructions on getting your system running need to be included here as well.
* A video play-test of your project - a short video capturing the game-play and all the interesting parts of your project you wish to showcase. A narrative overview or text captioning is required.
* Source code for your project - all files needed to run your project should be uploaded to the Box folder. If you can compile an executable of your system, that's great, though for some projects that will not be possible.

Your project may encompass any or all of the sound and music compositional/music/sonic techniques and technologies we have investigated during the course of this class including fixed media sound design and composition with Audacity or another DAW, workflow interactive programming systems with Pure Data, ChucK or some other audio programming environmnet, Unity, Unreal, Metasounds and/or WWise/FMod. Your proposal should address the integration (technical) and the mapping (artistic) of your data and control systems from game engine to audio layer, and vice versa.

REMINDER: Upload your project submission files to the shared "DMG Assignment Dropbox" Box folder.
- zip up all your files into one .zip archive
- use the following naming convention: "a" + assignment # + "_" + rcs username + ".zip" (e.g. "a1_hamilr4.zip")
- if files are too crazy large, you can break your submission into multiple .zip archives. For example if you have 3 large .zip files, each would be titled as: "a1_hamilr4_1_of_3.zip", etc.


Thursday, 11/23
NO CLASS (Thanksgiving)
Week 14:
Monday, 11/27
Final Project Hack-A-Thon
Week 15:
Monday, 12/4
Final Project hacking-testing session
FINAL PROJECT SUBMISSION

Saturday, 12/16

Final Project Absolute last turn-in day