Please check out "Hello, Magenta" and "Play, Magenta" before reading if you haven't!
It's always exciting when there are cool visuals to help bring a musical experience to life. I saw core visualizer for Magenta and decided it would be fun to try to create dynamic animations as the notes are being played. By visualizing notes in a dynamic way, we can see more information about the notes, and be entertained by more stylized visuals. This demo is remixed from Vibert's "Play, Magenta" demo, now with animations!
I wanted to make a basic animation effect where notes glow as they are being played, and their interior color fill shrinks as their duration runs out. You can see my effect and mess around with it in the editable code playground below. Below that, I'll explain how I made the demo, and try to deconstruct some of the core concepts needed to make it.
I was particularly inspired by a youtuber smalin. He creates eye catching visualizations for classical music that are great for visualizing the voices of different instruments in symphonic pieces. In this video we can see that the shrinking fill of the drawn shapes helps us visualize the note duration, and the glow of the outline helps draw our eye to the note that is being played.
I hadn't worked with Magenta.js prior to working on this demo. I went through "Hello, Magenta" and "Play, Magenta" and had a basic grasp of what was going on, but there were still some pieces of the puzzle that I was missing before I could make my animations. Only by going into the code in those demos and others, and doing some console.log statements and playing around, did I figure out some things about Magenta.js and Tone.js. Here's were my questions, and what I learned:
Those demos (and this one!) are glitch.com demos, which means that if you click the little fish (sidenote: why is fish the plural form of fish?) on the top right, you can inspect the source code, and even say "Remix on Glitch" to be able to edit it all of it! This was the most useful thing to me, as I could go in, put a lot of console.log statements, run the demo and open the inspector, and really see what was being stored in each variable. This clarified a lot for me.
As I rewrote and read through some of the script.js file, I tried to annotate it with comments that would clarify each step of the process. Also, when you edit the code mirrors in all of these demos, the text areas are listening for a change to their contents, and then the contents are evaluated and run in the same Javascript codespace as this script.
In this demo's script.js file, the first function "getToneEventsFromQuantizedNoteSequence(sequence)" helped me learn a lot about what is going on behind the scenes for Magenta.js.
We learned in the "Hello, Magenta" demo that Magenta.js NoteSequences are an array of Note objects that detail the melody that the model generates. I recommend looking at their definitions, linked above.
Magenta.js uses Tone.js to render all of its sounds, and Tone.js Instruments play notes by calling triggerAttackRelease. When called, Tone.js uses the web audio framework to make a sound with the specified digital instrument.
The array of objects that are created by getToneEventsFromQuantizedNoteSequence are then used to construct a Tone.js Part and this triggers a callback function that gets called on each of those generated objects, and within this callback we call triggerAttackRelease.
We can see that the Magenta.js Player class does this for us in its start function.
This tutorial assumes you know some of the basics about HTML Canvas. Please go through their basic tutorials and come back if you are not familiar with the basics.
When getting started with animation, we will draw our Canvas using window.requestAnimationFrame(callback). This function will listen to the browser window, and when the browser window says the screen is being refreshed, it will call the callback. In order to continue animating, the callback must request another animation frame at its end, and do so until the animation is over. As the resources also note, the rate at which this function is called is dependent on the refresh rate of your browser, and passes a timestamp (milliseconds elapsed since the origin of the page) as an argument into the callback.
After understanding the basics of how animation works, we can start working on creating the effect that we want in our demo rectangle. I knew I wanted the fill of the rectangle to shrink, and for the rectangle to glow. In the demo below, we can see how I achieved this effect from the comments in the code.
So now we have our effect ready, but how do we apply it to our Magenta visualizer rectangles? In previous demos, we noticed that in the draw function, there is an activeNote boolean, and we could make changes to that draw function to modify the visual representations of the notes.
However, this demo ended up being more complicated. This is because, previously in the "Play, Magenta" demo, the draw function was only called any time a note was played (in the Tone.js Part callback that we talked about in step 0).
In order to render our continuous animation effects, we need to keep calling our draw function using requestAnimationFrame for the duration of the audio. If you check script.js for this page, there is a boolean flag called "continueDrawing" that we set to true when the play button is called, and set to false when our Tone.js Part callback encounters a note that has an "isEnd" boolean flag set to true.
We then call our draw function when the play button is hit, and get rid of the draw call inside the Part callback. And at the end of our draw function, we call requestAnimationFrame and set the callback as the draw function yet again. This will make it so that we will call draw whenever the page requests an animation frame and music is playing.
Now, all we have to do is apply the effect to the rectangles in the renderer! Our animation effect is based on a time value between 0 and 1 of how far we are through the animation. So if we are applying our effect to the notes, we just need to calculate how far we are through each note at any given time.
Tone.js measures time in many ways, but one of the units is Ticks. These are the smallest subunit of time Tone.js keeps track of, and they will function like the millisecond timestamps that we were using earlier. We don't want to use the same timestamps that the requestAnimationFrame function is giving us, because we want to have an understanding of how much time has passed in our melody.
For this, we can take advantage of Tone.js Time. In the getToneEventsFromQuantizedNoteSequence function we discussed in step0, I added data to the Note object that was passed in, called startTimeInTicks and durationInTicks. These values are very similar to the animationStart and animationDuration values that we referenced in our animation demo. You can see how I used the Tone.Time library to convert from quantized step units that Magenta.js uses, to the Tick values that Tone.js uses. There is more information about that in the Tone.js Time documentation.
Then in our draw function, we can take advantage of Tone.js Transport. We can call the function Tone.Transport.position(), which returns a Tone.js Time of how far we are into our song. We can use the time library we talked about above to convert it to Ticks. With this, we can then iterate through the notes, just like in past demos, and use our new Tone.js Tick based timestamps to determine how far we are into the effect animation of each note!
We then put it all together in the draw function:
That's pretty much everything I had to change from the "Play, Magenta" demo to get this animation to work!
You're now ready to build your own amazing, Machine Learning powered, music animation demos! If you want more information, you can check out...
#MadeWithMagenta
Have fun!!!