In 2018, WiMIR hosted their first workshop. It was a high quality event and I would recommend anyone going to an ISMIR conference in the future to join. I spent a lot of time at ISMIR exploring the idea of turning music consumers into creators. I believe machine learning will play a huge role in this in the future, and it was a really interesting idea to explore in the WiMIR workshop group "Building Collaborations Among Artists, Coders and Machine Learning". With the aid of machine learning, not only can we narrow the gap between consumers and creators, but creators and developers.
Outside Hacks 2018 was sponsored by both Universal Music Group and Twitch, which resulted in a hackathon with a 14,000 song catalogue and the Twitch API. This sets the stage for an application that not only contains your favorite music, but allows you to collaboratively interact with it on a live stream. That's serious product potential.
Andrew Silverman, Isaac Chien, Emma Qian and I put together Beatcraft. In its early form, the idea was to build a collaborative drum machine where you can make beats with your friends remotely. Drum machines are fun, but to create usable content, people usually need a little more. We wanted to create something where:
- People can use familiar content
- People aren't excluded based on musical skill/ability
- People can collaborate with artists and/or friends
- Creation of novel and unique content is possible
Naturally this lead to a combination of a sampler/slicer and drum machine, but to fulfill these requirements we have to go a little further. Anyone who has done sampling based music production knows it takes a bit of talent to find samples, cut them in the correct place, and transform/combine them in a musically meaningful way.
I had recently been researching deep learning for music segmentation, and as part of the literature review I had been reading classics such as this paper by Levy and Sandler. I was inspired to see how far we could go in using similar methods to automate the sampling process, allowing people of any skill level to consume their music interactively with others and perhaps create something new in the process.
This sounds ambitious for a 24-hour hackathon, right from the outset this demanded a number of non-trivial technologies:
- Beat tracking
- Time stretching/resampling
- Automatic sample finding
We weren't sure if it was possible, but the very premise of 24 hours of non-stop development is highly unreasonable in its own right, so why not? The end product is a sampler with a workflow as follows:
- Begin a Twitch stream
- Search for and select your favourite song
- Program a series of 4 beat or 2 beat samples
- Add a drum beat
- Watch your friends make tweaks and changes
- Mix it up with filters and effects during playback
The key component is that you're exploring music interactively with your friends from the comfort of your own home.
Unfortunately, due to copyright constraints I can't serve this application with the full 14,000 song Universal catalogue. Fortunately, one of the team, Isaac Chien, is an artist himself and recorded a demo – watch Isaac sample The Carpenters using the complete application (MIDI and all) below: