NSynth

The model generates sounds through a neural network based synthesis, employing a WaveNet-style autoencoder to learn its own temporal embeddings from four different sounds.{{Cite journal|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|url=https://research.google/pubs/pub46119/|website=research.google|date=2017 |last1=Engel |first1=Jesse |last2=Resnick |first2=Cinjon |last3=Roberts |first3=Adam |last4=Dieleman |first4=Sander |last5=Eck |first5=Douglas |last6=Simonyan |first6=Karen |last7=Norouzi |first7=Mohammad |arxiv=1704.01279 }}{{Cite arXiv |eprint=1609.03499|author1=Aaron van den Oord |last2=Dieleman |first2=Sander |last3=Zen |first3=Heiga |last4=Simonyan |first4=Karen |last5=Vinyals |first5=Oriol |last6=Graves |first6=Alex |last7=Kalchbrenner |first7=Nal |last8=Senior |first8=Andrew |last9=Kavukcuoglu |first9=Koray |title=WaveNet: A Generative Model for Raw Audio |year=2016 |class=cs.SD }} Google then released an open source hardware interface for the algorithm called NSynth Super,{{Cite magazine|title=Google's open-source neural synth is creating totally new sounds|url=https://www.wired.co.uk/article/google-ai-nsynth-algorithm-music-creativity|magazine=Wired UK }} used by notable musicians such as Grimes and YACHT to generate experimental music using artificial intelligence.{{Cite web|title=73 | Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll|url=https://www.preposterousuniverse.com/podcast/2019/11/18/73-grimes-c-on-music-creativity-and-digital-personae/|website=www.preposterousuniverse.com}}{{Cite web |last=Mattise |first=Nathan |date=2019-08-31 |title=How YACHT fed their old music to the machine and got a killer new album |url=https://arstechnica.com/gaming/2019/08/yachts-chain-tripping-is-a-new-landmark-for-ai-music-an-album-that-doesnt-suck/ |access-date=2022-11-08 |website=Ars Technica |language=en-us}} The research and development of the algorithm was part of a collaboration between Google Brain, Magenta and DeepMind.{{Cite web|title=NSynth: Neural Audio Synthesis|url=https://magenta.tensorflow.org/nsynth|website=Magenta|date=6 April 2017 }}

The NSynth dataset is composed of 305,979 one-shot instrumental notes featuring a unique pitch, timbre, and envelope, sampled from 1,006 instruments from commercial sample libraries.{{Cite web |title=NSynth Dataset |url=https://datasets.activeloop.ai/docs/ml/datasets/nsynth-dataset/ |access-date=2022-11-08 |website=Machine Learning Datasets |language=en-US}} For each instrument the dataset contains four-second 16 kHz audio snippets by ranging over every pitch of a standard MIDI piano, as well as five different velocities.{{cite arXiv|eprint=1907.08520|last1=Ramires |first1=António |last2=Serra |first2=Xavier |title=Data Augmentation for Instrument Classification Robust to Audio Effects |year=2019 |class=cs.SD }} The dataset is made available under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.{{Cite web|title=The NSynth Dataset|url=https://magenta.tensorflow.org/datasets/nsynth|website=tensorflow.org|date=5 April 2017 }}

A spectral autoencoder model and a WaveNet autoencoder model are publicly available on GitHub.{{Cite web|title=NSynth: Neural Audio Synthesis|url=

https://github.com/magenta/magenta/tree/main/magenta/models/nsynth|website=GitHub}} The baseline model uses a spectrogram with fft_size 1024 and hop_size 256, MSE loss on the magnitudes, and the Griffin-Lim algorithm for reconstruction. The WaveNet model trains on mu-law encoded waveform chunks of size 6144. It learns embeddings with 16 dimensions that are downsampled by 512 in time.{{Cite arXiv|eprint=1704.01279|class=cs.LG|first1=Jesse|last1=Engel|first2=Cinjon|last2=Resnick|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|last3=Roberts|first3=Adam|last4=Dieleman|first4=Sander|first5=Douglas|last6=Simonyan|first6=Karen|last7=Norouzi|first7=Mohammad|year=2017|last5=Eck}}

{{Infobox synthesizer

| image = File:Nsynth-super.jpg

| alt = The NSynth Super front panel: a metal box with a bright colored screen input.

| image_caption = NSynth Super Front Panel

| synth_name = NSynth Super

| synth_manufacturer = Google Brain, Google Creative Lab

| dates = 2018

| synthesis_type = Neural Network Sample-based synthesis

| left_control = Pitch bend, ADSR

| ext_control = MIDI

}}

In 2018 Google released a hardware interface for the NSynth algorithm, called NSynth Super, designed to provide an accessible physical interface to the algorithm for musicians to use in their artistic production.{{Cite web|title=NSynth Super is an AI-backed touchscreen synth|url=https://www.theverge.com/circuitbreaker/2018/3/13/17114760/google-nsynth-super-ai-touchscreen-synth|website=The Verge|date=13 March 2018 }}{{Cite web |date=13 March 2018 |title=Google built a musical instrument that uses AI and released the plans so you can make your own |url=https://www.cnbc.com/2018/03/13/google-launches-nsynth-super-ai-hardware-prototype.html |website=CNBC}}

Design files, source code and internal components are released under an open source Apache License 2.0,{{Cite web |date=April 1, 2021 |title=googlecreativelab/open-nsynth-super |url=https://github.com/googlecreativelab/open-nsynth-super |via=GitHub}} enabling hobbyists and musicians to freely build and use the instrument.{{Cite web |title=Open NSynth Super |url=https://hackaday.io/project/89396-open-nsynth-super |access-date=2022-11-08 |website=hackaday.io |language=en}} At the core of the NSynth Super there is a Raspberry Pi, extended with a custom printed circuit board to accommodate the interface elements.{{Cite web|title=NSYNTH SUPER Hardware|url=https://github.com/googlecreativelab/open-nsynth-super/tree/master/pcb_hardware|website=GitHub}}

Despite not being publicly available as a commercial product, NSynth Super has been used by notable artists, including Grimes and YACHT.{{Cite magazine |last=Mattise |first=Nathan |title=How YACHT Used Machine Learning to Create Their New Album |language=en-US |magazine=Wired |url=https://arstechnica.com/gaming/2019/08/yachts-chain-tripping-is-a-new-landmark-for-ai-music-an-album-that-doesnt-suck/ |access-date=2023-01-19 |issn=1059-1028}}{{Cite web |title=Cover Story: Grimes is ready to play the villain |url=https://crackmagazine.net/article/long-reads/grimes-is-ready-to-play-the-villain/ |access-date=2023-01-19 |website=Crack Magazine}}

Grimes reported using the instrument in her 2020 studio album Miss Anthropocene.

YACHT announced an extensive use of NSynth Super in their album Chain Tripping.{{Cite web |date=2019-09-18 |title=What Machine-Learning Taught the Band YACHT About Themselves |url=https://losangeleno.com/people/what-machine-learning-taught-the-band-yacht-about-themselves/ |access-date=2023-01-19 |website=Los Angeleno |language=en-US}}

Claire L. Evans compared the potential influence of the instrument to the Roland TR-808.{{Citation |title=Music and Machine Learning (Google I/O'19) | date=8 May 2019 |url=https://www.youtube.com/watch?v=pM9u9xcM_cs |language=en |access-date=2023-01-19}}

The NSynth Super design was honored with a D&AD Yellow Pencil award in 2018.{{Cite web |title=NSynth Super {{!}} Google Creative Lab {{!}} Google {{!}} D&AD Awards 2018 Pencil Winner {{!}} Interactive Design for Products {{!}} D&AD |url=https://www.dandad.org/awards/professional/2018/product-design/27149/nsynth-super/ |access-date=2023-01-19 |website=www.dandad.org}}

{{reflist|30em}}

• {{cite arXiv |year=2017 |title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders |class=cs.LG |eprint=1704.01279|last1=Engel |first1=Jesse |last2=Resnick |first2=Cinjon |last3=Roberts |first3=Adam |last4=Dieleman |first4=Sander |last5=Eck |first5=Douglas |last6=Simonyan |first6=Karen |last7=Norouzi |first7=Mohammad }}

• [https://nsynthsuper.withgoogle.com/ Official Nsynth Super site]
• [https://magenta.tensorflow.org/ Official Magenta site]
• [https://experiments.withgoogle.com/sound-maker In-browser emulation of the Nsynth algorithm]

Category:Synthesizers

Category:Electronic music

Category:Datasets in machine learning

Category:Machine learning algorithms