Magenta RealTime: an open source model for generating music in real time

Magenta RealTime (Magenta RT for short) is an open source music generation model developed by Google DeepMind that focuses on real-time music creation. It is an open source version of Lyria RealTime that supports the generation of high-quality music clips via text or audio cues. The model is based on the Transformer architecture with 800M parameters, and the training data contains about 190,000 hours of instrumental stock music. Users can run it for free via Google Colab or deploy it on local devices.Magenta RT is released under the Apache 2.0 and CC-BY 4.0 licenses, with code and model weights publicly available to encourage musicians and developers to explore innovative applications. Its core goal is to augment human music creation with dynamic, interactive music generation experiences suitable for live performance and soundscape creation.

Function List

• Real-time music generation: Generates 2-second music clips based on text cues (e.g., "light jazz") or audio clips, and is very responsive.
• Multi-modal inputs: Supports mixed text and audio cues for flexible control of music style and tempo.
• Cross-stylistic fusion: Different music styles can be mixed in real time, such as electronic and classical music.
• Open Source and Local Deployment: Provides model code and weights and supports running on Colab TPUs or native GPU/TPUs.
• Low latency generation: Every 2 seconds of music generation takes about 1.25 seconds, making it suitable for live performances.
• Cross-platform support: Can be integrated in browsers, DAW plug-ins or game engines to extend the music creation scene.

Using Help

Installation process

Magenta RealTime supports a wide range of runtimes for users with different technical backgrounds. Detailed installation and usage instructions are provided below.

1. Run through Google Colab (recommended for novices)

Google Colab is the easiest way to run Magenta RT without having to configure hardware locally. The steps are as follows:

• Visit the official Colab Demo:https://colab.research.google.com/github/magenta/magenta-realtime/blob/main/notebooks/Magenta_RT_Demo.ipynbThe
• Log in to your Google account, click the "Run" button, and Colab will automatically allocate free TPU resources.
• Follow the instructions in the notebook, enter text (e.g. "gentle piano") or upload an audio clip, and click Run to generate the music.
• The output results in 48kHz stereo audio that can be downloaded or played back in real time.

2. Local installation (for developers)

To run Magenta RT locally, GPU or TPU hardware is required. Below are the installation steps:

• Cloning the code base:

  git clone https://github.com/magenta/magenta-realtime.git
  cd magenta-realtime
  

• Installation commands are selected based on the hardware:
  • GPU Support:

    pip install 'git+https://github.com/magenta/magenta-realtime#egg=magenta_rt[gpu]'
    

  • TPU Support:

    pip install 'git+https://github.com/magenta/magenta-realtime#egg=magenta_rt[tpu]'
    

  • CPU (test only):

    pip install 'git+https://github.com/magenta/magenta-realtime'
    

• Once the installation is complete, run the sample script:

  python -m magenta_rt.demo
  

• Ensure that Python version is 3.5 or above, the Anaconda environment is recommended to simplify dependency management.

3. Operation of core functions

At the heart of Magenta RT is real-time music generation, which operates as follows:

• Text Tip Generation::
  • Enter descriptive text such as "upbeat drums" or "classical strings" in Colab or a local script.
  • The model generates 2-second audio clips based on the input and smooths the articulation based on the first 10 seconds of audio context (using a cross-fade technique to reduce boundary distortion).
  • Example: Run in Colab:

    from magenta_rt import generate
    audio = generate(text_prompt="soft piano melody", duration=2.0)
    

  • Output audio can be played back in real time or saved as a WAV file.
• Audio cue generation::
  • Upload a short piece of audio (e.g., a 10-second guitar clip) and the model will continue its style to generate a new clip.
  • Sample code:

    audio = generate(audio_prompt="input.wav", duration=2.0)
    

  • Text and audio cues can be mixed by adjusting the weights, e.g. "50% jazz + 50% input audio".
• Stylistic fusion::
  • Enter multiple cues, such as "60% electronic + 40% classical", to generate a mix of styles.
  • Adjust the "temperature" parameter (0.0-1.0) in the Colab interface to control the randomness of the generated music.
• real time control::
  • Dynamically change cues during a live performance via the API or DAW plugin and the model will respond to the new style within 2 seconds.
  • Example: Integrate into Ableton Live using the Lyria RealTime API (application required) to adjust the tempo of music in real time.

4. Advanced use

• Integration with DAWs: Magenta RT can be used with Ableton Live, FL Studio, etc. via VST plug-ins such as DDSP-VST. After installing DDSP-VST, drag and drop to a MIDI track and enter the prompts to generate music.
• Game Engine IntegrationMagenta RT can be embedded into Unity or Unreal Engine to dynamically generate soundscapes based on game events. For example, when the player enters a battle scene, enter a "tense strings" cue.
• Model Tuning: Magenta RT supports fine-tuning, which allows developers to optimize models using custom datasets (e.g., specific music styles). Technical details will be described in an upcoming report.

5. Cautions

• procrastinate: It takes about 1.25 seconds to generate 2 seconds of audio, and there may be a 2-second delay for cue changes.
• context limit: The model only references the first 10 seconds of audio, and is unable to generate long-term musical themes.
• hardware requirement: At least 16GB RAM and an NVIDIA GPU (e.g. GTX 1080) is recommended for local operation.

application scenario

1. Live Music Performance
   • DJs or musicians can generate music in real time through text or audio cues on the spot, quickly switching styles to enhance performance interactivity.
2. Game Soundscape Design
   • Developers can integrate Magenta RT into their games to generate dynamic background music based on player behavior, such as "soft piano" for calm scenes and "intense drums" for battle scenes.
3. art installation
   • Artists can use the models to create interactive soundscapes for exhibitions, where viewers can change the ambient music by typing in a cue (e.g., "dream electronics").
4. Music Composition Assistance
   • Musicians can use Colab to quickly generate snippets of inspiration for songwriting or to break creative bottlenecks.

QA

1. Does Magenta RT support generating lyrics?
   • Not supported. The model generates primarily instrumental music, which can produce non-verbal humming, but cannot generate clear lyrics. To avoid inappropriate content, it is recommended to use the Lyria RealTime API for broader style support.
2. How to run Magenta RT locally?
   • Clone the GitHub repository, install the corresponding hardware dependencies (GPU/TPU/CPU), and run the example script. We recommend using Colab to minimize hardware requirements.
3. Are there any copyright issues with generating music?
   • The user owns full copyright to the generated music and Google does not claim any rights. However, it is necessary to ensure that you do not infringe the copyrights of others and that you comply with the Apache 2.0 and CC-BY 4.0 licenses.
4. What music styles are supported by the model?
   • Mainly supports western instrumental styles (e.g. jazz, electronic, classical) with limited support for global music traditions and lyrics. Recommend Lyria RealTime API for wider coverage.