Pop2Piano 深度实战指南：用 Transformers 从流行音频直接生成钢琴翻弹 MIDI

Pop2Piano 深度实战指南用 Transformers 从流行音频直接生成钢琴翻弹 MIDI【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformersPop2Piano 是首个无需旋律与和弦抽取模块、直接从流行歌曲音频波形端到端生成钢琴翻弹Piano CoverMIDI 的 Transformer 模型。本文以 Pop2Piano 模型文档 为主体结合 Transformers 仓库内该模型的完整源码与测试系统讲解其工作原理、环境安装、推理配置与多场景实战用法帮助你直接复现上传音频 → 生成可播放的钢琴 MIDI的完整链路。一、Pop2Piano 是什么Piano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great expertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you can directly generate a cover from a songs audio waveform. It is the first model to directly generate a piano cover from pop audio without melody and chord extraction modules.论文摘要原文Piano covers of pop music are enjoyed by many people. However, the task of automatically generating piano covers of pop music is still understudied. This is partly due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it challenging to apply the latest>pip install pretty-midi0.2.9 essentia2.1b6.dev1034 librosa scipy安装完成后可能需要重启运行环境runtime。依赖的深层原因在于essentia提供RhythmExtractor2013节拍抽取算法见 feature_extraction_pop2piano.pylibrosa负责音频加载与重采样scipy负责节拍插值interp1dpretty_midi负责把 token 渲染成PrettyMIDI对象音符起止、速度、删除非法音符等torch模型推理框架。上述库缺失时对应的类会直接以装饰器方式在导入阶段声明硬依赖Pop2PianoFeatureExtractor声明需要essentia/librosa/scipy/torchPop2PianoTokenizer声明需要pretty_midi/torchPop2PianoProcessor声明需要全部五个见各文件顶部requires。官方提供了一键式高层封装Pop2PianoProcessorprocessing_pop2piano.py它组合了特征提取器与分词器同时支持音频→特征与音符→token两条通路。四、快速上手实战4.1 使用 HuggingFace Dataset 示例from datasets import load_dataset from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor model Pop2PianoForConditionalGeneration.from_pretrained(sweetcocoa/pop2piano, device_mapauto) processor Pop2PianoProcessor.from_pretrained(sweetcocoa/pop2piano) ds load_dataset(sweetcocoa/pop2piano_ci, splittest) inputs processor( audiods[audio][0][array], sampling_rateds[audio][0][sampling_rate], return_tensorspt ) model_output model.generate(input_featuresinputs[input_features], composercomposer1) tokenizer_output processor.batch_decode( token_idsmodel_output, feature_extractor_outputinputs )[pretty_midi_objects][0] tokenizer_output.write(./Outputs/midi_output.mid)batch_decode后得到的BatchEncoding同时包含notes与pretty_midi_objects两类字段见 tokenization_pop2piano.py。取第 0 个PrettyMIDI对象调用.write()即导出.mid文件。4.2 使用自己的音频文件import librosa from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor audio, sr librosa.load(your_audio_file_here, sr44100) # feel free to change the sr to a suitable value. model Pop2PianoForConditionalGeneration.from_pretrained(sweetcocoa/pop2piano, device_mapauto) processor Pop2PianoProcessor.from_pretrained(sweetcocoa/pop2piano) inputs processor(audioaudio, sampling_ratesr, return_tensorspt).to(model.device) model_output model.generate(input_featuresinputs[input_features], composercomposer1) tokenizer_output processor.batch_decode( token_idsmodel_output, feature_extractor_outputinputs )[pretty_midi_objects][0] tokenizer_output.write(./Outputs/midi_output.mid)性能提示加载音频时把采样率设为44.1 kHzsr44100通常能获得不错的生成效果。特征提取器内部若检测到输入采样率与自身目标采样率默认 22050 Hz不一致会调用librosa.core.resampleres_typekaiser_best自动重采样见 feature_extraction_pop2piano.py。4.3 批量处理多个音频文件import librosa from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor # feel free to change the sr to a suitable value. audio1, sr1 librosa.load(your_first_audio_file_here, sr44100) audio2, sr2 librosa.load(your_second_audio_file_here, sr44100) model Pop2PianoForConditionalGeneration.from_pretrained(sweetcocoa/pop2piano, device_mapauto) processor Pop2PianoProcessor.from_pretrained(sweetcocoa/pop2piano) inputs processor(audio[audio1, audio2], sampling_rate[sr1, sr2], return_attention_maskTrue, return_tensorspt).to(model.device) # Since we now generating in batch(2 audios) we must pass the attention_mask model_output model.generate( input_featuresinputs[input_features], attention_maskinputs[attention_mask], composercomposer1, ) tokenizer_output processor.batch_decode( token_idsmodel_output, feature_extractor_outputinputs )[pretty_midi_objects] # Since we now have 2 generated MIDI files tokenizer_output[0].write(./Outputs/midi_output1.mid) tokenizer_output[1].write(./Outputs/midi_output2.mid)批量关键点传入多条音频时sampling_rate必须是与音频一一对应的列表[sr1, sr2]否则Pop2PianoFeatureExtractor.__call__会直接抛出 ValueError见 feature_extraction_pop2piano.py必须显式传return_attention_maskTrue并把attention_mask一并传给model.generate()batch_decode返回的pretty_midi_objects是列表需按索引逐个写出文件。4.4 拆分使用 FeatureExtractor 与 Tokenizer如果希望更细粒度地控制预处理可以不使用Pop2PianoProcessor而分别显式使用Pop2PianoFeatureExtractor与Pop2PianoTokenizerimport librosa from transformers import Pop2PianoFeatureExtractor, Pop2PianoForConditionalGeneration, Pop2PianoTokenizer # feel free to change the sr to a suitable value. audio1, sr1 librosa.load(your_first_audio_file_here, sr44100) audio2, sr2 librosa.load(your_second_audio_file_here, sr44100) model Pop2PianoForConditionalGeneration.from_pretrained(sweetcocoa/pop2piano, device_mapauto) feature_extractor Pop2PianoFeatureExtractor.from_pretrained(sweetcocoa/pop2piano) tokenizer Pop2PianoTokenizer.from_pretrained(sweetcocoa/pop2piano) inputs feature_extractor( audio[audio1, audio2], sampling_rate[sr1, sr2], return_attention_maskTrue, return_tensorspt, ) # Since we now generating in batch(2 audios) we must pass the attention_mask model_output model.generate( input_featuresinputs[input_features], attention_maskinputs[attention_mask], composercomposer1, ) tokenizer_output tokenizer.batch_decode( token_idsmodel_output, feature_extractor_outputinputs )[pretty_midi_objects] # Since we now have 2 generated MIDI files tokenizer_output[0].write(./Outputs/midi_output1.mid) tokenizer_output[1].write(./Outputs/midi_output2.mid)Pop2PianoProcessor.batch_decode本质上是转发调用Pop2PianoTokenizer.batch_decode见 processing_pop2piano.py因此两条路径结果一致。五、预处理流水线深度解析Pop2PianoFeatureExtractorfeature_extraction_pop2piano.py把一段音频变成模型可用的三个输入顺序如下节拍抽取Rhythm调用 essentia 的RhythmExtractor2013(methodmultifeature)返回 BPM、节拍时间点beat_times、置信度、速度估计与节拍区间。该算法仅处理原始音频见extract_rhythm节拍插值Beatsteps用scipy.interpolate.interp1dbounds_errorFalse, fill_valueextrapolate按steps_per_beat默认 2与外推参数对beat_times插值得到细粒度的beatsteps见interpolate_beat_timesMel 预处理与频谱计算preprocess_mel按每num_bars * 4步切分音频片段、统一 pad 到最长片段静音补零并外推一个extrapolated_beatstep供分词器使用mel_spectrogram使用 Hanning 窗 STFT 得到 Mel 频谱窗口大小window_size4096、跳步hop_length1024、Mel 滤波器个数feature_size512、最低频率min_frequency10.0对频谱做np.log(np.clip(mel_specs, a_min1e-6))得到log-mel 频谱。最终__call__返回BatchFeature其中model_input_names [input_features, beatsteps, extrapolated_beatstep]。关于 padding 的坑处理批量输入时pad方法会在每个样本特征之间插入一整行全零数组用于分隔样本因此attention_mask中会出现周期性为 0 的行参见 docstring 中 mask 的分隔示例feature_extraction_pop2piano.pytokenizer.batch_decode正是依据attention_mask第 0 列中的 0 行来切分各样本生成结果见 tokenization_pop2piano.py。单样本且未请求 attention_mask 时该分隔行会被自动移除。特征提取器的核心可调参数如下参数默认值说明sampling_rate22050送入模型的目标采样率推理时应与音频采样率匹配并开启重采样padding_value0填充值对应静音window_size4096傅里叶变换窗口长度样本数hop_length1024相邻窗口步长样本数min_frequency10.0log-mel 频谱使用的最低频率feature_size512特征维度Mel 滤波器个数同时等于模型d_modelnum_bars2决定每个子序列的间隔长度小节数steps_per_beat、resample、return_attention_mask、return_tensors是__call__的运行时参数其中resample推理时必须为True批量输入时return_attention_mask会被强制置True。六、模型配置与类参考6.1 Pop2PianoConfig 全参数配置类Pop2PianoConfigconfiguration_pop2piano.py核心参数如下括号内为默认值参数默认值说明vocab_size2400解码词汇表大小composer_vocab_size21作曲家数量对应composer_to_feature_token中的 composer 数d_model512隐藏层维度d_kv64注意力 Q/K/V 投影维度d_ff2048FFN 中间层维度num_layers6编码器层数解码器默认取同值见num_decoder_layersnum_heads8注意力头数relative_attention_num_buckets32每层相对位置注意力使用的桶bucket数量relative_attention_max_distance128桶划分的最大相对距离dropout_rate0.1Dropout 比率layer_norm_epsilon1e-6LayerNorm 的 epsiloninitializer_factor1.0初始化因子feed_forward_projgated-geluFFN 类型可选relu或gated-geludense_act_fnreluDenseActDense与DenseGatedActDense中的激活函数is_encoder_decoderTrue编码器-解码器架构标志pad_token_id/eos_token_id0 / 1特殊 token idbos 为 2unk 为 -1tie_word_embeddingsTrue权重绑定几点源码细节值得注意attribute_map将num_hidden_layers→num_layers、hidden_size→d_model、num_attention_heads→num_heads做了别名映射兼容 T5 系命名习惯configuration_pop2piano.py__post_init__中根据feed_forward_proj是否以gated开头自动推断is_gated_act由于官方 checkpoint 只存shared.weight权重实际始终绑定scale_decoder_outputs由tie_word_embeddings决定与 T5 相同的处理方式configuration_pop2piano.pykeys_to_ignore_at_inference [past_key_values]避免推理时 KV cache 相关键干扰。6.2 模型与生成Pop2PianoForConditionalGenerationmodeling_pop2piano.py整体入口包含共享嵌入shared、mel_conditioner、T5 风格编码器与解码器、lm_head。generate(input_features, attention_maskNone, composercomposer1, generation_configNone, **kwargs)modeling_pop2piano.pyPop2Piano 定制的生成入口。input_features需是(batch, seq_len, feature_dim)的张量传入 composer 名称时会自动从generation_config.composer_to_feature_token解析 composer token并在批量输入时处理对应的 mask 拼接。forward(...)同时接受input_ids训练时给解码器用与input_features返回标准Seq2SeqLMOutput训练阶段标签可直接喂给labels配对的prepare_decoder_input_ids_from_labels会自动做_shift_right构造解码器输入。模型权重可通过 convert_pop2piano_weights_to_hf.py 将官方仓库sweetcocoa/pop2piano的原始状态字典转换为 HF 格式逐层映射 encoder/decoder 的相对注意力偏置、嵌入、mel_conditioner.embedding、lm_head等。七、测试验证与仓库证据仓库针对该模型提供了四套独立测试可作为行为契约参考tests/models/pop2piano/test_modeling_pop2piano.py模型前向与生成测试。slow 测试从sweetcocoa/pop2piano加载真实权重执行generate并断言输出sequences.ndim 2同时覆盖多 batch 自定义composer及 attention_mask 的生成路径。tests/models/pop2piano/test_feature_extraction_pop2piano.py验证特征提取器对单/多音频的input_features、beatsteps、extrapolated_beatstep输出与各种 padding/mask 行为。tests/models/pop2piano/test_tokenization_pop2piano.py验证 notes/token 互转、词汇表加载与 decode 一致性。tests/models/pop2piano/test_processing_pop2piano.py验证Pop2PianoProcessor对音频/音符双通路的组合与分发。八、使用建议与已知边界以下结论均来自文档或源码可以确认的事实采样率加载音频建议使用 44.1 kHz特征提取器目标采样率默认 22050 Hz不一致时会自动用kaiser_best插值重采样也可自行调低特征提取器sampling_rate减少输入长度。Composer 选择generate(..., composercomposerX)中可用的 composer 集合取决于模型generation_config.json的composer_to_feature_token传入不存在名称会报错并打印可用列表。切换 composer 是获得不同编曲风格最直接的手段。批量推理多音频必须逐个传入采样率列表并返回 attention_mask解码依赖 mask 中的零行分隔各样本缺少 mask 会触发显式 ValueError。训练侧重模型主要针对韩国流行乐K-Pop训练但对西方流行乐、Hip Hop 等也表现不错——这是官方文档给出的经验描述具体效果请以实际试听为准。Token 后处理notes_to_midi会以resolution384, initial_tempo120.0初始化PrettyMIDI用program0Acoustic Grand Piano承载音符并调用remove_invalid_notes()清理非法音符tokenization_pop2piano.py。生成的 MIDI 可直接用任意播放器或 DAW 打开试听。轻量推理起点若想快速验证链路仓库慢测试使用随机特征张量(batch, seq, 512)也能驱动generatetest_modeling_pop2piano.py可作为自建流水线冒烟测试的模板。结合本文的配置表、四段可运行示例与源码级原理说明你现在可以基于 Transformers 从任意流行音频出发探索音频 → 钢琴翻弹 MIDI的端到端生成并通过 composer 参数解锁更多编曲风格。【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考