{"id":18335,"date":"2025-01-14T18:24:39","date_gmt":"2025-01-14T10:24:39","guid":{"rendered":"https:\/\/www.aisharenet.com\/?p=18335"},"modified":"2025-01-14T18:24:39","modified_gmt":"2025-01-14T10:24:39","slug":"joygen","status":"publish","type":"post","link":"https:\/\/www.kdjingpai.com\/ja\/joygen\/","title":{"rendered":"JoyGen\uff1a\u97f3\u9891\u9a71\u52a8\u76843D\u6df1\u5ea6\u611f\u77e5\u4eba\u50cf\u8bf4\u8bdd\u89c6\u9891\u7f16\u8f91\u5de5\u5177"},"content":{"rendered":"<p>JoyGen\u662f\u4e00\u4e2a\u521b\u65b0\u7684\u4e24\u9636\u6bb5\u8bf4\u8bdd\u4eba\u8138\u89c6\u9891\u751f\u6210\u6846\u67b6\uff0c\u4e13\u6ce8\u4e8e\u89e3\u51b3\u97f3\u9891\u9a71\u52a8\u7684\u9762\u90e8\u8868\u60c5\u751f\u6210\u95ee\u9898\u3002\u8be5\u9879\u76ee\u7531\u4eac\u4e1c\u79d1\u6280\u56e2\u961f\u5f00\u53d1\uff0c\u91c7\u7528\u5148\u8fdb\u76843D\u91cd\u5efa\u6280\u672f\u548c\u97f3\u9891\u7279\u5f81\u63d0\u53d6\u65b9\u6cd5\uff0c\u80fd\u591f\u51c6\u786e\u6355\u6349\u8bf4\u8bdd\u4eba\u7684\u8eab\u4efd\u7279\u5f81\u548c\u8868\u60c5\u7cfb\u6570\uff0c\u5b9e\u73b0\u9ad8\u8d28\u91cf\u7684\u5507\u5f62\u540c\u6b65\u548c\u89c6\u89c9\u5408\u6210\u3002JoyGen\u6846\u67b6\u5305\u542b\u4e24\u4e2a\u4e3b\u8981\u9636\u6bb5\uff1a\u9996\u5148\u662f\u57fa\u4e8e\u97f3\u9891\u7684\u5507\u90e8\u8fd0\u52a8\u751f\u6210\uff0c\u5176\u6b21\u662f\u89c6\u89c9\u5916\u89c2\u5408\u6210\u3002\u901a\u8fc7\u6574\u5408\u97f3\u9891\u7279\u5f81\u548c\u9762\u90e8\u6df1\u5ea6\u56fe\uff0c\u4e3a\u7cbe\u786e\u7684\u5507\u97f3\u540c\u6b65\u63d0\u4f9b\u5168\u9762\u7684\u76d1\u7763\u3002\u8be5\u9879\u76ee\u4e0d\u4ec5\u652f\u6301\u4e2d\u6587\u548c\u82f1\u6587\u97f3\u9891\u9a71\u52a8\uff0c\u8fd8\u63d0\u4f9b\u4e86\u5b8c\u6574\u7684\u8bad\u7ec3\u548c\u63a8\u7406pipeline\uff0c\u662f\u4e00\u4e2a\u529f\u80fd\u5f3a\u5927\u7684\u5f00\u6e90\u5de5\u5177\u3002<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-18336\" title=\"JoyGen\uff1a\u97f3\u9891\u9a71\u52a8\u76843D\u6df1\u5ea6\u611f\u77e5\u4eba\u50cf\u8bf4\u8bdd\u89c6\u9891\u7f16\u8f91\u5de5\u5177-1\" src=\"https:\/\/www.kdjingpai.com\/wp-content\/uploads\/2025\/01\/407d7bea1469eb9.png\" alt=\"JoyGen\uff1a\u97f3\u9891\u9a71\u52a8\u76843D\u6df1\u5ea6\u611f\u77e5\u4eba\u50cf\u8bf4\u8bdd\u89c6\u9891\u7f16\u8f91\u5de5\u5177-1\" width=\"942\" height=\"723\" srcset=\"https:\/\/www.kdjingpai.com\/wp-content\/uploads\/2025\/01\/407d7bea1469eb9.png 942w, https:\/\/www.kdjingpai.com\/wp-content\/uploads\/2025\/01\/407d7bea1469eb9-300x230.png 300w, https:\/\/www.kdjingpai.com\/wp-content\/uploads\/2025\/01\/407d7bea1469eb9-768x589.png 768w, https:\/\/www.kdjingpai.com\/wp-content\/uploads\/2025\/01\/407d7bea1469eb9-16x12.png 16w\" sizes=\"auto, (max-width: 942px) 100vw, 942px\" \/><\/p>\n<p>&nbsp;<\/p>\n<h2>\u529f\u80fd\u5217\u8868<\/h2>\n<ul>\n<li>\u97f3\u9891\u9a71\u52a8\u76843D\u9762\u90e8\u8868\u60c5\u751f\u6210\u548c\u7f16\u8f91<\/li>\n<li>\u7cbe\u786e\u7684\u5507\u5f62-\u97f3\u9891\u540c\u6b65\u6280\u672f<\/li>\n<li>\u652f\u6301\u4e2d\u6587\u548c\u82f1\u6587\u97f3\u9891\u8f93\u5165<\/li>\n<li>3D\u6df1\u5ea6\u611f\u77e5\u7684\u89c6\u89c9\u5408\u6210<\/li>\n<li>\u9762\u90e8\u8eab\u4efd\u7279\u5f81\u4fdd\u6301\u529f\u80fd<\/li>\n<li>\u9ad8\u8d28\u91cf\u89c6\u9891\u751f\u6210\u548c\u7f16\u8f91\u80fd\u529b<\/li>\n<li>\u5b8c\u6574\u7684\u8bad\u7ec3\u548c\u63a8\u7406\u6846\u67b6\u652f\u6301<\/li>\n<li>\u9884\u8bad\u7ec3\u6a21\u578b\u652f\u6301\u5feb\u901f\u90e8\u7f72<\/li>\n<li>\u652f\u6301\u81ea\u5b9a\u4e49\u6570\u636e\u96c6\u8bad\u7ec3<\/li>\n<li>\u63d0\u4f9b\u8be6\u7ec6\u7684\u6570\u636e\u9884\u5904\u7406\u5de5\u5177<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h2>\u4f7f\u7528\u5e2e\u52a9<\/h2>\n<h3>1. \u73af\u5883\u914d\u7f6e<\/h3>\n<h4>1.1 \u57fa\u7840\u73af\u5883\u8981\u6c42<\/h4>\n<ul>\n<li>\u652f\u6301\u7684GPU\uff1aV100, A800<\/li>\n<li>Python\u7248\u672c\uff1a3.8.19<\/li>\n<li>\u7cfb\u7edf\u4f9d\u8d56\uff1affmpeg<\/li>\n<\/ul>\n<h4>1.2 \u5b89\u88c5\u6b65\u9aa4<\/h4>\n<ol>\n<li>\u521b\u5efa\u5e76\u6fc0\u6d3bconda\u73af\u5883\uff1a<\/li>\n<\/ol>\n<pre><code>conda create -n joygen python=3.8.19 ffmpeg\r\nconda activate joygen\r\npip install -r requirements.txt\r\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u5b89\u88c5Nvdiffrast\u5e93\uff1a<\/li>\n<\/ol>\n<pre><code>git clone https:\/\/github.com\/NVlabs\/nvdiffrast\r\ncd nvdiffrast\r\npip install .\r\n<\/code><\/pre>\n<ol start=\"3\">\n<li>\u4e0b\u8f7d\u9884\u8bad\u7ec3\u6a21\u578b<br \/>\n\u4ece\u63d0\u4f9b\u7684<a href=\"https:\/\/drive.google.com\/file\/d\/1kvGsljFRnXKUK_ETdd49jJy8DbdgZKkE\">\u4e0b\u8f7d\u94fe\u63a5<\/a>\u83b7\u53d6\u9884\u8bad\u7ec3\u6a21\u578b\uff0c\u5e76\u6309\u7167\u6307\u5b9a\u76ee\u5f55\u7ed3\u6784\u653e\u7f6e\u5728<code>.\/pretrained_models\/<\/code>\u76ee\u5f55\u4e0b\u3002<\/li>\n<\/ol>\n<h3>2. \u4f7f\u7528\u6d41\u7a0b<\/h3>\n<h4>2.1 \u63a8\u7406\u8fc7\u7a0b<\/h4>\n<p>\u6267\u884c\u5b8c\u6574\u63a8\u7406pipeline\uff1a<\/p>\n<pre><code>bash scripts\/inference_pipeline.sh \u97f3\u9891\u6587\u4ef6 \u89c6\u9891\u6587\u4ef6 \u7ed3\u679c\u76ee\u5f55\r\n<\/code><\/pre>\n<p>\u5206\u6b65\u6267\u884c\u63a8\u7406\u8fc7\u7a0b\uff1a<\/p>\n<ol>\n<li>\u4ece\u97f3\u9891\u63d0\u53d6\u9762\u90e8\u8868\u60c5\u7cfb\u6570\uff1a<\/li>\n<\/ol>\n<pre><code>python inference_audio2motion.py --a2m_ckpt .\/pretrained_models\/audio2motion\/240210_real3dportrait_orig\/audio2secc_vae --hubert_path .\/pretrained_models\/audio2motion\/hubert --drv_aud .\/demo\/xinwen_5s.mp3 --seed 0 --result_dir .\/results\/a2m --exp_file xinwen_5s.npy\r\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u4f7f\u7528\u65b0\u7684\u8868\u60c5\u7cfb\u6570\u9010\u5e27\u6e32\u67d3\u6df1\u5ea6\u56fe\uff1a<\/li>\n<\/ol>\n<pre><code>python -u inference_edit_expression.py --name face_recon_feat0.2_augment --epoch=20 --use_opengl False --checkpoints_dir .\/pretrained_models --bfm_folder .\/pretrained_models\/BFM --infer_video_path .\/demo\/example_5s.mp4 --infer_exp_coeff_path .\/results\/a2m\/xinwen_5s.npy --infer_result_dir .\/results\/edit_expression\r\n<\/code><\/pre>\n<ol start=\"3\">\n<li>\u57fa\u4e8e\u97f3\u9891\u7279\u5f81\u548c\u9762\u90e8\u6df1\u5ea6\u56fe\u751f\u6210\u9762\u90e8\u52a8\u753b\uff1a<\/li>\n<\/ol>\n<pre><code>CUDA_VISIBLE_DEIVCES=0 python -u inference_joygen.py --unet_model_path pretrained_models\/joygen --vae_model_path pretrained_models\/sd-vae-ft-mse --intermediate_dir .\/results\/edit_expression --audio_path demo\/xinwen_5s.mp3 --video_path demo\/example_5s.mp4 --enable_pose_driven --result_dir results\/talk --img_size 256 --gpu_id 0\r\n<\/code><\/pre>\n<h4>2.2 \u8bad\u7ec3\u8fc7\u7a0b<\/h4>\n<ol>\n<li>\u6570\u636e\u9884\u5904\u7406\uff1a<\/li>\n<\/ol>\n<pre><code>python -u preprocess_dataset.py --checkpoints_dir .\/pretrained_models --name face_recon_feat0.2_augment --epoch=20 --use_opengl False --bfm_folder .\/pretrained_models\/BFM --video_dir .\/demo --result_dir .\/results\/preprocessed_dataset\r\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u68c0\u67e5\u9884\u5904\u7406\u6570\u636e\u5e76\u751f\u6210\u8bad\u7ec3\u5217\u8868\uff1a<\/li>\n<\/ol>\n<pre><code>python -u preprocess_dataset_extra.py data_dir\r\n<\/code><\/pre>\n<ol start=\"3\">\n<li>\u5f00\u59cb\u8bad\u7ec3\uff1a<br \/>\n\u4fee\u6539config.yaml\u6587\u4ef6\u540e\u6267\u884c\uff1a<\/li>\n<\/ol>\n<pre><code>accelerate launch --main_process_port 29501 --config_file config\/accelerate_config.yaml train_joygen.py<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>JoyGen\u662f\u4e00\u4e2a\u521b\u65b0\u7684\u4e24\u9636\u6bb5\u8bf4\u8bdd\u4eba\u8138\u89c6\u9891\u751f\u6210\u6846\u67b6\uff0c\u4e13\u6ce8\u4e8e\u89e3\u51b3\u97f3\u9891\u9a71\u52a8\u7684\u9762\u90e8\u8868\u60c5\u751f\u6210\u95ee\u9898\u3002\u8be5\u9879\u76ee\u7531\u4eac\u4e1c\u79d1\u6280\u56e2\u961f\u5f00\u53d1\uff0c\u91c7\u7528\u5148\u8fdb\u76843D\u91cd\u5efa\u6280\u672f\u548c\u97f3\u9891\u7279\u5f81\u63d0\u53d6\u65b9\u6cd5\uff0c\u80fd\u591f\u51c6\u786e\u6355\u6349\u8bf4\u8bdd\u4eba\u7684\u8eab\u4efd\u7279\u5f81\u548c\u8868\u60c5\u7cfb\u6570\uff0c\u5b9e\u73b0\u9ad8\u8d28\u91cf\u7684\u5507\u5f62\u540c\u6b65\u548c\u89c6\u89c9\u5408\u6210\u3002JoyGe&#8230;<\/p>\n","protected":false},"author":1,"featured_media":61624,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[230,228],"class_list":["post-18335","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tool","tag-aikaiyuanxiangmu","tag-aishuziren"],"_links":{"self":[{"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/posts\/18335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/comments?post=18335"}],"version-history":[{"count":0,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/posts\/18335\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/media\/61624"}],"wp:attachment":[{"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/media?parent=18335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/categories?post=18335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kdjingpai.com\/ja\/wp-json\/wp\/v2\/tags?post=18335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}