Quick Sign In:  

Forum: General Discussion

Topic: How to create own Stem files ("Stem creator" for VDJ) - Page: 4
Again thanks for the plugin.
Another option would be awesome. Since I have a lot of stems pre-made it's quite some work to manualy add them and make stem files out of them. Is it somehow possible to automate this? Maybe via scripting, or via the command line of windows?
 

geposted Fri 14 Oct 22 @ 11:35 pm
Now you can create your own Stems with Stem Creator Tool in virtual DJ. I am doing it and it's amazing

 

geposted Sat 15 Oct 22 @ 5:19 pm
Now since the Stems 2.0 version when I load in a stem mp4 file VDJ is asking to precompute the file. Is this just a bug or is vdj actually doing something to the file. And is the Stem Creator tool when adding just the bass, drums, vocals & Instruments using the stem 2.0 to split the drums into Hihat & Kick?
 

geposted Fri 18 Nov 22 @ 8:53 pm
AdionPRO InfinityCTOMember since 2006
That's probably a bug, will check it
 

geposted Sat 19 Nov 22 @ 5:42 am
JirSimPRO InfinityMember since 2021
Thank you very much for Stems Creator. Till now I used ffmpeg method.

I did some Stems DJ-Research and I have designed "PRODUCERS STEMS V3" (8 channels stem file for VDJ)

I can not add the pictures here and for this reason I have to write it:

My stems concept (4 Producers Only / No realtime stems) :
------------------------------
- 8 CHANNELS STEMS WORKING ON STANDARD HW (5 KNOBS EQ SECTION ON MIXER plus 8 PADs)

- max 8 stereo channels: Bass,Kick,Hihat,Vocal,Fx*,Synth1(Instru1),Synth2(Instru2)*,Percussion(Instru3)*
(*-optional)
- every channel has own Pad (for this reason 8 channels)
- 5 potenciometers EQ section on the mixer needed !!! (Gain,Hi,Mi,Lo,Filter)
- every channel should be mapped to 1 knob
(some channels muss share their knob with another channels, see next chapter STEMsV3 EQ for Techno Music)
- information about knob mapping has to be stored in stems-file to user get it automaticaly

STEMsV3 EQ for Techno Music:
---------------------------------------------------
EQ:
GAIN - Synth1,Synth2,Vocal,Fx (Gain has no "zero" poimt in the middle )
Hi - Hihat
Mi - Percussion
Lo - Kick
Filter - Bass
PADs:
Kick - Perc - Hihat - FX
Bass - Syn1 - Synt2 - Vocal


What do you think about it? I have found out, that I need for some channels Knob+Pad and by some channels is only Pad OK.
For Example:
Vocal - No Vocal
Fx - No Fx (Risers etc.)
 

geposted Sun 20 Nov 22 @ 12:58 pm
This sounds really great! I was trying to figure out ffmpeg, yet I always came accross the issue which you described in this message:
Hi,
JirSim wrote :
I have found a Bug by my own Stems. If only Vocal and Intru play together, then I hear a little bit of Drums too. If I play Instru only or Vocal only is all OK. I have tested my files in VLC and there is all OK too ... Someone with Idea what is wrong? Thank you


I also tried mp4box, but then stem creator came out. But it sounds really interesting to add more channels. Would you be willing to share your method?

JirSim wrote :
Thank you very much for Stems Creator. Till now I used ffmpeg method.

I did some Stems DJ-Research and I have designed "PRODUCERS STEMS V3" (8 channels stem file for VDJ)

I can not add the pictures here and for this reason I have to write it:

My stems concept (4 Producers Only / No realtime stems) :
------------------------------
- 8 CHANNELS STEMS WORKING ON STANDARD HW (5 KNOBS EQ SECTION ON MIXER plus 8 PADs)

- max 8 stereo channels: Bass,Kick,Hihat,Vocal,Fx*,Synth1(Instru1),Synth2(Instru2)*,Percussion(Instru3)*
(*-optional)
- every channel has own Pad (for this reason 8 channels)
- 5 potenciometers EQ section on the mixer needed !!! (Gain,Hi,Mi,Lo,Filter)
- every channel should be mapped to 1 knob
(some channels muss share their knob with another channels, see next chapter STEMsV3 EQ for Techno Music)
- information about knob mapping has to be stored in stems-file to user get it automaticaly

STEMsV3 EQ for Techno Music:
---------------------------------------------------
EQ:
GAIN - Synth1,Synth2,Vocal,Fx (Gain has no "zero" poimt in the middle )
Hi - Hihat
Mi - Percussion
Lo - Kick
Filter - Bass
PADs:
Kick - Perc - Hihat - FX
Bass - Syn1 - Synt2 - Vocal


What do you think about it? I have found out, that I need for some channels Knob+Pad and by some channels is only Pad OK.
For Example:
Vocal - No Vocal
Fx - No Fx (Risers etc.)
 

geposted Mon 21 Nov 22 @ 11:44 pm
JirSimPRO InfinityMember since 2021
Hi,
my concept of the 8 stereo channel stems file for VDJ was suggestion only for VDJteam to implement it. It does not exist yet ...
 

geposted Tue 22 Nov 22 @ 7:40 am
JirSimPRO InfinityMember since 2021
I did my Research in Ableton/Bitwig to find out, what channels DJs need ...
 

geposted Tue 22 Nov 22 @ 7:42 am
JirSim wrote :
2GROOVINDJ

VDJ and Traktor use the same Technology (5 stereo channels in mp4), but different concept to store stems.

Traktor:
1-MASTER
2-Drums*
3-Bass*
4-Synths*
5-Vox*
*-it is not standard, you can store what you want, where you want ...

VDJ:
1-Vocal
2-Hihat
3-Bass
4-Instruments
5-Kick
(VDJ uses this like standard!)

You can use STEMS CREATOR for VDJ too, but you cannot store MASTER channel + you have to add the stems in following way:

1 - Hihat
2 - Bass
3 - Instruments
4 - Kick
5 - Vocal
(Vocal is stored in MASTER)

I tried STEMS CREATOR to make .vdjstems, but I have found a problem, that VDJ uses some kind of audio processing over stems. I.E. if I mute Vocal channel or Instruments channel, than the Drums (Kick channel + Hihat channel) are loader. I think, that it is audio processing for realtime stems (maybe to sound better), that by source stems is not necesseary (and is wrong) ...


to work : read the stem Traktor SDK

your .stem.mp4 should have udta box "stem" to work in Traktor :
moov.udta.stem :

▪ The file must be a valid MP4 file (ISO base media file format: ISO/IEC 14496-12) with the “.mp4” file extension. Preferably the file name should end with “.stem.mp4”, but just “.mp4” is also valid.
▪ The MP4 file must contain a box (i.e. data container) of type stem as a sub-box of moov.udta. moov is the root box of the MP4 file, as shown in the Stem File Structure diagram in section ↑7.1, File Structure.

 

geposted Wed 30 Nov 22 @ 8:44 am
JirSimPRO InfinityMember since 2021
2 OLIVIER REMUX

The Topic 'How to make own stems for VDJ' is already solved.

1.) FFMPEG Method
2.) STEAM CREATOR in VDJ
3.)MP4BOX (I did not try)
 

geposted Sun 04 Dec 22 @ 5:11 pm
Hi,

I've got a bug, where the drums PAD is not canceling the Kick & Clap, while the Hihatis controlling all the drums. Yet, when I click on the DRUM pads I see the kick being canceled out in the Waveform. Somehow the audio routing goes wrong. When filling in the stem creator I do not use the optional Hihat option.
Is there something I could do?
 

geposted Fri 09 Dec 22 @ 11:53 am
Ime GeradeHouse wrote :
Hi,

I've got a bug, where the drums PAD is not canceling the Kick & Clap, while the Hihatis controlling all the drums. Yet, when I click on the DRUM pads I see the kick being canceled out in the Waveform. Somehow the audio routing goes wrong. When filling in the stem creator I do not use the optional Hihat option.
Is there something I could do?


I've tried everything I can think of to fix the bug with the stem creation tool in Virtual DJ, but it continues to cause problems. It's frustrating that there are so few options for troubleshooting or modifying the tool, and it's not clear what's going wrong or how Virtual DJ is processing the stems. I would really appreciate it if there were an update that gave us more control over how the stem creation tool works, such as the ability to choose which engine to use (stems 2.0 or reduced quality), and options for separating drums into individual elements like kick and drums. It would also be helpful to have the option to adjust the quality of the output, as well as the ability to select a folder containing different types of audio files (e.g. bass, drums, vocals, other) and have Virtual DJ automatically recognize and process them into stem files.

In addition, it would be great if Virtual DJ could automatically process multiple folders at once, saving us the time and effort of having to do it manually for each folder. Overall, having more options and control over the stem creation tool would make it much more useful and reliable, and help us create high-quality stem files without having to deal with frustrating bugs and limitations.
 

geposted Mon 12 Dec 22 @ 2:14 pm
AdionPRO InfinityCTOMember since 2006
Tried to see if something went wrong with VDJ 2023 perhaps, but stem creator still seems to work correctly here.
To split a drums stem into kicks and hihat the stems engine does need to work on your computer, so try and make sure that stems are working when loading a regular track to the deck.
 

geposted Tue 13 Dec 22 @ 4:10 pm
Again I come upon a problem. Somehow some files are recognized by virtualdj as stem files and some not. I don't know whether I have changed something in the metadeta so that the files are not. Here is the metadeta of m4a stem file made with the stem creator tool. Is there something wrong so that it's not recognized by VDJ?

Meta-Data Tags:
title: Swirling
artist: GeradeHouse
album: Swirling EP
genre: House
tracknum: 0 / 0
disk: 1 / 1
created: 2022-12-22
tempo: 117
tool: VirtualDJ 2023.7360
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
----: (null)
cover: JPEG File
rate: 0
album_artist: GeradeHouse
sort_album_artist: GeradeHouse
sort_artist: GeradeHouse

# Track 1 Info - ID 1 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Enabled In Movie
Media Samples: 8295
1 UDTA types:
name: mixed track
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100

# Track 2 Info - ID 2 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Disabled In Movie
Media Samples: 8295
1 UDTA types:
name: vocal
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100

# Track 3 Info - ID 3 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Disabled In Movie
Media Samples: 8295
1 UDTA types:
name: hihat
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100

# Track 4 Info - ID 4 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Disabled In Movie
Media Samples: 8295
1 UDTA types:
name: bass
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100

# Track 5 Info - ID 5 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Disabled In Movie
Media Samples: 8295
1 UDTA types:
name: instruments
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100

# Track 6 Info - ID 6 - TimeScale 44100
Media Duration 00:03:12.609
Track has 1 edits: track duration is 00:03:12.563
Track flags: Disabled In Movie
Media Samples: 8295
1 UDTA types:
name: kick
Alternate Group ID 1
Media Type: soun:mp4a
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
RFC6381 Codec Parameters: mp4a.40.2
All samples are sync
Max sample duration: 1024 / 44100
 

geposted Sat 07 Jan 23 @ 4:28 pm
Hey fellow coders, after countless hours and an immense amount of dedication to coding, I've successfully crafted an m4a stem file that operates smoothly within VirtualDJ, ensuring all tracks play perfectly as expected.

This achievement was made possible by using the combination of FFMPEG, utilizing the fdkaac encoder, and MP4Box. This combination enabled me to get the right aac-lc encoding (ffmpeg) and all metadata set correctly with mp4box, which allowed VirtualDJ to recognize the m4a file as a valid stem file.

Is there anyone here interested in learning more about this? I'd be glad to share more about this journey and provide insights on my process.

I really hope that VDJ will enable the possibility to allow more channels than 5 to be playable, so that forexample the following stems could be controled:
1 - vocals
2 - backing vocals
3 - Instruments
4 - Synthesizer
5 - Kick
6 - Clap
7 - Hihat
8 - Bass

Greetingz,
GeradeHouse
 

geposted Wed 24 May 23 @ 11:21 pm
Hello!
Please help me regarding this situation: I use VDJ Stem Creator for generating some m4a file (very good quality). They are playing well in VDJ, BUT when I want to add some lyrics with Video Editor, VDJ ask me to generate stems (again) with his internal stem engine.... I do not want this since I have used Stem Creator with my external tracks... How can VDJ use the original stems file in Video Editor, too?
 

geposted Tue 11 Jul 23 @ 5:10 pm
Ime GeradeHouse wrote :
Hey fellow coders, after countless hours and an immense amount of dedication to coding, I've successfully crafted an m4a stem file that operates smoothly within VirtualDJ, ensuring all tracks play perfectly as expected.

This achievement was made possible by using the combination of FFMPEG, utilizing the fdkaac encoder, and MP4Box. This combination enabled me to get the right aac-lc encoding (ffmpeg) and all metadata set correctly with mp4box, which allowed VirtualDJ to recognize the m4a file as a valid stem file.

Is there anyone here interested in learning more about this? I'd be glad to share more about this journey and provide insights on my process.

I really hope that VDJ will enable the possibility to allow more channels than 5 to be playable, so that forexample the following stems could be controled:
1 - vocals
2 - backing vocals
3 - Instruments
4 - Synthesizer
5 - Kick
6 - Clap
7 - Hihat
8 - Bass

Greetingz,
GeradeHouse


Hi! Sure I am interested. I will make some test with my stems tracks. Also I'm looking for a workflow to add Lyrics and Backgorund image with Video Editor but in the same time to bypass the internal stems analyzer of the VDJ. So, I want to use m4a file (even with Stem Creator) or with MP4BOX and then the VDJ's Video Editor. But at this step, VDJ will ignore the prepared m4a file and make another vdjedit with own analysis... Thank you
 

geposted Mon 17 Jul 23 @ 8:01 am
This script is a streamlined version tailored for processing audio files to generate `.m4a` stem files. Before executing the script, ensure you have the following prerequisites set up:

1. `mp4box` - Essential for creating the final `.m4a` file.
2. `ffmpeg` - Used for audio file processing tasks like mixing and conversion.
3. `fdkaac` - An AAC codec required for the conversion of `.wav` files to `.aac` format, which `ffmpeg` should be compiled with.

Once you've ensured the above tools are installed and accessible in your system's PATH, you can proceed with the script. Make sure to set the paths to these tools correctly in the script if they're different from the defaults.

import os
import subprocess
import tempfile

mp4box_exe_path = r"C:\Program Files\GPAC\mp4box.exe"
ffmpeg_exe_path = r"C:\FFMPEG\ffmpeg.exe"
fdkaac_exe_path = r"C:\FFMPEG\fdkaac.exe"

folder = r"C:\demucs_seperated\test\08~Shamela~Amor de Colegio~I Dance Cuban Salsa 2016 (Salsa y Timba Hits)~ITAC21600150"

track_names = ["mixed track", "vocal", "hihat", "bass", "instruments", "kick"]

file_mapping = {
"drums-main.wav": "kick.wav",
"drums-hihats.wav": "hihat.wav",
"other.wav": "instruments.wav",
"vocals.wav": "vocal.wav",
}

def get_standard_filename(filename):
return file_mapping.get(filename.lower(), filename.lower())

def process_folder(folder):
input_files = {
"kick.wav": None,
"instruments.wav": None,
"vocal.wav": None,
"bass.wav": None,
"hihat.wav": None,
}

for file in os.listdir(folder):
if not file.lower().endswith(".wav"):
continue
standard_filename = get_standard_filename(file)
if standard_filename in input_files:
input_files[standard_filename] = os.path.join(folder, file)

return input_files

def create_mixed_wav_file(input_files, temp_output_folder, ffmpeg_exe_path):
mixed_wav_file = os.path.join(temp_output_folder, "temp_mixed.wav")
input_files["mixed_track"] = mixed_wav_file
mix_cmd = [ffmpeg_exe_path, "-y"]
for input_file in list(input_files.values())[:-1]:
mix_cmd.extend(["-i", input_file])
mix_cmd.extend([
"-filter_complex", "amix=inputs={}[mixed];[mixed]volume=5[out]".format(len(input_files) - 1),
"-map", "[out]",
"-acodec", "pcm_f32le",
mixed_wav_file,
])
subprocess.run(mix_cmd, check=True)
return mixed_wav_file


def convert_wav_files_to_aac(input_files, track_names, temp_output_folder, fdkaac_exe_path):
temp_aac_files = []

def convert_wav_to_aac(input_file, track_name):
temp_aac_file = os.path.join(temp_output_folder, f"temp_{track_name}.aac")
temp_aac_files.append(temp_aac_file)

# Check if the AAC file already exists
if os.path.exists(temp_aac_file):
print(f"{track_name}.aac already exists, skipping conversion")
return

fdkaac_cmd = [
fdkaac_exe_path,
"-p", "2", # AAC-LC (Low Complexity) profile
"-b", "320k", # Set the bitrate to 320 kbps
"-m", "0", # Set the bitrate mode to CBR
"--afterburner", "1", # Enable the Afterburner feature
"-w", "20000", # Set the frequency bandwidth to 20000 Hz (optional)
"-o", temp_aac_file,
input_file, # Use the input file directly
]
result = subprocess.run(fdkaac_cmd, stderr=subprocess.PIPE, text=True)
if result.returncode != 0:
print("Error converting to AAC:")
print(result.stderr)
exit(1)

print("Start the process of converting each WAV file to AAC")
for input_file, track_name in zip(input_files, track_names):
convert_wav_to_aac(input_file, track_name)
print("Process of converting each WAV file to AAC is finished")

for temp_aac_file in temp_aac_files:
if not os.path.isfile(temp_aac_file):
print(f"Error: temp_aac_file '{temp_aac_file}' not found!")
exit(1)

return temp_aac_files



def create_m4a_with_ffmpeg(temp_output_folder, temp_aac_files, ffmpeg_exe_path):
print("Start the process of creating an M4A file with FFmpeg (major brand isom)")

temp_ffmpeg_file = os.path.join(temp_output_folder, "temp_ffmpeg_m4a.m4a")

create_m4a_with_ffmpeg_cmd = [
ffmpeg_exe_path,
"-i", temp_aac_files[0], # kick
"-i", temp_aac_files[1], # Other
"-i", temp_aac_files[2], # Vocals
"-i", temp_aac_files[3], # bass
"-i", temp_aac_files[4], # Hihat
"-i", temp_aac_files[5], # mixed track
"-map", "5:a", # mixed track
"-map", "2:a", # Vocals
"-map", "4:a", # Hihat
"-map", "3:a", # bass
"-map", "1:a", # Other
"-map", "0:a", #
"-c:a", "copy",
"-disposition:a:0", "default", # mixed track, default
"-disposition:a:1", "0", # vocals, non-default
"-disposition:a:2", "0", # hihat, non-default
"-disposition:a:3", "0", # bass, non-default
"-disposition:a:4", "0", # vocals, non-default
"-disposition:a:5", "0", # other, non-default
"-metadata", "title=virtualdj",
"-metadata", "artist=output",
"-brand", "isom",
temp_ffmpeg_file,
]

subprocess.run(create_m4a_with_ffmpeg_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print("FFmpeg command finished.")
return temp_ffmpeg_file

# definition of the function that will run mp4box to create the final m4a file
def run_mp4box(output_folder, temp_ffmpeg_file, track_names=None):
if track_names is None:
track_names = ["mixed track", "vocal", "hihat", "bass", "instruments", "kick"]

mp4box_file = os.path.join(output_folder, f"{os.path.basename(output_folder)}-custom.m4a")
Mp4Box_cmd = ["MP4Box"]

# Use the -udta flag to update track names
for i, track_name in enumerate(track_names):
track_index = i + 1
Mp4Box_cmd.extend(["-udta", f"{track_index}:type=name", f"-udta", f"{track_index}:type=name:str={track_name}"])

# Create a temporary file with metadata
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix=".txt") as metadata_file:
metadata_file.write("tool=VirtualDJ 2023.7544\n")
metadata_file.write("created=0\n")
metadata_file.write("tempo=127\n")
metadata_file.write("INITIALKEY=F\n")
metadata_file.write("rate=0\n")
metadata_file.flush() # Ensure the content is written to the file

Mp4Box_cmd.extend([
"-itags",
metadata_file.name,
])
Mp4Box_cmd.extend([
"-flat",
])
Mp4Box_cmd.extend([
"-brand", "isom:512",
])
Mp4Box_cmd.extend([
"-rb", "mp42",
])
Mp4Box_cmd.extend([
"-ab", "mp41",
])

Mp4Box_cmd.extend(["-out", mp4box_file, temp_ffmpeg_file])
result = subprocess.run(Mp4Box_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
if result.returncode != 0:
print("Error running MP4Box command:")
print(result.stderr)
exit(1)
else:
pass

def remove_temp_files(file_paths):
for file_path in file_paths:
try:
os.remove(file_path)
print(f"Successfully deleted: {file_path}")
except OSError as e:
print(f"Error deleting file {file_path}: {e}")

def main():
input_files = process_folder(folder)
temp_output_folder = os.path.join(folder, "temp_output")
os.makedirs(temp_output_folder, exist_ok=True)

mixed_wav_file = create_mixed_wav_file(input_files, temp_output_folder, ffmpeg_exe_path)
temp_aac_files = convert_wav_files_to_aac(list(input_files.values()), track_names, temp_output_folder, fdkaac_exe_path)
temp_ffmpeg_file = create_m4a_with_ffmpeg(temp_output_folder, temp_aac_files, ffmpeg_exe_path)
run_mp4box(folder, temp_ffmpeg_file, track_names)
remove_temp_files([temp_ffmpeg_file, *temp_aac_files, mixed_wav_file])

if __name__ == "__main__":
main()
 

geposted Sat 16 Sep 23 @ 3:11 am
This code is a script that also performs the audio stem generation process. It uses multi-threading to process audio files in parallel, and uses the `asyncio` library for asynchronous execution.

The script reads a weights file that contains information about the progress of previous executions and the timing of different steps in the process. It also sets up a progress bar to track the overall progress of the audio stem generation process.

The main function, `stem_builder`, runs the stem generation process by iterating over the folders to process and creating a futures task for each folder. Each task executes the `process_folder_with_files` function, which processes the audio files in a folder and moves the resulting stem file to a specified destination folder. The progress of each folder is tracked using a separate progress bar, and the weights and timing information are updated accordingly.

The script also includes utility functions for handling audio files, managing threads and locks, and handling exceptions. Additionally, it includes a function to interrupt the stem generation process by pressing a stop button.

mp4box_exe_path = r"C:\Program Files\GPAC\mp4box.exe"
ffmpeg_exe_path = r"C:\FFMPEG\ffmpeg.exe"
fdkaac_exe_path = r"C:\FFMPEG\fdkaac.exe"

root_directory = "C:\\demucs_seperated"
missing_files_directory = "C:\\demucs_MISSING_FILES"

processing_folder_names = [
"Prunk",
]


# Built-in libraries
import os
import sys
import subprocess
import threading
import tempfile
import json
import time
import re
import uuid
import traceback
from concurrent.futures import ThreadPoolExecutor, as_completed
from contextlib import contextmanager
from collections import defaultdict
from typing import Dict, Optional
from json.decoder import JSONDecodeError
from threading import Lock

# Third-party libraries
import numpy as np
from tqdm.notebook import tqdm
from tqdm.contrib.concurrent import thread_map
import soundfile as sf
import glob
from mutagen.flac import FLAC, Picture
from mutagen.mp4 import MP4, MP4Cover
import base64
import logging


# Local application/library specific imports
import shutil

from fastprogress.fastprogress import master_bar, progress_bar

@contextmanager
def timed_block(name, times):
start_time = time.perf_counter()
yield
end_time = time.perf_counter()
elapsed_time = end_time - start_time
times[name] = elapsed_time

track_names = ["mixed track", "vocal", "hihat", "bass", "instruments", "kick"]

file_mapping = {
"drums-main.wav": ["drums-main.wav", "kick.wav"],
"drums-hihats.wav": ["drums-hihats.wav", "hihat.wav"],
"other.wav": ["other.wav", "instruments.wav"],
"vocals.wav": ["vocals.wav", "vocal.wav"],
}

print_lock = Lock()
parent_pbar_lock = Lock()

field_map = {
"ALBUM": "©alb",
"ALBUMARTIST": "aART",
"ALBUM ARTIST": "aART",
"ALBUMARTISTSORT": "soaa",
"ALBUMSORT": "soal",
"ARTISTSORT": "soar",
"METADATA_BLOCK_PICTURE": "covr",
"CATALOGNUMBER": "----:com.apple.iTunes:CATALOGNUMBER",
"COMMENT": "©cmt",
"COMPILATION": "cpil",
"COMPOSERSORT": "soco",
"CONDUCTOR": "cond",
"COPYRIGHT": "cprt",
"GENRE": "©gen",
"GROUPING": "©grp",
"INITIALKEY": "----:com.apple.iTunes:initialkey",
"KEY": "----:com.apple.iTunes:initialkey",
"ISRC": "----:com.apple.iTunes:ISRC",
"KEY": "----:com.apple.iTunes:initialkey",
"LANGUAGE": "©cmt",
"LYRICIST": "----:com.apple.iTunes:LYRICIST",
"LYRICS": "©lyr",
"MEDIA": "----:com.apple.iTunes:MEDIA",
"MOOD": "----:com.apple.iTunes:MOOD",
"MOVEMENTNAME": "©mvn",
"MOVEMENT": "©mvi",
"MOVEMENTNUMBER": "©mvi",
"MOVEMENTTOTAL": "©mvc",
"ORIGINALALBUM": "----:com.apple.iTunes:ORIGINALALBUM",
"ORIGINALARTIST": "©ope",
"PRODUCER": "----:com.apple.iTunes:PRODUCER",
"LABEL": ["©grp", "----:com.apple.iTunes:LABEL"],
"PUBLISHER": ["©grp", "----:com.apple.iTunes:LABEL"],
"MIXARTIST": "----:com.apple.iTunes:REMIXER",
"REMIXER": "©wrt",
"ARTIST": "©ART",
"TITLE": "©nam",
"TITLESORT": "sonm",
"URL": "©url",
"WEBSITE": "©url",
"WORK": "©wrk",
"DATE": "©day",
"YEAR": "©day",
}

def find_folders_to_process(root_directory, processing_folder_names, folder_times):
folders_to_process = []
for folder_name in processing_folder_names:
processing_folder = os.path.join(root_directory, folder_name)
if os.path.exists(processing_folder):
for root, subdirs, _ in os.walk(processing_folder):
for subdir in subdirs:
full_subdir_path = os.path.join(root, subdir)
if (
os.path.isdir(full_subdir_path)
and full_subdir_path not in folder_times
):
folders_to_process.append(full_subdir_path)
else:
print(
f"Skipping {full_subdir_path} since it has already been processed."
)
return folders_to_process

def load_initial_weights(weights_file, root_directory, processing_folder_names):
folders = []
for folder_name in processing_folder_names:
folders.append(os.path.join(root_directory, folder_name))

if not os.path.exists(weights_file):
default_data = {
"weights": {
"create_m4a_from_wavs": 1,
"run_mp4box": 1,
"remove_temp_files": 1,
},
"folder_times": {},
"processed_items": [],
}
with open(weights_file, "w") as f:
json.dump(default_data, f)

with open(weights_file, "r") as f:
data = json.load(f)

weights = data.get("weights", {})
folder_times = data.get("folder_times", {})

for folder in folders:
if folder not in folder_times:
folder_times[folder] = {
"create_m4a_from_wavs": 75,
"run_mp4box": 2,
"remove_temp_files": 0.05,
}
return weights, folder_times


def analyze_and_update_weights(folder, times):
total_times = defaultdict(float)
total_counts = defaultdict(int)
for key, value in times.items():
total_times[key] += value
total_counts[key] += 1
average_times = {
key: total_times[key] / total_counts[key] for key in total_times.keys()
}

weights = {}

for key, value in average_times.items():
if key not in weights:
weights[key] = value
else:
weights[key] = 0.9 * weights[key] + 0.1 * value

# directly update the input times dictionary
times.update(average_times)

updated_data = {
"weights": weights,
"folder_times": times, # use the updated times dictionary
"processed_items": [folder],
}

return weights, times, updated_data

def check_audio_file(path):
try:
with sf.SoundFile(path) as f:
return True
except:
return False


def safe_move(src, dst, max_retries=5, delay=1):
retries = 0
while retries < max_retries:
try:
shutil.move(src, dst)
break # If the move is successful, break out of the loop
except Exception as e:
print(f"Error moving file: {e}")
time.sleep(delay)
retries += 1
else: # If the loop completed and max_retries were hit, print an error
print(f"Failed to move {src} to {dst} after {max_retries} attempts.")


def check_original_files(folder):
original_files = {
"drums-main.wav": "kick.wav",
"drums-hihats.wav": "hihat.wav",
"other.wav": "instruments.wav",
"vocals.wav": "vocal.wav",
"bass.wav": "bass.wav",
}

if not os.path.isdir(folder):
print(f"{folder} is not a valid directory path. Skipping folder.")
return None, False # Skip this folder by returning None and False

# Check if all original_files are present
found_files = {}
for primary, secondary in original_files.items():
primary_path = os.path.join(folder, primary)
secondary_path = os.path.join(folder, secondary)
if os.path.exists(primary_path) and check_audio_file(primary_path):
found_files[primary] = primary_path
elif os.path.exists(secondary_path) and check_audio_file(secondary_path):
found_files[primary] = secondary_path
else:
print(
f"Skipping folder {folder} due to missing or unplayable original file: {primary} or {secondary}"
)
destination = folder.replace(root_directory, missing_files_directory)
print(f"Moving {folder} to {destination}")
safe_move(folder, destination)
return None, False # Skip this folder by returning None and False

# Check for .flac file
flac_filename = f"{os.path.basename(folder)}.flac"
flac_file_path = os.path.join(folder, flac_filename)
if not os.path.exists(flac_file_path) or not check_audio_file(flac_file_path):
print(
f"Skipping folder {folder} due to missing or unplayable .flac file: {flac_filename}"
)
destination = folder.replace(root_directory, missing_files_directory)
print(f"Moving {folder} to {destination}")
safe_move(folder, destination)
return None, False

found_files["flac"] = flac_file_path

return found_files, True # Return the found_files dictionary and True


def get_standard_filename(filename):
filename = filename.lower()
if filename.endswith("_amplified.wav"):
filename = filename.replace(
"_amplified.wav", ".wav"
) # Get the non-amplified filename
return file_mapping.get(filename, filename)


class AmplificationError(Exception):
pass

def amplify_wav_file(filepath, output_filepath, factor=3.2):
"""
Amplify the volume of a .wav file using ffmpeg.

Args:
filepath (str): Path to the .wav file.
output_filepath (str): Path to save the amplified .wav file.
factor (int): Amplification factor. Defaults to 3.

Returns:
str: Path to the amplified .wav file.
"""
ffmpeg_exe_path = r"C:\FFMPEG\ffmpeg.exe"
ffmpeg_cmd = [
ffmpeg_exe_path,
"-i",
filepath,
"-loglevel",
"48",
"-report",
"-filter:a",
f"volume={factor}",
"-sample_fmt",
"flt",
"-acodec",
"pcm_f32le",
output_filepath,
]

# Change the working directory to the output files directory
original_working_directory = os.getcwd()
os.chdir(os.path.dirname(output_filepath))
try:
subprocess.run(ffmpeg_cmd, check=True)
except subprocess.CalledProcessError:
print("Failed to amplify audio file.")
raise AmplificationError("Failed to amplify audio file.")
finally:
# Restore the original working directory
os.chdir(original_working_directory)

return output_filepath


def rollback_files(renamed_files):
# Rollback renamed files by moving them back to their original names
for new_path, original_path in renamed_files:
shutil.move(new_path, original_path)

def process_folder(
folder: str, temp_output_folder: str
) -> Optional[Dict[str, Optional[str]]]:
found_files, is_valid = check_original_files(folder)
if not is_valid:
return None # If the original files check failed, skip this folder

input_files: Dict[str, Optional[str]] = {
"kick_amplified.wav": None,
"instruments.wav": None,
"vocal.wav": None,
"bass.wav": None,
"hihat.wav": None,
}

try:
# Step 1: Rename files
renamed_files = []
for original_file, new_files in file_mapping.items():
for new_file in new_files:
original_path = os.path.join(folder, original_file)
new_path = os.path.join(folder, new_file)
if os.path.exists(original_path):
shutil.move(original_path, new_path)
renamed_files.append((new_path, original_path))

# Step 2: Amplify kick.wav if it exists
kick_files = file_mapping.get("drums-main.wav", [])
for kick_file in kick_files:
kick_file_path = os.path.join(folder, kick_file)
if os.path.exists(kick_file_path):
try:
# Save the amplified file in the temp_output_folder
amplified_path = os.path.join(
temp_output_folder, "kick_amplified.wav"
)
amplify_wav_file(kick_file_path, amplified_path)
input_files["kick_amplified.wav"] = amplified_path
break
except AmplificationError:
print("Failed to amplify kick.wav. Restoring the original file.")
rollback_files(renamed_files)
return input_files

# Step 3: Collect input files
collected_files = []
for file in os.listdir(folder):
if not file.lower().endswith(".wav"):
continue
if file.lower() in input_files:
file_path = os.path.join(folder, file)
collected_files.append(file_path)
input_files[file.lower()] = file_path

return input_files

except Exception as e:
# print the error and name the folder where the error happened in the error message
print(
f"Error occurred in the function process_folder during processing of folder {folder}: {e}"
)
traceback.print_exc()
return None


def validate_paths(
input_files, output_file, ffmpeg_exe_path, fdkaac_exe_path, temp_output_folder
):
for input_file in input_files:
if input_file is None or input_file == "":
print(f"Warning: Input file is None or empty: '{input_file}'!")
continue
try:
with open(input_file, "rb"):
pass
except FileNotFoundError:
print(f"Error: Input file '{input_file}' not found!")
exit(1)
if os.path.isfile(output_file):
print(f"Error: Output file '{output_file}' already exists!")
exit(1)
if not os.path.isfile(ffmpeg_exe_path):
print(f"Error: FFmpeg executable '{ffmpeg_exe_path}' not found!")
exit(1)
if not os.path.isfile(fdkaac_exe_path):
print(f"Error: fdkaac executable '{fdkaac_exe_path}' not found!")
exit(1)
if not os.path.isdir(temp_output_folder):
print(f"Error: Temp output folder '{temp_output_folder}' not found!")
exit(1)
elif not os.access(temp_output_folder, os.W_OK):
print(f"Error: Temp output folder '{temp_output_folder}' not writable!")
exit(1)

def get_wav_duration_ms(wav_file):
with sf.SoundFile(wav_file, "r") as f:
frames = len(f)
rate = f.samplerate
duration_ms = (frames / float(rate)) * 1000
# print(f"Duration of {wav_file}: {duration_ms:.2f} ms")
return duration_ms


def copy_metadata_from_flac_to_m4a(flac_file, m4a_file):
flac = FLAC(flac_file)
m4a = MP4(m4a_file)
extra_info = []

virtual_dj_can_read = {
"tmpo",
"©alb",
"©ART",
"©cmt",
"©wrt",
"©gen",
"©grp",
"©gen",
"©nam",
"trkn",
"©day",
}

for flac_key, m4a_keys in field_map.items():
if flac_key in flac:
if not isinstance(m4a_keys, list):
m4a_keys = [m4a_keys]

for m4a_key in m4a_keys:
try:
if m4a_key in (
"----:com.apple.iTunes:ISRC",
"----:com.apple.iTunes:CATALOGNUMBER",
"----:com.apple.iTunes:LABEL",
"----:com.apple.iTunes:initialkey",
):
m4a[m4a_key] = [value.encode() for value in flac[flac_key]]
elif m4a_key == "©grp":
m4a[m4a_key] = flac[flac_key]
elif flac_key == "GENRE":
m4a[m4a_key] = "; ".join(flac[flac_key])
elif isinstance(flac[flac_key], list):
m4a[m4a_key] = "; ".join(flac[flac_key])
elif flac_key in ("INITIALKEY", "KEY"):
m4a[m4a_key] = flac[flac_key][0]
elif flac_key == "DISCNUMBER":
m4a[m4a_key] = [flac[flac_key][0], "1"]
elif flac_key == "BPM":
m4a[m4a_key] = int(flac[flac_key][0])
else:
m4a[m4a_key] = flac[flac_key]

except Exception as e:
print(
f"Could not set key {m4a_key} with value {flac[flac_key]} due to error: {e}"
)

if m4a_key not in virtual_dj_can_read:
extra_info.append(f"#{flac_key}: {flac[flac_key]}")

# Add extra info to comment tag
if extra_info:
try:
m4a["©cmt"] += " ".join(extra_info)
except KeyError:
m4a["©cmt"] = " ".join(extra_info)

# Handle cover art separately
if "METADATA_BLOCK_PICTURE" in flac:
picture = Picture(base64.b64decode(flac["METADATA_BLOCK_PICTURE"][0]))
cover = MP4Cover(picture.data, picture.type)
m4a["covr"] = [cover]

m4a.save()


def monitor_ffmpeg_progress(
progress_log,
mixed_wav_file,
ffmpeg_progress,
parent_pbar,
normalized_weights,
folders, # Include 'folders' in the arguments list
):
total_duration_ms = get_wav_duration_ms(mixed_wav_file)

prev_percentage = 0
while True:
time.sleep(2) # Check the progress every second
if not os.path.exists(progress_log):
continue
with open(progress_log, "r") as log:
lines = log.readlines()
if not lines:
continue
out_time_ms = None
progress_continue = False
for line in reversed(lines):
if "out_time_ms=" in line:
out_time_ms = int(re.findall(r"\d+", line)[0])
elif "progress=continue" in line:
progress_continue = True
if out_time_ms is not None:
break
if out_time_ms is not None:
out_time_s = out_time_ms / 1000000
progress_percentage = (out_time_s / (total_duration_ms / 1000)) * 100
ffmpeg_progress.n = int(progress_percentage)
ffmpeg_progress.refresh()
delta_progress = progress_percentage - prev_percentage
with parent_pbar_lock:
parent_pbar.update(
normalized_weights["create_m4a_from_wavs"]
* delta_progress
/ len(folders)
)
prev_percentage = progress_percentage
if not progress_continue:
ffmpeg_progress.n = 100
ffmpeg_progress.refresh()
break

def create_m4a_from_wavs(
input_files,
temp_output_folder,
ffmpeg_exe_path,
ffmpeg_progress,
parent_pbar,
normalized_weights,
folders,
folder,
progress_log, # Add progress log parameter
report_log, # Add report log parameter
):
track_order = {
"temp_mixed": 0,
"vocals": 1,
"hihat": 2,
"bass": 3,
"instruments": 4,
"kick": 5,
}

input_files.sort(
key=lambda x: track_order.get(
os.path.splitext(os.path.basename(x))[0].lower(), 6
)
)

temp_ffmpeg_file = os.path.join(temp_output_folder, "temp_ffmpeg_m4a.m4a")
parent_folder = os.path.dirname(temp_output_folder)
# the flac_file is in the same folder as the output file and has the same name as folder but with the .flac extension
flac_file = os.path.join(parent_folder, f"{os.path.basename(parent_folder)}.flac")

os.environ["FFREPORT"] = f"file='{report_log}':level=48"

ffmpeg_cmd = [
ffmpeg_exe_path,
"-loglevel",
"repeat+debug",
"-i",
flac_file,
] # Change here

for input_file in input_files:
if os.path.splitext(os.path.basename(input_file))[0].lower() != "temp_mixed":
ffmpeg_cmd.extend(["-i", input_file])
ffmpeg_cmd.extend(
[
"-map",
"0:a",
"-map",
"5:a",
"-filter:a:5", # Add this line to apply the volume filter to the "other" track
"volume=1dB", # Add this line to set the volume factor for the "other" track
"-map",
"1:a",
"-filter:a:1", # Add this line to apply the volume filter to the "other" track
"volume=1dB", # Add this line to set the volume factor for the "other" track
"-map",
"2:a",
"-filter:a:2", # Add this line to apply the volume filter to the "other" track
"volume=1dB", # Add this line to set the volume factor for the "other" track
"-map",
"3:a",
"-filter:a:3", # Add this line to apply the volume filter to the "other" track
"volume=1dB", # Add this line to set the volume factor for the "other" track
"-map",
"4:a",
"-filter:a:4", # Add this line to apply the volume filter to the "other" track
"volume=1dB", # Add this line to set the volume factor for the "other" track
"-map_metadata",
"0",
"-c:a",
"libfdk_aac",
"-profile:a",
"aac_low",
"-b:a",
"380k",
"-afterburner",
"1",
"-cutoff",
"20000",
]
)
for i in range(6):
if i == 0:
ffmpeg_cmd.extend(["-disposition:a:" + str(i), "default"])
else:
ffmpeg_cmd.extend(["-disposition:a:" + str(i), "0"])
ffmpeg_cmd.extend(
[
"-brand",
"isom",
"-stats_period",
"1",
]
)
ffmpeg_cmd.extend(
[
"-progress",
progress_log,
"-loglevel", # Add the '-loglevel' option to print debug statements in the output
"48",
"-report",
"-y",
temp_ffmpeg_file,
]
)
monitor_thread = threading.Thread(
target=monitor_ffmpeg_progress,
args=(
progress_log,
flac_file, # Change here
ffmpeg_progress,
parent_pbar,
normalized_weights,
folders,
),
)
monitor_thread.start()

# Change the working directory to the input/output files directory
original_working_directory = os.getcwd()
os.chdir(parent_folder)

try:
subprocess.run(ffmpeg_cmd, check=True)
ffmpeg_success = True
except subprocess.CalledProcessError as e:
print(f"FFmpeg subprocess failed: {e}")
ffmpeg_success = False
return None, None, None, ffmpeg_success, None # return None also for flac_file
finally:
monitor_thread.join()
# Restore the original working directory
os.chdir(original_working_directory)

# Return temp_ffmpeg_file, progress_log, report_log, and ffmpeg_success, and flac_file
return temp_ffmpeg_file, progress_log, report_log, ffmpeg_success, flac_file

def run_mp4box(output_folder, temp_ffmpeg_file, track_names, progress=None):
# print(f"temp_ffmpeg_file: {temp_ffmpeg_file}") # add this line
if track_names is None:
track_names = ["mixed track", "vocal", "hihat", "bass", "instruments", "kick"]

mp4box_file = os.path.join(
output_folder, f"{os.path.basename(output_folder)}-custom.m4a"
)
Mp4Box_cmd = ["MP4Box"]

for i, track_name in enumerate(track_names):
track_index = i + 1
Mp4Box_cmd.extend(
[
"-udta",
f"{track_index}:type=name",
f"-udta",
f"{track_index}:type=name:str={track_name}",
]
)

with tempfile.NamedTemporaryFile(
mode="w", delete=False, suffix=".txt"
) as metadata_file:
metadata_file.write("tool=VirtualDJ 2023.7544\n")
metadata_file.write("created=0\n")
metadata_file.write("rate=0\n")
metadata_file.flush() # Ensure the content is written to the file

Mp4Box_cmd.extend(
[
"-itags",
metadata_file.name,
]
)
Mp4Box_cmd.extend(
[
"-flat",
]
)
Mp4Box_cmd.extend(
[
"-brand",
"isom:512",
]
)
Mp4Box_cmd.extend(
[
"-rb",
"mp42",
]
)
Mp4Box_cmd.extend(
[
"-ab",
"mp41",
]
)

Mp4Box_cmd.extend(["-out", mp4box_file, temp_ffmpeg_file])
result = subprocess.run(
Mp4Box_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
)
if result.returncode != 0:
print("Error running MP4Box command:")
print(result.stderr)
exit(1)

return mp4box_file # Add this line at the end of the function

def normalize_weights_based_on_duration(folder_times):
total_duration = sum(
[sum(task_durations.values()) for task_durations in folder_times.values()]
)
task_weight_sum = {}
for folder_task_durations in folder_times.values():
for task, duration in folder_task_durations.items():
task_weight_sum[task] = task_weight_sum.get(task, 0) + duration

return {key: value / total_duration for key, value in task_weight_sum.items()}

def update_parent_pbar_total(
parent_pbar, updated_weights, total_folders, initial_total_progress
):
new_total = sum(updated_weights.values()) * 100 * total_folders
parent_pbar.total = initial_total_progress + (new_total - initial_total_progress)
parent_pbar.refresh()
# print(f'Parent progress bar total updated to {parent_pbar.total}')

import os
import time
import threading

# Create a lock for thread-safe file deletion
lock = threading.Lock()


def delete_file_with_retry(file_path, num_retries=7):
for i in range(num_retries):
with lock:
if not os.path.exists(file_path):
# If the file doesn't exist, consider it a successful deletion
return
try:
os.remove(file_path)
return
except Exception as e:
if i < num_retries - 1: # if this is not the last retry
time.sleep(2) # Wait before next retry
else: # only print error message at the last retry
print(
f"Error deleting file {file_path} after {num_retries} attempts: {e}"
)


import threading

# Create a lock for thread-safety
lock = threading.Lock()


def delete_directory_with_retry(directory_path, num_retries=7):
for i in range(num_retries):
try:
with lock:
# Check if the directory exists before attempting to delete it
if os.path.exists(directory_path):
shutil.rmtree(directory_path)
return
except FileNotFoundError:
# If the directory was already deleted, treat it as a successful deletion
return
except Exception as e:
if i < num_retries - 1: # if this is not the last retry
time.sleep(2) # Wait before next retry
else: # only print error message at the last retry
print(
f"[Thread: {threading.current_thread().name}] Error deleting directory {directory_path} after {num_retries} attempts: {e}"
)

import os
import shutil
import threading
import traceback


import time
import threading

# Create a lock for thread-safety
lock = threading.Lock()


def move_to_vdjstems(src_folder, dest_folder, max_retries=3, delay_between_retries=3):
# Ensure the destination directory exists
with lock:
if not os.path.exists(dest_folder):
os.makedirs(dest_folder)

for file_name in os.listdir(src_folder):
src_file = os.path.join(src_folder, file_name)
dest_file = os.path.join(dest_folder, file_name)

# Retry mechanism
for attempt in range(max_retries): # type: ignore
try:
# Move the file
with lock:
if os.path.exists(src_file):
shutil.move(src_file, dest_file)
# If the operation is successful, break out of the retry loop
break
except (FileNotFoundError, PermissionError):
# Wait for some time before the next retry
time.sleep(delay_between_retries)
except Exception as e:
# If there's an unexpected error, raise it immediately
raise
else:
# If we've exhausted all retries and still have an exception, we print a message
print(
f"[Thread: {threading.current_thread().name}] File {src_file} could not be moved after {max_retries} attempts"
)

# Now that all files have been moved (or logged if they couldn't be moved), remove the source directory
for attempt in range(max_retries): # type: ignore
try:
with lock:
os.rmdir(src_folder)
# If the operation is successful, break out of the retry loop
break
except OSError:
# Wait for some time before the next retry
time.sleep(delay_between_retries)
except Exception as e:
# If there's an unexpected error, raise it immediately
raise
else:
# If we've exhausted all retries and still have an exception, we print a message
print(
f"[Thread: {threading.current_thread().name}] Could not remove directory {src_folder} after {max_retries} attempts"
)

from collections import defaultdict

folder_locks = defaultdict(threading.Lock)
print_lock = Lock()
parent_pbar_lock = Lock()
lock = threading.Lock()


def process_folder_with_files(
folder,
parent_pbar,
weights_file,
folder_times,
print_lock,
folders,
):
all_steps_successful = True
temp_output_folder = os.path.join(folder, "temp_output")
os.makedirs(temp_output_folder, exist_ok=True)
output_file = os.path.join(folder, f"{os.path.basename(folder)}-custom.m4a")
normalized_weights = normalize_weights_based_on_duration(folder_times)

if folder not in folder_times:
folder_times[folder] = {}

input_files = {} # initialize input_files to an empty dictionary
ffmpeg_success = False # initialize ffmpeg_success to False

progress_log = None # initialize log files to None
report_log = None

flac_file = None # Initialize flac_file to None

try:
input_files = process_folder(folder, temp_output_folder)
if input_files is None:
raise ValueError(f"No valid input files found in folder: {folder}")

validate_paths(
input_files.values(),
output_file,
ffmpeg_exe_path,
fdkaac_exe_path,
temp_output_folder,
)
parent_folder = os.path.basename(os.path.dirname(folder))
current_folder = os.path.basename(folder)
terminal_width = 400

half_terminal_width = int(terminal_width / 2)
max_desc_length = half_terminal_width

folder_description = f'Processing "{parent_folder}" folder: {current_folder}'
cropped_description = folder_description[:max_desc_length]

# Ensure cropped_description isn't longer than half of the terminal width
if len(cropped_description) > half_terminal_width:
cropped_description = cropped_description[:half_terminal_width]

# Calculate padding
padding_length = half_terminal_width - len(cropped_description)
padding = "." * padding_length # use '.' for padding

# Concatenate description and padding
final_description = cropped_description + padding

# Initialize progress_log and report_log here
unique_id = uuid.uuid4()
progress_log = os.path.join(folder, f"progress-log-{unique_id}.txt")
report_log = os.path.join(folder, f"report-log-{unique_id}.log")

with timed_block("create_m4a_from_wavs", folder_times[folder]):
ffmpeg_progress = tqdm(
total=100,
desc=final_description,
unit="it/s",
position=2,
leave=True,
dynamic_ncols=True,
ncols=terminal_width,
)
(
temp_ffmpeg_file,
progress_log,
report_log,
ffmpeg_success,
flac_file,
) = create_m4a_from_wavs(
list(input_files.values()),
temp_output_folder,
ffmpeg_exe_path,
ffmpeg_progress,
parent_pbar,
normalized_weights,
folders,
folder,
progress_log, # Pass the initialized progress log file
report_log, # Pass the initialized report log file
)

# If temp_ffmpeg_file is None, stop processing the current folder and continue to the next one
if temp_ffmpeg_file is None:
print("FFmpeg process failed. Skipping current folder.")
return

ffmpeg_progress.refresh()
ffmpeg_progress.close()

with timed_block("run_mp4box", folder_times[folder]):
run_mp4box(folder, temp_ffmpeg_file, track_names, parent_pbar)

with parent_pbar_lock:
update_value = normalized_weights["run_mp4box"] * 100 / len(folders)
parent_pbar.update(update_value)

except Exception as e:
all_steps_successful = False
print(f"Error occurred during processing in the process_folder_with_files: {e}")

finally:
# Only proceed if all previous steps were successful and the FFmpeg operation was successful
if all_steps_successful and ffmpeg_success:
# Start a timed operation block for the removal of temporary files
with timed_block("remove_temp_files", folder_times[folder]):
# If there are input files, iterate over them
if input_files is not None:
for input_file in input_files.values():
# Only try to remove files that exist
if input_file is not None:
delete_file_with_retry(input_file)

# Define the path of the "kick_amplified.wav" file and try to remove it
amplified_kick_file_path = os.path.join(
temp_output_folder, "kick_amplified.wav"
)
if os.path.exists(amplified_kick_file_path):
delete_file_with_retry(amplified_kick_file_path)

# Define the path of the "kick.wav" file and try to remove it
kick_file_path = os.path.join(folder, "kick.wav")
if os.path.exists(kick_file_path):
delete_file_with_retry(kick_file_path)

# If there is a FLAC file, try to remove it
if flac_file and os.path.isfile(flac_file):
delete_file_with_retry(flac_file)

# Try to delete the progress and report log files
for log_file in [progress_log, report_log]:
if log_file is not None:
delete_file_with_retry(log_file)

# Try to delete any FFmpeg log files in the folder
for log_file2 in glob.glob(os.path.join(folder, "ffmpeg-*.log")):
delete_file_with_retry(log_file2)

# Call a function to remove the temporary folder
delete_directory_with_retry(temp_output_folder)

# With the lock for the parent progress bar, update its value
with parent_pbar_lock:
update_value = (
normalized_weights["remove_temp_files"] * 100 / len(folders)
)
parent_pbar.update(update_value)
parent_pbar.update(0.001)

# Copy folder contents to VDJstems folder
VDJparent_folder_name = os.path.basename(os.path.dirname(folder))
VDJparent_folder_path = os.path.join(
"C:\\OneDrive\\Muziek\\VDJstems", VDJparent_folder_name
)

if VDJparent_folder_path not in folder_locks:
folder_locks[VDJparent_folder_path] = threading.Lock()

with folder_locks[VDJparent_folder_path]:
try:
move_to_vdjstems(folder, VDJparent_folder_path)
except Exception as e:
print(f"An error occurred during moving to VDJstems: {e}")
# You can decide whether to set `all_steps_successful` to False here
# or handle this exception in a different way
all_steps_successful = False

elif not all_steps_successful:
print(
"Skipping removal of input files due to errors in the previous steps."
)

updated_weights, updated_folder_times, updated_data = analyze_and_update_weights(
folder, folder_times[folder]
)
folder_times[folder] = updated_folder_times
updated_data = {
"folder": folder,
"weights": updated_weights,
"folder_times": updated_folder_times,
}

return updated_weights, folder_times, updated_data

import asyncio
import concurrent.futures
import threading
import time
import ipywidgets as widgets
from IPython.display import display
import ipywidgets
import notebook
import json
from tqdm import tqdm
from tqdm.notebook import tqdm

print("ipywidgets version:", ipywidgets.__version__)
print("notebook version:", notebook.__version__)
import logging

logging.getLogger("ipykernel").setLevel(logging.WARNING)


interrupted = False


def interrupt(b):
global interrupted
interrupted = True
print("Button pressed! No new tasks will be started.")


async def stem_builder(root_directory, processing_folder_names, executor):
folder_times_lock = threading.Lock()
weights_file = r"C:\demucs_seperated\stemweights.json"
if not os.path.exists(weights_file):
weights, folder_times = load_initial_weights(
weights_file, root_directory, processing_folder_names
)
data = {"weights": weights, "folder_times": folder_times}
else:
with open(weights_file, "r") as f:
data = json.load(f)

folder_times = data.get("folder_times", {})
folders = find_folders_to_process(
root_directory, processing_folder_names, folder_times
)
weights = data.get("weights", {})
normalized_weights = normalize_weights_based_on_duration(folder_times)
total_progress = sum(normalized_weights.values()) * 100
results = {}

with tqdm(total=total_progress, desc="Overall Progress") as parent_pbar:

def wrapped_process_folder_with_files(folder):
if interrupted:
return None, None, None
original_input_files = check_original_files(folder)
if original_input_files is None or not original_input_files[1]:
return None, None, None
else:
try:
(
updated_weights,
updated_folder_times,
updated_data,
) = process_folder_with_files(
folder,
parent_pbar,
weights_file,
folder_times,
print_lock,
folders,
)
return updated_weights, updated_folder_times, updated_data
except Exception as e:
import traceback

traceback.print_exc()
print(f"An error occurred in 'process_folder_with_files': {e}")
return None, None, None

futures = []
for folder in folders:
if interrupted:
print("Interrupted. No new tasks will be started.")
break
future = loop.run_in_executor(
executor, wrapped_process_folder_with_files, folder
)
futures.append(future)
time.sleep(2)

# await all futures in parallel
await asyncio.gather(*futures)

for future in futures:
if not interrupted:
try:
result = future.result()
if result is not None:
updated_weights, updated_folder_times, updated_data = result
results[updated_data["folder"]] = {
"weights": updated_weights,
"folder_times": updated_folder_times,
}
except Exception as e:
import traceback

traceback.print_exc()
print(f"An error occurred in 'future.result()': {e}")
continue
else:
print("Interrupted. No new tasks will be started.")
break

for result in results.values():
weights.update(result["weights"])
folder_times.update(result["folder_times"])

if interrupted:
break

final_data = {
"weights": weights,
"folder_times": folder_times,
}

with open(weights_file, "w") as f:
json.dump(final_data, f, indent=4)

parent_pbar.update(parent_pbar.total - parent_pbar.n)
parent_pbar.refresh()
parent_pbar.n = 100
parent_pbar.refresh()

def create_button():
button = widgets.Button(description="Stop")
button.on_click(interrupt)
display(button)
return button


if __name__ == "__main__":
print("Starting StemBuilder v9.0")
print("Creating stems...")
executor = concurrent.futures.ThreadPoolExecutor(max_workers=8)
button = create_button()
loop = asyncio.get_event_loop()
loop.create_task(stem_builder(root_directory, processing_folder_names, executor))

 

geposted Sat 16 Sep 23 @ 3:27 am