🐛 [BUG] Soundfont - SF2 file > 1gb will fail to load

[guizmox]

Hardware and software

• mt32-pi version: v0.12.1
• Raspberry Pi model: Raspberry Pi 4 CM4104000 module (4gb Lite, wifi) on a Serda McCake32 board.
• USB devices: None
• MIDI host: Miditemp Mulstistation MSX using Waveblaster port

Bug description

Any attempt to load SF2 files bigger than 1gb will fail. "SF switch error".

Those soundfonts successfully load using Fluidsynth for Windows (Fluidsynth 2.3.0).
They also load on Fluidsynth installed on a Raspberry PI 4 with 4gb RAM
I did tried a lot of files, edited them with Polyphone or Viena to reach the limit, which seems to be between 910mb and 1000mb

Steps to reproduce

1. Use a SF2 file that is bigger than 1gb
2. Use "change bank" to choose it
3. Wait a couple of seconds. "SF switch failed !"

Expected behavior

File should load successfully : 3488 samples, 59231 instrument generators, 1611 preset generators

Configuration file

Click to expand

#                                  ________    _______                      __
#                         __      /_____   `. /____   `.                   /__`
#     ____   ____     __/  /____  _______)  / ______)  / ____  ______      __
#   /   __ v  __  `./__   _____//_____    < /   _____. /____//   ___  `. /  /
#  /  /  /  /  /  /   /  /____  _______)  //  /______       /  /____/  //  .__
# /__/  /__/  /__/    \______//_________. /_________/      /   ______.  \____/
# /////////////////////////////////////////////////////// /  / //// /// // /
#                                                         ```
# mt32-pi.cfg: mt32-pi configuration file.
# Default options are marked with an asterisk (*).

# -----------------------------------------------------------------------------
# System options
# -----------------------------------------------------------------------------
[system]

# Enable or disable verbose startup and error output.
#
# When enabled, outputs more information to the LCD when starting up, and when
# MIDI/UART errors are detected.
#
# This also may hide the boot logo on smaller graphical displays.
#
# Values: on, off*
verbose = on

# Set the default synthesizer to be made active on startup.
#
# If the default synthesizer is unavailable (e.g. missing ROMs or SoundFonts),
# the first working synth is made active.
#
# Values: mt32*, soundfont
#
# mt32:      Use mt32emu (Munt) for Roland MT-32 emulation
# soundfont: Use FluidSynth for SoundFont synthesis
default_synth = soundfont

# Enable or disable support for USB devices.
#
# Disable this to speed up boot time if you are not using any USB devices.
#
# Values: on*, off
usb = off
# Set the I2C baud rate/clock speed for all peripherals (Hz).
#
# Most peripherals will work fine at the default speed (400KHz "fast mode"),
# but larger LCD/OLED displays (e.g. 4-line I2C HD44780 and 64 pixel high
# SSD1306) won't be able to refresh at 60FPS at the default setting.
#
# Try increasing this value to 1000000 (1MHz) for a smoother LCD refresh rate.
# If your display doesn't work, try backing off the speed 100KHz at a time
# until it does.
#
# Values: 100000-1000000 (400000*)
i2c_baud_rate = 400000

# Set the timeout for power saving mode (seconds).
#
# After the specified number of seconds of silence, the CPU clock speed will be
# reduced, the audio device will be stopped, and and the LCD's backlight will
# be turned off to save energy (certain I2C displays only).
#
# Any MIDI activity will instantly bring the system out of power saving mode.
#
# If set to 0, power saving mode is disabled.
#
# Values: 0-3600 (300*)
power_save_timeout = 300

# -----------------------------------------------------------------------------
# MIDI options
# -----------------------------------------------------------------------------
[midi]

# Set the baud rate used for GPIO MIDI.
#
# For connecting to standard MIDI devices (i.e. via DIN cable), this should be
# left at the default rate of 31250.
#
# For connecting to PCs, set this to match the baud rate of the other host.
# SoftMPU's serial MIDI mode, for example, uses a baud rate of 38400.
#
# Values: 300-4000000 (31250*)
gpio_baud_rate = 31250

# Enable or disable software "MIDI thru" on the GPIO Tx pin.
#
# When enabled, all data received via the GPIO Rx pin will be re-transmitted
# verbatim on the Tx pin. This may be useful for debugging or for passing MIDI
# data through to another synth.
#
# Values: on, off*
gpio_thru = off

# Set the baud rate used for USB serial MIDI.
#
# The same considerations from the gpio_baud_rate setting above apply here.
# The default value is a PC baud rate and matches SoftMPU's serial MIDI mode.
#
# The range of valid baud rates may vary depending on the chipset of your USB
# serial device, so the range of values suggested below may actually be greater.
#
# Values: 9600-115200 (38400*)
usb_serial_baud_rate = 38400

# -----------------------------------------------------------------------------
# Audio options
# -----------------------------------------------------------------------------
[audio]

# Select audio output device.
#
# Values: pwm*, i2s
#
# pwm: Use the headphone jack
# hdmi: Use the HDMI port
# i2s: Use an I2S DAC
output_device = i2s

# Sample rate of audio output (Hz).
#
# mt32emu uses an internal sample rate of 32000Hz (just like the real hardware)
# which is then resampled to this value.
#
# FluidSynth renders at this sample rate directly.
#
# Values: 32000-192000 (48000*)
sample_rate = 48000

# Set audio rendering chunk size (samples).
#
# A single stereo frame of audio has two samples, and so this value is double
# the number of frames per chunk.
# The smaller the chunk size, the lower the latency, but too low a value will
# cause underruns (distortion artifacts).
#
# Latency is a function of chunk size and sample rate, for example:
# 256 samples / 2 channels / 48000Hz * 1000ms = 2.67ms of latency.
# See documentation for recommended values for various Raspberry Pi models.
#
# The minimum value varies depending on audio output device.
# For PWM, the minimum is 2, for I2S the minimum is 32.
# For HDMI, the minimum is 384, and will be rounded to the nearest multiple of
# 384.
#
# Values: 2-2048 (256*)
chunk_size = 128

# Set address (hexadecimal) of I2C DAC control interface.
#
# This will be used for the initialization sequence (see below) if enabled.
# You can find the address of your DAC by using the i2cdetect utility in Linux.
#
# Values: 00-80 (4c*)
i2c_dac_address = 4c

# Select an initialization sequence for the DAC.
#
# Some DACs require some initialization commands to be sent via I2C before they
# will produce any sound.
#
# Values: none*, pcm51xx
#
# pcm51xx: for DACs based on PCM5121 or similar (e.g. PCM5141, PCM5242)
i2c_dac_init = none

# -----------------------------------------------------------------------------
# Control options
# -----------------------------------------------------------------------------
[control]

# Set the physical control scheme.
#
# See documentation for GPIO pinouts/wiring details.
#
# Values: none*, simple_buttons, simple_encoder
#
# none:           No physical controls
# simple_buttons: Simple 4-button scheme
# simple_encoder: Simple 2-button + rotary encoder scheme
scheme = simple_buttons
# Set the rotary encoder type (if used by control scheme).
#
# Different rotary encoders may complete different fractions of a Gray-code
# cycle per detent ("click").
#
# If four clicks are needed for a single movement, try "quarter".
# If two clicks are needed for a single movement, try "half".
#
# Values: quarter, half, full*
encoder_type = full

# Reverse the rotary encoder direction (if used by control scheme).
#
# Some rotary encoders may have their CLK/DAT signals swapped, resulting in
# a reversed rotation direction.
#
# Use this option to correct the direction.
#
# Values: on, off*
encoder_reversed = off

# Enable or disable the I2C MiSTer control interface.
#
# If using mt32-pi with a MiSTer FPGA system and custom hardware to interface
# with MiSTer's User Port, enable this option to allow controlling mt32-pi via
# the MiSTer's on-screen display.
#
# Values: on, off*
mister = off

# Set the timeout for switching SoundFonts (seconds).
#
# When switching SoundFonts using the physical button, there is a short delay
# before loading begins. This option allows you to set the number of seconds to
# wait before loading.
#
# Values: 0-3600 (3*)
switch_timeout = 3

# -----------------------------------------------------------------------------
# MT-32 emulator options
# -----------------------------------------------------------------------------
[mt32emu]

# Set gain factor applied to synthesizer output channels.
#
# This is independent of the master volume that can be set via MIDI SysEx or
# the volume knob.
#
# Values: 0.0-256.0 (1.0*)
gain = 1.0

# Set gain factor applied to reverb wet output channels.
#
# Values: 0.0-infinity (1.0*)
reverb_gain = 1.0

# Select quality level for the resampler.
#
# If set to none, audio output will sound wrong unless you set the sample rate
# option to 32000Hz, which is the MT-32's native sample rate.
#
# Values: none, fastest, fast, good*, best
resampler_quality = good

# Select initial MIDI channel assignment.
#
# The MT-32 uses an unusual MIDI channel assignment by default. On a real MT-32
# this is set using a button combination. Use this option to change the initial
# channel assignment on startup.
#
# Values: standard*, alternate
#
# standard:  Parts 1-8 = MIDI channels 2-9, Rhythm part = MIDI channel 10
# alternate: Parts 1-8 = MIDI channels 1-8, Rhythm part = MIDI channel 10
midi_channels = standard

# Select initial ROM set to use.
#
# If multiple ROM sets are available, this option determines which set to use
# on startup. If the ROM set specified here is unavailable, the first available
# set is used instead.
#
# Values: old*, new, cm32l
rom_set = old

# Set whether the stereo channels should be swapped or not.
#
# The MT-32 interprets values for MIDI CC#10 (panpot) differently to later
# synthesizers, which means that 0 = right and 127 = left; the opposite of what
# is stated in the most recent versions of the MIDI specification.
#
# Enable this option to swap the channels and make MT-32 mode's panning
# behavior match the behavior of SoundFont mode. Note that this can also be
# switched at runtime with a custom SysEx command.
#
# Values: on, off*
reversed_stereo = off

# -----------------------------------------------------------------------------
# SoundFont synthesizer options
# -----------------------------------------------------------------------------
[fluidsynth]

# Set the initial SoundFont to use.
#
# If multiple SoundFonts are available, this option determines which SoundFont
# to use on startup.
#
# On startup, the "soundfonts" directory is scanned for valid SoundFonts, which
# are added to a list and sorted into alphabetical order.
#
# This setting is a zero-indexed offset into that list (i.e. 0 is the first,
# 1, is the second, and so on).
#
# If the index specified is unavailable, the first available SoundFont will be
# used.
#
# Values: 0-255 (0*)
soundfont = 0

# Set the maximum number of voices that can be played simultaneously.
#
# Depending on the complexity of your SoundFont, you may need to reduce this
# value to prevent audio buffer underruns (distortion) when playing music
# featuring lots of notes being played at once.
#
# On the other hand, you may want to try raising this value if your Pi is
# being run overclocked or has a more powerful CPU (e.g. Pi 4/CM4).
#
# N.B. larger file size of the SoundFont does not imply higher CPU usage.
# SoundFonts that use more real-time effects (modulators) are more likely to
# require a reduction in polyphony.
#
# Values: 1-65535 (200*)
polyphony = 256

# The following settings set the default parameters for FluidSynth's master
# volume gain, reverb and chorus effects.
#
# Each setting can be overridden on a per-SoundFont basis by adding extra
# sections, e.g. [fluidsynth.soundfont.x], where x is the zero-based index of
# the SoundFont. See the next section for an example.
#
# Full descriptions and valid value ranges for each setting can be found in the
# FluidSynth documentation: https://www.fluidsynth.org/api/fluidsettings.xml
gain = 0.6

reverb = on
reverb_damping = 0.2
reverb_level = 0.9
reverb_room_size = 0.5
reverb_width = 20.0

chorus = on
chorus_depth = 4.0
chorus_level = 0.5
chorus_voices = 3
chorus_speed = 0.3

# -----------------------------------------------------------------------------
# FluidSynth effects profile for SoundFont 0 (GeneralUser GS)
# -----------------------------------------------------------------------------
[fluidsynth.soundfont.0]

# The following settings are recommended for GeneralUser GS by its author.
gain = 0.6

reverb = on
reverb_damping = 0.2
reverb_level = 0.9
reverb_room_size = 0.5
reverb_width = 20.0
#
chorus = on
chorus_depth = 4.0
chorus_level = 0.5
chorus_voices = 3
chorus_speed = 0.3

# -----------------------------------------------------------------------------
# LCD/OLED display options
# -----------------------------------------------------------------------------
[lcd]

# Select LCD driver.
#
# Note that LCDs connected via I2C, you must also set the correct address for
# your device via the i2c_lcd_address option. Consult its datasheet, or see
# our documentation for tested models and their configurations.
#
# Values: none*, hd44780_4bit, hd44780_i2c, sh1106_i2c, ssd1306_i2c
#
# none:         No LCD
# hd44780_4bit: Hitachi HD44780 or compatible (e.g. WS0010, RS0010) character
#               LCD connected to GPIO pins in 4-bit mode (see documentation for
#               pinout)
# hd44780_i2c:  As above, but using an I2C "backpack"
# sh1106_i2c:   Small I2C-based OLED graphical display (usually 1.3")
# ssd1306_i2c:  Small I2C-based OLED graphical display (usually 0.96")
type = ssd1306_i2c

# Set the width of the LCD.
#
# If the display is a character display, this value is measured in characters.
# Otherwise, for a graphical display, this is measured in pixels.
#
# Note that not all dimension settings are valid; see documentation for valid
# configurations for each LCD driver.
#
# Values: 20-128 (128*)
width = 128

# Set the height of the LCD.
#
# Same characters vs. pixels considerations as for width.
#
# Values: 2-64 (32*)
height = 32

# Set address (hexadecimal) of I2C LCD.
#
# This will be used to communicate with LCDs connected via the I2C bus.
#
# Values: 00-80 (3c*)
i2c_lcd_address = 3c

# Rotate the display output (graphical LCDs only).
#
# Some graphical displays support rotation. Use this option if you need to turn
# the display around.
#
# Values: normal*, inverted
#
# normal:   No rotation
# inverted: The display output is upside down
rotation = inverted

# Mirror the display output horizontally (graphical LCDs only).
#
# Some graphical displays display columns right-to-left rather than left-to-
# right. Use this option if you need to mirror the display horizontally.
#
# Values: normal*, mirrored
#
# normal:   No mirroring
# mirrored: The display output is mirrored horizontally
mirror = normal

# -----------------------------------------------------------------------------
# Network options
# -----------------------------------------------------------------------------
[network]

# Select the network mode.
#
# For setting your Wi-Fi SSID and encryption key, see wpa_supplicant.conf.
# Note that if using Ethernet on a Raspberry Pi 3B/3B+, USB must be enabled.
#
# off:      Disable networking
# ethernet: Enable using the Ethernet interface
# wifi:     Enable using the the Wi-Fi interface
#
# Values: off*, ethernet, wifi
mode = off

# Enable or disable DHCP for configuring the network.
#
# If disabled, the manual settings below will be used to configure your network
# interface instead.
#
# Values: on*, off
dhcp = off

# Manual settings for configuring the network interface.
#
# These settings will be ignored if DHCP is enabled.
#
# Values: correctly-formatted IP address/subnet mask, e.g. AAA.BBB.CCC.DDD
#         (four numbers in the range 0-255 separated by periods)
ip_address = 192.168.0.100
subnet_mask = 255.255.255.0
default_gateway = 192.168.0.1
dns_server = 192.168.0.1

# Set the network hostname.
#
# Values: a valid hostname using ASCII letters 'a' to 'z', digits 0-9, and
#         hyphens (mt32-pi*)
hostname = mt32-pi

# Enable or disable the RTP-MIDI/AppleMIDI server.
#
# This allows you to send MIDI data to mt32-pi over the network using macOS'
# built-in network MIDI features, or rtpMIDI by Tobias Erichsen on Windows.
#
# Values: on*, off
rtp_midi = off

Videos/audio recordings

Nothing

Additional information

I can provide the SF2 file if needed.

[guizmox]

I used a HDMI out kernel version to check the LOG, and here's the result :
It seems that the zoneallocator is fixed to 940 megabytes (line 4)

[dwhinham]

Thank you for the detailed bug report and providing the logs, it's much appreciated.

The reason for this is because of the way Circle's memory map/allocation is implemented. Unfortunately, we don't get access to the whole 4GB (or 8GB on those models) at the moment.

On Pi 4, the first 1GB is called the "low heap", and Circle's internal memory allocator uses this. The next 2GB is the "high heap" and is handled by mt32-pi's custom allocator - this is what FluidSynth uses. The remaining 1GB on Pi 4/4GB (or 5GB on Pi4/8GB) is not usable right now.

That said, we are only getting 940MB of "high heap" according to your log, when it should be closer to 2GB, so this needs investigating.

(On Pi 2/3/02W there is no "high heap", only a "low heap". Here, 32MB is reserved for Circle's internal allocator, and the remainder is handled by ours).

Circle's internal memory allocator cannot keep track of large allocations, hence the need for a custom allocator, otherwise FluidSynth cannot unload SoundFonts and free the memory for when the user wants to load another one.

There are a few things that need to happen so that we can use the rest of the memory on Pi 4:

• Treat the "low heap" like we do on Pi 2/3/02W; reserve 32MB for the internal allocator, then give the remainder to a custom allocator so that FluidSynth can use it - this will give us (almost) an additional 1GB. This shouldn't be too difficult.
• Figure out why the custom allocator isn't getting the full 2GB that Circle should be able to provide from the upper heap.
• Investigate how we can extend the range of the upper heap so that the rest of the RAM can be used. The reasons for the memory range being limited are given in the Circle documentation but it's not clear if we need to worry about the "specific DMA-handling" or not for this project.

I will need some time to figure out the best way to solve this.

P.S. thank you very much for the donation, I really appreciate it! 🙂

[guizmox]

Hello and thank you very much for the detailled answer, your project is really great, and useful for a lot of people including me.
That's seems to match with what I read from the "zoneallocator.cpp" file

const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}

I would first try to set a fixed value for "nHighHeapSize" and see what happens, but I'm unfortunately not able to fork your code, I'm a C# dev. with no experience with C++
An additionnal 1gb would be great, I'm fully aware that bigger SF2 files does not mean better sound, but I'm not using MT32-pi as a GM/GS machine but more as a VST instrument to extend my sound palette :)

[guizmox]

I read the memory.h Circle code, and this may explain things :

In case of Pi4, Your custom allocator does the following :
pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);

According to Circle, using HEAP_ANY should also add the HEAP_LOW Free Space and return an addition of both, even if it's not clear why the HEAP_HIGH zone only returns 940mb.

I now have to figure out how I can compile your code :D

[dwhinham]

It's not that simple. The custom allocator needs to make one large, single contiguous allocation from either one of those heaps, so using the size of "any" (the two heap sizes added together) doesn't make sense. Changing the GetHeapFreeSpace() call to use HEAP_ANY will give it a bigger number, but it will be incorrect.

The best solution (which I have wanted to do for a long time due to other related issues) will be to remove Circle's allocator completely and use the Zone allocator system-wide. There is a branch (allocator) where I have been working on this in the past, but there are some show-stopping bugs that need fixing before it can be used.

[guizmox]

Interesting ! I'll look into that branch and see what I can find. Thank you :)

[guizmox]

I was able to compile the code and made a slight change :

#if RASPI >= 4
	const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}
	else
	{
#endif
		// Allocate the majority of the remaining LOW region; leave some space for Circle/libc malloc()
		m_nHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_LOW) - MallocHeapSize;
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_LOW);
#if RASPI >= 4
	}
#endif

HEAP_HIGH part of the code dedicated to RPI4 was always ignored. I believe because of the RASPI variable that is never initialized in the compilation process.

My change :

	const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}
	else
	{
		// Allocate the majority of the remaining LOW region; leave some space for Circle/libc malloc()
		m_nHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_LOW) - MallocHeapSize;
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_LOW);
	}

As you explained to me, the HIGH_HEAP zone was able to allocate 2048mb :)
I was able to successfully load a 1.5gb SF2 file (in 83 seconds)