GC disabled during native function execution

[adamschackart]

I'm calling gravity_vm_runclosure from inside a long-running C callback function, but every gravity object allocated inside that closure is leaked until the C function returns. Manually calling gravity_gc_start directly or indirectly causes weird issues likely related to memory corruption - is there any other way to call arbitrary gravity code from inside C without ballooning memory use?

[marcobambini]

Hi @adamschackart, can you please paste here some sample code?

[adamschackart]

I've created this contrived example that leaks 1.5gb on my system before finishing:

static bool NativeFunction(gravity_vm *vm, gravity_value_t *args,
                                uint16_t nargs, uint32_t rindex)
{
    gravity_closure_t* closure = (
    gravity_closure_t*)gravity_vm_getvalue(vm, "GravityFunction", strlen("GravityFunction")).p;

    gravity_vm_runclosure(vm, closure, VALUE_FROM_NULL, NULL, 0);
    RETURN_VALUE(VALUE_FROM_NULL, rindex);
}

int main(int argc, char** argv)
{
    gravity_delegate_t delegate = {};
    gravity_compiler_t* compiler = gravity_compiler_create(&delegate);

    const char* source_code =
    "extern var NativeClass\n"
    "\n"
    "func GravityFunction()\n"
    "{\n"
    "    for (var i in 0..<100000)\n"
    "    {\n"
    "        var s = []\n"
    "\n"
    "        for (var j in 0..<1000)\n"
    "        {\n"
    "            s.push(j)\n"
    "        }\n"
    "    }\n"
    "}\n"
    "\n"
    "func main()\n"
    "{\n"
    "    NativeClass()\n"
    "}\n";

    gravity_closure_t* closure = gravity_compiler_run(compiler, source_code,
                                        strlen(source_code), 0, true, true);

    gravity_vm* vm = gravity_vm_new(&delegate);
    gravity_compiler_transfer(compiler, vm);
    gravity_compiler_free(compiler);

    gravity_class_t* klass = gravity_class_new_pair(NULL, "NativeClass", NULL, 0, 0);
    gravity_class_bind(klass, "init", NEW_CLOSURE_VALUE(NativeFunction));
    gravity_vm_setvalue(vm, "NativeClass", gravity_value_from_object(klass));

    gravity_vm_runmain(vm, closure);

    gravity_vm_free(vm);
    gravity_core_free();

    return EXIT_SUCCESS;
}

[marcobambini]

Hi @adamschackart, thanks for the example.

The design choice was to trust external user code (for performance reason) and to disable GC during external user C code execution. You can see the actual VM code here:

// gravity_vm_exec:1244
BEGIN_TRUST_USERCODE(vm);
bool result = closure->f->internal(vm, &stackstart[rwin], r3, r1);
END_TRUST_USERCODE(vm);

// BEGIN/END_TRUST_USERCODE are macros that disable/enable GC

You can change this behavior by adding the line:

System.gcEnabled = true"

before the first for loop in GravityFunction.

If you need a more aggressive GC you can modify the System values:

• gcMinThreshold
• gcThreshold
• gcRatio

See gravity_vm_set in gravity_vm.c.

Hope this helps.

[adamschackart]

That works for my example, but still segfaults on my (much more complicated) game state machine. However, I can do a little restructuring so the state "manager" emits one frame at a time, and the outer loop is run in Gravity.

[marcobambini]

@adamschackart, if you give me a way to reproduce the crash, I would be more than willing to fix it.

[adamschackart]

Reopening - I've written the whole event loop in gravity, and it still crashes. It appears that when the GC kicks in, the entire program is being grayed before it's finished running - the root fiber, the names of all classes and methods, closures, everything.

My repro involves a large (400k SLOC) proprietary codebase, so I'm unsure of how to proceed on that. Maybe I could make a super cut-down version and email it to you or something.

EDIT debug output and GDB stack trace:

GC TRANSFER LIST 0x55555782cdb0 (0 items)
GC TRANSFER STRING 0x55555782e590 5654
GC TRANSFER STRING 0x55555782e5e0 3.68343
GC TRANSFER STRING 0x55555782e660 0.000575515
GC TRANSFER STRING 0x55555782e6e0 update 5654 3.68343 0.000575515
GRAY FIBER 0x555556f731d0
GRAY STRING 0x555557414140 sx
GRAY STRING 0x555556f809c0 System
GRAY STRING 0x55555747efc0 nv
GRAY STRING 0x555556f7b4d0 Bool
GRAY STRING 0x555556f7e8d0 Map
GRAY STRING 0x555556f7d3e0 Func
GRAY STRING 0x555556f7d500 Object
GRAY STRING 0x555556f77fd0 Closure
GRAY STRING 0x555556f7b670 File
GRAY STRING 0x55555742fc60 vz
GRAY STRING 0x555556f7d290 main
GRAY CLOSURE 0x555557504c20 main
GRAY STRING 0x5555572da8e0 aeImage
GRAY STRING 0x555556f7d350 Float
GRAY STRING 0x555557504b20 og
GRAY STRING 0x55555739ecd0 xlState
GRAY CLASS 0x555556f73510 xlState
GRAY STRING 0x555556f7ea50 Upvalue
GRAY STRING 0x555556f7d550 Class
GRAY STRING 0x555556f7e840 List
GRAY STRING 0x55555710a2d0 aeVec4
GRAY STRING 0x55555732e110 xlEvent
GRAY STRING 0x555556f78060 Fiber
GRAY STRING 0x555556f7e990 Range
GRAY STRING 0x555556f7bff0 ENV
GRAY STRING 0x5555574b60c0 dv
GRAY STRING 0x5555572daa60 ae
GRAY STRING 0x5555573e9e00 xlWindow
GRAY STRING 0x55555746ad20 mx
GRAY STRING 0x555556f7be50 Test
GRAY CLASS 0x555556f75890 Test
GRAY STRING 0x555556f71ca0 Math
GRAY STRING 0x555556f780f0 String
GRAY STRING 0x555556f78180 Instance
GRAY STRING 0x55555710a450 aeRGBA
GRAY STRING 0x5555573e9f80 xl
GRAY STRING 0x555556f7d6d0 Int
GRAY STRING 0x555556f7d640 Null
GRAY STRING 0x555556f7b780 JSON
GRAY CLASS 0x555556f7c960 Test meta
GRAY CLOSURE 0x555556f82580 init
GRAY CLOSURE 0x555556f825e0 draw
GRAY CLOSURE 0x555556f82640 handle_event
GRAY CLOSURE 0x555556f826a0 $init
GRAY CLOSURE 0x555556f82700 overlay
GRAY CLOSURE 0x555556f82760 leave
GRAY CLOSURE 0x555556f827c0 enter
GRAY CLOSURE 0x555556f82820 postprocess
GRAY CLOSURE 0x555556f82880 update
GRAY FUNCTION 0x555556f7ceb0 update
GRAY STRING 0x555556f74250 System
GRAY STRING 0x555556f75570 print
GRAY STRING 0x555556f74a30 update 
GRAY STRING 0x555556f74070  
GRAY STRING 0x555556f7d170 join
GRAY FUNCTION 0x555556f78550 postprocess
GRAY STRING 0x555556f7a270 System
GRAY STRING 0x555556f7a2c0 print
GRAY STRING 0x555556f7a310 postprocess
GRAY FUNCTION 0x555556f7cb10 enter
GRAY STRING 0x555556f7cce0 System
GRAY STRING 0x555556f7cd30 print
GRAY STRING 0x555556f7cd80 enter
GRAY FUNCTION 0x555556f7cbd0 leave
GRAY STRING 0x555556f7cc90 System
GRAY STRING 0x555556f7cf70 print
GRAY STRING 0x555556f7cfc0 leave
GRAY FUNCTION 0x555556f7c310 overlay
GRAY STRING 0x555556f7c3d0 System
GRAY STRING 0x555556f7c7f0 print
GRAY STRING 0x555556f7c840 overlay
GRAY FUNCTION 0x555556f73b70 $init
GRAY STRING 0x555556f74390 ae
GRAY STRING 0x555556f74440 RGBA
GRAY FUNCTION 0x555556f7d0a0 handle_event
GRAY STRING 0x555556f7e1e0 System
GRAY STRING 0x555556f7e2c0 print
GRAY STRING 0x555556f7e310 handle_event
GRAY FUNCTION 0x555556f7b0b0 draw
GRAY STRING 0x555556f782c0 System
GRAY STRING 0x555556f7a4f0 print
GRAY STRING 0x555556f7a680 draw
GRAY FUNCTION 0x555556f7d750 init
GRAY STRING 0x555556f7c630 $init
GRAY CLASS 0x555556f681e0 xlState meta
GRAY CLOSURE 0x555556f81980 call_postprocess
GRAY CLOSURE 0x555556f819e0 ANONYMOUS
GRAY CLOSURE 0x555556f81a40 draw
GRAY CLOSURE 0x555556f81aa0 ANONYMOUS
GRAY CLOSURE 0x555556f81b00 call_draw
GRAY CLOSURE 0x555556f81b60 set_running
GRAY CLOSURE 0x555556f81bc0 run
GRAY CLOSURE 0x555556f81c20 ANONYMOUS
GRAY CLOSURE 0x555556f81c80 ANONYMOUS
GRAY CLOSURE 0x555556f81ce0 $init
GRAY CLOSURE 0x555556f81d40 ANONYMOUS
GRAY CLOSURE 0x555556f81da0 handle_events
GRAY CLOSURE 0x555556f81e00 leave
GRAY CLOSURE 0x555556f81e60 wrap
GRAY CLOSURE 0x555556f81ec0 ANONYMOUS
GRAY CLOSURE 0x555556f81f20 update
GRAY CLOSURE 0x555556f81f80 init
GRAY CLOSURE 0x555556f81fe0 call_update
GRAY CLOSURE 0x555556f82040 ANONYMOUS
GRAY CLOSURE 0x555556f820a0 handle_event
GRAY CLOSURE 0x555556f82100 ANONYMOUS
GRAY CLOSURE 0x555556f82160 call_enter
GRAY CLOSURE 0x555556f821c0 call_leave
GRAY CLOSURE 0x555556f82220 call_overlay
GRAY CLOSURE 0x555556f82280 ANONYMOUS
GRAY CLOSURE 0x555556f822e0 ANONYMOUS
GRAY CLOSURE 0x555556f82340 ANONYMOUS
GRAY CLOSURE 0x555556f823a0 ANONYMOUS
GRAY CLOSURE 0x555556f82400 enter
GRAY CLOSURE 0x555556f82460 set_window
GRAY CLOSURE 0x555556f824c0 ANONYMOUS
GRAY CLOSURE 0x555556f82520 ANONYMOUS
GRAY FUNCTION 0x555556f755d0 ANONYMOUS
GRAY FUNCTION 0x555556f753f0 ANONYMOUS
GRAY FUNCTION 0x555556f79f70 set_window
GRAY STRING 0x555556f7a220 xl
GRAY STRING 0x555556f7a380 primary_window
GRAY FUNCTION 0x555556f75d50 enter
GRAY FUNCTION 0x555556f74c70 ANONYMOUS
GRAY FUNCTION 0x555556f74e50 ANONYMOUS
GRAY FUNCTION 0x555556f747a0 ANONYMOUS
GRAY FUNCTION 0x555556f75210 ANONYMOUS
GRAY FUNCTION 0x555556f79e00 call_overlay
GRAY FUNCTION 0x555556f77e20 call_leave
GRAY STRING 0x555556f77ee0 leave
GRAY FUNCTION 0x555556f77660 call_enter
GRAY STRING 0x555556f77bc0 enter
GRAY FUNCTION 0x555556f74a90 ANONYMOUS
GRAY FUNCTION 0x555556f763b0 handle_event
GRAY FUNCTION 0x555556f740d0 ANONYMOUS
GRAY FUNCTION 0x555556f79540 call_update
GRAY STRING 0x555556f71040 ae
GRAY STRING 0x555556f79600 get_frame_delta
GRAY STRING 0x555556f79a70 update
GRAY FUNCTION 0x555556f7daf0 init
GRAY STRING 0x555556f7dbb0 $init
GRAY FUNCTION 0x555556f768b0 update
GRAY FUNCTION 0x555556f73ef0 ANONYMOUS
GRAY FUNCTION 0x555556f771b0 wrap
GRAY FUNCTION 0x555556f75f30 leave
GRAY FUNCTION 0x555556f783e0 handle_events
GRAY STRING 0x555556f78660 xl
GRAY STRING 0x555556f786b0 event_poll
GRAY STRING 0x555556f78960 type
GRAY STRING 0x555556f78c60 EVENT_NOTHING
GRAY STRING 0x555556f78e50 handle_event
GRAY FUNCTION 0x555556f737b0 ANONYMOUS
GRAY FUNCTION 0x555556f73d10 ANONYMOUS
GRAY FUNCTION 0x555556f74640 ANONYMOUS
GRAY FUNCTION 0x555556f7a790 run
GRAY STRING 0x555556f7ac10 set_window
GRAY STRING 0x555556f7ad80 set_running
GRAY STRING 0x555556f7aee0 ae
GRAY STRING 0x555556f7af30 get_frame_delta
GRAY STRING 0x555556f7add0 handle_events
GRAY STRING 0x555556f7ab60 xl
GRAY STRING 0x555556f7b2d0 window_get_open
GRAY STRING 0x555556f7b030 window_clear
GRAY STRING 0x555556f7b590 call_update
GRAY STRING 0x555556f7bc00 call_draw
GRAY STRING 0x555556f7c200 call_postprocess
GRAY STRING 0x555556f7c460 call_overlay
GRAY STRING 0x555556f7c8e0 window_flip
GRAY STRING 0x555556f7ba00 run
GRAY FUNCTION 0x555556f7a410 set_running
GRAY STRING 0x555556f7a740 call_enter
GRAY STRING 0x555556f7a8a0 call_leave
GRAY FUNCTION 0x555556f735b0 call_draw
GRAY STRING 0x555556f73670 draw
GRAY FUNCTION 0x555556f75030 ANONYMOUS
GRAY FUNCTION 0x555556f76d30 draw
GRAY FUNCTION 0x555556f748b0 ANONYMOUS
GRAY FUNCTION 0x555556f79d40 call_postprocess
GRAY FUNCTION 0x555556f7bea0 main
GRAY STRING 0x555556f7c100 xl
GRAY STRING 0x555556f7a9d0 mouse_set_int
GRAY STRING 0x555556f7b410 primary_mouse
GRAY STRING 0x555556f7b700 relative
GRAY STRING 0x555556f7d4b0 Test
GRAY STRING 0x555556f7d5c0 run
GRAY INSTANCE 0x555557504cc0 OF Test
GRAY STRING 0x55555782e6e0 update 5654 3.68343 0.000575515
GRAY LIST 0x55555782cdb0 (6 items)
GRAY INSTANCE 0x555557504e00 OF Vec4
update 5654 3.68343 0.000575515
draw

Thread 1 "run_orogen" received signal SIGSEGV, Segmentation fault.
0x0000555555b1cda3 in gravity_class_lookup (c=0x55500267a14d, key=...) at extern/gravity/src/shared/gravity_value.c:400
400	        gravity_value_t *v = gravity_hash_lookup(c->htable, key);
(gdb) bt
#0  0x0000555555b1cda3 in gravity_class_lookup (c=0x55500267a14d, key=...) at extern/gravity/src/shared/gravity_value.c:400
#1  0x0000555555b1ce95 in gravity_class_lookup_closure (c=0x55500267a14d, key=...) at extern/gravity/src/shared/gravity_value.c:416
#2  0x0000555555b086aa in gravity_vm_exec (vm=0x555556f730a0) at extern/gravity/src/runtime/gravity_vm.c:758
#3  0x0000555555b170ef in gravity_vm_runmain (vm=0x555556f730a0, closure=0x555556f73040) at extern/gravity/src/runtime/gravity_vm.c:1809
#4  0x0000555555ac9c43 in og_internal_run_script (filename=0x7ffff7595660 <ae_internal_tiny_string_memory_ex()::ae_tiny_str_mem_buf+368> "main.gravity", fatal=0) at orogen/code/og_intro.cpp:793
#5  0x00005555557bd51b in og_run_script (filename=0x0, fatal=0) at orogen/code/og_intro.cpp:848
#6  0x00005555557bc5b9 in og_main (window=170131459) at orogen/code/og_core.cpp:255
#7  0x00005555557bc4c1 in xl_main (argc=1, argv=0x7fffffffde78) at ./pyxl2D/misc/xl_main.c:79
#8  0x00005555557bc483 in main (argc=1, argv=0x7fffffffde78) at aecore/misc/ae_main.c:72

[marcobambini]

If you are able to send me a code that is able to reproduce the crash that would be perfect.
Please email the code to marco AT creolabs DOT com.

[adamschackart]

I believe I've found the bug. It appears that passing a valid (non-NULL) VM pointer into computed_property_create and then using that property in gravity code will trigger this behavior.