[FastRGF] FastRGF doesn't work for small sample and need to fix integration test for FastRGF

[fukatani]

#Now, sklearn integration tests for FastRGFClassifier and FastRGFClassifier.

FastRGF doesn't work well for small samples, that is reason for test failed.
I doubt inside Fast RGF executable inside.
I inspect Fast RGF by debugger, discretization boundaries are invalid.

At least we should raise understandable error from RGF python if discretization failed.

[StrikerRUS]

I've asked Tong Zhang to look into this issue (and also another one about small values of weights) and he answered the following:

Appreciate the bug reports. I’d try to look into them this weekend.

Also he asked you to create a PR if you fixed this

I inspect Fast RGF by debugger, discretization boundaries are invalid.

Also if your friend finds some bugs he fixed, please let me know, and maybe he’d like to contribute directly to the source code.

[fukatani]

Unfortunately, fixing this issue is difficult for me...

[StrikerRUS]

But maybe you can share your findings though isssue in their repo then?

[fukatani]

Of course I can!
But please give me time, I'm little busy and forgot about debugging.

[StrikerRUS]

@fukatani Any updates?

[StrikerRUS]

Some details from debugging

Steps to reproduce:

1. git clone https://github.com/baidu/fast_rgf.git
2. compile forest_train and forest_predict
3. cd fast_rgf/examples/ex2
4. leave less than 28 (I used 25) first rows in fast_rgf/examples/ex2/inputs/housing.train
5. run command gdb --args "../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x" trn.target=REAL model.save="outputs/model-rgf"

Operating System: Windows 10 64-bit
Compiler: MinGW-w64 (x86_64-posix-seh-rev0) g++ 5.4.0

Output:

D:\Users\nekit\Downloads\fast_rgf\examples\ex2>gdb --args "../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x"  trn.target=REAL model.save="outputs/model-rgf"
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../../bin/forest_train...done.
(gdb) run
Starting program: D:\Users\nekit\Downloads\fast_rgf\bin\forest_train.exe "-config=inputs/config" "trn.x-file=inputs/housing.train" "trn.x-file_format=y.x" "trn.target=REAL" "model.save=outputs/model-rgf"
[New Thread 5716.0x19d0]
[New Thread 5716.0x1ec8]
[New Thread 5716.0x348]
[New Thread 5716.0x17ec]

reading options from configuration file <inputs/config>

 using up to 8 openmp threads
 the number of threads is set to 8, which is the maximum number of logical hardware threads including hyperthreads
 the optimal number of threads is often the number of physical cores that may be smaller than 8
 for example, to achieve better performance, you may try to set the number of threads to 4

loading training data ...
  trn.target=REAL
  trn.x-file_format=y.x
  trn.x-file=inputs/housing.train
  trn.y-file=
  trn.w-file=
[New Thread 5716.0x213c]
[New Thread 5716.0x2548]
[New Thread 5716.0x94c]
[New Thread 5716.0x2500]
[New Thread 5716.0x1a5c]
[New Thread 5716.0x1bf8]
[New Thread 5716.0x2750]
loading time: wall time=0 seconds; cpu time=0 seconds.
discretizing training data ...
  discretize.dense.min_bucket_weights=5.000000
  discretize.dense.max_buckets=250
  discretize.dense.lamL2=10
  discretize.sparse.min_bucket_weights=5.000000
  discretize.sparse.max_buckets=200
  discretize.sparse.max_features=80000
  discretize.sparse.min_occrrences=5
  discretize.sparse.missing_type=MIN
  discretize.sparse.lamL2=2.000000
discritizer training time: wall time=0 seconds; cpu time=0 seconds.


training decision forest ...
  dtree.loss=LS
  dtree.max_level=6
  dtree.max_nodes=50
  dtree.new_tree_gain_ratio=1.0
  dtree.min_sample=5
  dtree.lamL1=10
  dtree.lamL2=1000
  forest.opt=rgf
  forest.ntrees=1000
  forest.eval_frequency=50
  forest.save_frequency=0


  training data size= 25 with 0 dense features and 1 sparse feature groups


build tree     1/ 1000
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 5716.0x2750]
0x000000000042fa86 in _decisionTreeTrainer::YW_struct::add (this=0xabc0b0310, yp=-0, wp=1) at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
40            y+=yp;
(gdb) backtrace
#0  0x000000000042fa86 in _decisionTreeTrainer::YW_struct::add (this=0xabc0b0310, yp=-0, wp=1) at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
#1  0x0000000000426f99 in _decisionTreeTrainer::TrainTarget::yw_LS_add (this=0x1626100, yw=..., w=1, res=0) at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:118
#2  0x000000000042596a in _decisionTreeTrainer::TrainTarget::compute_yw (this=0x1626100, reverse_index=0x162cfe0, b=28, e=25, yw=0x162d190, num_yw=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:268
#3  0x00000000005300b8 in _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR::map_range(int, int, int) (this=0xcfef70, tid=7, b=28, e=25)
    at D:\Users\nekit\Downloads\fast_rgf\src\forest\dtree_trainer.cpp:550
#4  0x0000000000445e0d in rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool) (this=0xcfefb0, mr=..., begin=0, end=25, tid=7, nthreads=8, run_range=true) at D:/Users/nekit/Downloads/fast_rgf/include/utils.h:211
#5  0x0000000000404ddf in rgf::MapReduceRunner::run_threads<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR> () at D:/Users/nekit/Downloads/fast_rgf/include/utils.h:256
#6  0x0000000063607498 in libgomp-1!gomp_free_thread () from D:\Program Files\MinGW\mingw64\bin\libgomp-1.dll
#7  0x000000006494b98c in pthread_create_wrapper () from D:\Program Files\MinGW\mingw64\opt\bin\libwinpthread-1.dll
#8  0x00007ffef4ecb2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#9  0x00007ffef4ecb38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007ffef4f38364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#11 0x00007ffef5717091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#12 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

UPD:

Operating System: Ubuntu 18.04 64-bit
Compiler: g++ 5.5.0/7.3.0

Output:
No errors.

UPD2:

Operating System: Ubuntu 16.04 32-bit
Compiler: g++ 5.4.0

Output:
No errors.

UPD3:

Operating System: macOS 10.12 64-bit
Compiler: g++ 8.1.0

Output:
No errors.

UPD4:

Operating System: Windows 10 64-bit
Compiler: MinGW-w64 (x86_64-posix-seh-rev0) g++ 8.1.0

Output:

D:\Users\nekit\Downloads\fast_rgf\examples\ex2>gdb --args "../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x" trn.target=REAL model.save="outputs/model-rgf"
GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../../bin/forest_train...done.
(gdb) run
Starting program: D:\Users\nekit\Downloads\fast_rgf\bin\forest_train.exe "-config=inputs/config" "trn.x-file=inputs/housing.train" "trn.x-file_format=y.x" "trn.target=REAL" "model.save=outputs/model-rgf"
[New Thread 4548.0x1c98]
[New Thread 4548.0x26cc]
[New Thread 4548.0x23d0]
[New Thread 4548.0x1c04]

reading options from configuration file <inputs/config>

 using up to 8 openmp threads
 the number of threads is set to 8, which is the maximum number of logical hardware threads including hyperthreads
 the optimal number of threads is often the number of physical cores that may be smaller than 8
 for example, to achieve better performance, you may try to set the number of threads to 4

loading training data ...
  trn.target=REAL
  trn.x-file_format=y.x
  trn.x-file=inputs/housing.train
  trn.y-file=
  trn.w-file=
[New Thread 4548.0x270c]
[New Thread 4548.0xb0c]
[New Thread 4548.0x23b8]
[New Thread 4548.0xe30]
[New Thread 4548.0x2468]
[New Thread 4548.0x1b74]
[New Thread 4548.0x2740]
loading time: wall time=0 seconds; cpu time=0 seconds.
discretizing training data ...
  discretize.dense.min_bucket_weights=5.000000
  discretize.dense.max_buckets=250
  discretize.dense.lamL2=10
  discretize.sparse.min_bucket_weights=5.000000
  discretize.sparse.max_buckets=200
  discretize.sparse.max_features=80000
  discretize.sparse.min_occrrences=5
  discretize.sparse.missing_type=MIN
  discretize.sparse.lamL2=2.000000
discritizer training time: wall time=0.0010082 seconds; cpu time=0.001 seconds.


training decision forest ...
  dtree.loss=LS
  dtree.max_level=6
  dtree.max_nodes=50
  dtree.new_tree_gain_ratio=1.0
  dtree.min_sample=5
  dtree.lamL1=10
  dtree.lamL2=1000
  forest.opt=rgf
  forest.ntrees=1000
  forest.eval_frequency=50
  forest.save_frequency=0


  training data size= 25 with 0 dense features and 1 sparse feature groups


build tree     1/ 1000
Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4548.0x2740]
0x000000000042f5f6 in _decisionTreeTrainer::YW_struct::add (this=0xabac29e60, yp=-0, wp=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
40            y+=yp;
(gdb) backtrace
#0  0x000000000042f5f6 in _decisionTreeTrainer::YW_struct::add (this=0xabac29e60, yp=-0, wp=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
#1  0x0000000000426cd0 in _decisionTreeTrainer::TrainTarget::yw_LS_add (this=0x7e550, yw=..., w=1, res=0)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:118
#2  0x000000000042575a in _decisionTreeTrainer::TrainTarget::compute_yw (this=0x7e550, reverse_index=0x7c960, b=28,
    e=25, yw=0x7c8a0, num_yw=1) at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:268
#3  0x00000000005316e6 in _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR::map_range(int, int, int) (this=0xd3ef70, tid=7, b=28, e=25)
    at D:\Users\nekit\Downloads\fast_rgf\src\forest\dtree_trainer.cpp:550
#4  0x0000000000445649 in rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool) (this=0xd3efb0, mr=...,
    begin=0, end=25, tid=7, nthreads=8, run_range=true) at D:/Users/nekit/Downloads/fast_rgf/include/utils.h:211
#5  0x0000000000404d28 in rgf::MapReduceRunner::run_threads<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, bool) (this=0x100525f3c, mr=..., begin=4635445,
    end=62914048, run_range=false) at D:/Users/nekit/Downloads/fast_rgf/include/utils.h:256
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

UPD5:

Operating System: Windows Server 2016 64-bit
Compiler: MinGW-w64 (x86_64-posix-seh-rev0) g++ 6.4.0

Output:

C:\Users\ntitov\Downloads\fast_rgf\examples\ex2>gdb --args "../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x" trn.target=REAL model.save="outputs/model-rgf"
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../../bin/forest_train...done.
(gdb) run
Starting program: C:\Users\ntitov\Downloads\fast_rgf\bin\forest_train.exe "-config=inputs/config" "trn.x-file=inputs/housing.train" "trn.x-file_format=y.x" "trn.target=REAL" "model.save=outputs/model-rgf"
[New Thread 8556.0x1748]
[New Thread 8556.0xec0]
[New Thread 8556.0x1d28]
[New Thread 8556.0x8e4]

reading options from configuration file <inputs/config>

 using up to 12 openmp threads
 the number of threads is set to 12, which is the maximum number of logical hardware threads including hyperthreads
 the optimal number of threads is often the number of physical cores that may be smaller than 12
 for example, to achieve better performance, you may try to set the number of threads to 6

loading training data ...
  trn.target=REAL
  trn.x-file_format=y.x
  trn.x-file=inputs/housing.train
  trn.y-file=
  trn.w-file=
[New Thread 8556.0x5e8]
[New Thread 8556.0x1c2c]
[New Thread 8556.0x19f0]
[New Thread 8556.0x10e0]
[New Thread 8556.0x1740]
[New Thread 8556.0x1120]
[New Thread 8556.0x1820]
[New Thread 8556.0x908]
[New Thread 8556.0xb04]
[New Thread 8556.0x21b4]
[New Thread 8556.0x95c]
loading time: wall time=0.0160004 seconds; cpu time=0.016 seconds.
discretizing training data ...
  discretize.dense.min_bucket_weights=5.000000
  discretize.dense.max_buckets=250
  discretize.dense.lamL2=10
  discretize.sparse.min_bucket_weights=5.000000
  discretize.sparse.max_buckets=200
  discretize.sparse.max_features=80000
  discretize.sparse.min_occrrences=5
  discretize.sparse.missing_type=MIN
  discretize.sparse.lamL2=2.000000
discritizer training time: wall time=0.0029965 seconds; cpu time=0.002 seconds.


training decision forest ...
  dtree.loss=LS
  dtree.max_level=6
  dtree.max_nodes=50
  dtree.new_tree_gain_ratio=1.0
  dtree.min_sample=5
  dtree.lamL1=10
  dtree.lamL2=1000
  forest.opt=rgf
  forest.ntrees=1000
  forest.eval_frequency=50
  forest.save_frequency=0


  training data size= 25 with 0 dense features and 1 sparse feature groups


build tree     1/ 1000
Thread 15 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 8556.0x95c]
0x0000000000430656 in _decisionTreeTrainer::YW_struct::add (this=0xfeef2dc70, yp=-0, wp=1)
    at C:/Users/ntitov/Downloads/fast_rgf/src/forest/training_target.h:40
40            y+=yp;
(gdb) backtrace
#0  0x0000000000430656 in _decisionTreeTrainer::YW_struct::add (this=0xfeef2dc70, yp=-0, wp=1)
    at C:/Users/ntitov/Downloads/fast_rgf/src/forest/training_target.h:40
#1  0x0000000000427b60 in _decisionTreeTrainer::TrainTarget::yw_LS_add (this=0xd14700, yw=..., w=1, res=0)
    at C:/Users/ntitov/Downloads/fast_rgf/src/forest/training_target.h:118
#2  0x000000000042656a in _decisionTreeTrainer::TrainTarget::compute_yw (this=0xd14700, reverse_index=0xd32fa0, b=33,
    e=25, yw=0x26320, num_yw=1) at C:/Users/ntitov/Downloads/fast_rgf/src/forest/training_target.h:268
#3  0x0000000000532f78 in _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR::map_range(int, int, int) (this=0xcfef70, tid=11, b=33, e=25)
    at C:\Users\ntitov\Downloads\fast_rgf\src\forest\dtree_trainer.cpp:550
#4  0x000000000044675d in rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool) (this=0xcfefb0, mr=...,
    begin=0, end=25, tid=11, nthreads=12, run_range=true) at C:/Users/ntitov/Downloads/fast_rgf/include/utils.h:211
#5  0x0000000000404d98 in rgf::MapReduceRunner::run_threads<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR> () at C:/Users/ntitov/Downloads/fast_rgf/include/utils.h:256
#6  0x000000006360d548 in omp_in_final ()
   from C:\Program Files\mingw-w64\x86_64-6.4.0-posix-seh-rt_v5-rev0\mingw64\bin\libgomp-1.dll
#7  0x0000000064944af4 in pthread_create_wrapper ()
   from C:\Program Files\mingw-w64\x86_64-6.4.0-posix-seh-rt_v5-rev0\mingw64\opt\bin\libwinpthread-1.dll
#8  0x00007ffb987db2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#9  0x00007ffb987db38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007ffb985d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#11 0x00007ffb9aede821 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#12 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

[StrikerRUS]

@TongZhang-ML Please take a look at logs.

[StrikerRUS]

I've decided to reproduce the bug at continuous integration service, but still with no luck.

Tests (refer to #159) have been run on Ubuntu 14.04 and macOS 10.12 with g++ versions from 5 to 8. No errors occurred.

UPD:
Appveyor CI service with Windows Server 2012 and g++ 6.3.0 doesn't report any problems too.

It seems to me, that according to the logs the bug is hidden around the multithreading here (Race Condition?), and that why it's so hard to reproduce.
https://github.com/RGF-team/rgf_python/blob/5202dcec94d30a575ea45ec9ec6d5d47d0335b06/include/fast_rgf/include/utils.h#L254-L257

[StrikerRUS]

Tried without OpenMP:

Set this OFF:
https://github.com/RGF-team/rgf_python/blob/3e7858c5b600d699b8d76154f93bc8b409a13370/include/fast_rgf/CMakeLists.txt#L8

D:\Users\nekit\Downloads\fast_rgf\examples\ex2>gdb --args "../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x" trn.target=REAL model.save="outputs/model-rgf"
GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../../bin/forest_train...done.
(gdb) run
Starting program: D:\Users\nekit\Downloads\fast_rgf\bin\forest_train.exe "-config=inputs/config" "trn.x-file=inputs/housing.train" "trn.x-file_format=y.x" "trn.target=REAL" "model.save=outputs/model-rgf"
[New Thread 4352.0x28bc]
[New Thread 4352.0xc90]
[New Thread 4352.0x26a0]
[New Thread 4352.0x2540]

reading options from configuration file <inputs/config>

 using up to 8 threads
 the number of threads is set to 8, which is the maximum number of logical hardware threads including hyperthreads
 the optimal number of threads is often the number of physical cores that may be smaller than 8
 for example, to achieve better performance, you may try to set the number of threads to 4

loading training data ...
  trn.target=REAL
  trn.x-file_format=y.x
  trn.x-file=inputs/housing.train
  trn.y-file=
  trn.w-file=
[New Thread 4352.0x1644]
[New Thread 4352.0x13dc]
[Thread 4352.0x1644 exited with code 0]
[Thread 4352.0x13dc exited with code 0]
[New Thread 4352.0x29c4]
[New Thread 4352.0x1e0c]
[Thread 4352.0x29c4 exited with code 0]
[Thread 4352.0x1e0c exited with code 0]
[New Thread 4352.0x2b9c]
[Thread 4352.0x2b9c exited with code 0]
[New Thread 4352.0xdc4]
[Thread 4352.0xdc4 exited with code 0]
[New Thread 4352.0x2654]
[Thread 4352.0x2654 exited with code 0]
[New Thread 4352.0x1c54]
[Thread 4352.0x1c54 exited with code 0]
loading time: wall time=0.0156195 seconds; cpu time=0.016 seconds.
discretizing training data ...
  discretize.dense.min_bucket_weights=5.000000
  discretize.dense.max_buckets=250
  discretize.dense.lamL2=10
  discretize.sparse.min_bucket_weights=5.000000
  discretize.sparse.max_buckets=200
  discretize.sparse.max_features=80000
  discretize.sparse.min_occrrences=5
  discretize.sparse.missing_type=MIN
  discretize.sparse.lamL2=2.000000
[New Thread 4352.0xe00]
[Thread 4352.0xe00 exited with code 0]
[New Thread 4352.0x1898]
[Thread 4352.0x1898 exited with code 0]
[New Thread 4352.0x10f8]
[New Thread 4352.0x1f28]
[Thread 4352.0x10f8 exited with code 0]
[New Thread 4352.0x1f80]
[Thread 4352.0x1f28 exited with code 0]
[New Thread 4352.0xa7c]
[New Thread 4352.0x17a8]
[Thread 4352.0xa7c exited with code 0]
[Thread 4352.0x1f80 exited with code 0]
[New Thread 4352.0x2afc]
[Thread 4352.0x17a8 exited with code 0]
[Thread 4352.0x2afc exited with code 0]
[New Thread 4352.0x247c]
[New Thread 4352.0x1434]
[New Thread 4352.0x2148]
[Thread 4352.0x247c exited with code 0]
[Thread 4352.0x1434 exited with code 0]
[Thread 4352.0x2148 exited with code 0]
[New Thread 4352.0x2bc]
[New Thread 4352.0x1404]
[New Thread 4352.0x2514]
[Thread 4352.0x2bc exited with code 0]
[Thread 4352.0x1404 exited with code 0]
[Thread 4352.0x2514 exited with code 0]
[New Thread 4352.0x1edc]
[New Thread 4352.0x2470]
[Thread 4352.0x1edc exited with code 0]
[Thread 4352.0x2470 exited with code 0]
discritizer training time: wall time=0.0312523 seconds; cpu time=0.032 seconds.


training decision forest ...
  dtree.loss=LS
  dtree.max_level=6
  dtree.max_nodes=50
  dtree.new_tree_gain_ratio=1.0
  dtree.min_sample=5
  dtree.lamL1=10
  dtree.lamL2=1000
  forest.opt=rgf
  forest.ntrees=1000
  forest.eval_frequency=50
  forest.save_frequency=0


  training data size= 25 with 0 dense features and 1 sparse feature groups


build tree     1/ 1000[New Thread 4352.0x12d4]
[Thread 4352.0x12d4 exited with code 0]
[New Thread 4352.0x2bc0]
[Thread 4352.0x2bc0 exited with code 0]
[New Thread 4352.0x444]
[Thread 4352.0x444 exited with code 0]
[New Thread 4352.0x18dc]
[Thread 4352.0x18dc exited with code 0]
[New Thread 4352.0x29a8]
[Thread 4352.0x29a8 exited with code 0]
[New Thread 4352.0x2708]
[Thread 4352.0x2708 exited with code 0]
[New Thread 4352.0x374]
[Thread 4352.0x374 exited with code 0]
[New Thread 4352.0x2a74]
[Thread 4352.0x2a74 exited with code 0]
[New Thread 4352.0x14b4]
[Thread 4352.0x14b4 exited with code 0]
[New Thread 4352.0x1088]
[Thread 4352.0x1088 exited with code 0]
[New Thread 4352.0x63c]
[Thread 4352.0x63c exited with code 0]
[New Thread 4352.0x28c4]
[Thread 4352.0x28c4 exited with code 0]
[New Thread 4352.0x99c]
[Thread 4352.0x99c exited with code 0]
[New Thread 4352.0x1940]
[Thread 4352.0x1940 exited with code 0]
[New Thread 4352.0x1270]
[Thread 4352.0x1270 exited with code 0]
[New Thread 4352.0x1ef0]
[Thread 4352.0x1ef0 exited with code 0]
[New Thread 4352.0x2658]
[Thread 4352.0x2658 exited with code 0]
[New Thread 4352.0x1b3c]
[Thread 4352.0x1b3c exited with code 0]
[New Thread 4352.0x28b0]
[Thread 4352.0x28b0 exited with code 0]
[New Thread 4352.0x40c]
[Thread 4352.0x40c exited with code 0]
[New Thread 4352.0x6d0]
[Thread 4352.0x6d0 exited with code 0]
[New Thread 4352.0x2464]
[Thread 4352.0x2464 exited with code 0]
[New Thread 4352.0x2330]
[Thread 4352.0x2330 exited with code 0]
[New Thread 4352.0x260]

Thread 52 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4352.0x260]
0x000000000042d086 in _decisionTreeTrainer::YW_struct::add (this=0xabc21d750, yp=-0, wp=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
40            y+=yp;
(gdb) backtrace
#0  0x000000000042d086 in _decisionTreeTrainer::YW_struct::add (this=0xabc21d750, yp=-0, wp=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:40
#1  0x0000000000424760 in _decisionTreeTrainer::TrainTarget::yw_LS_add (this=0x1757d70, yw=..., w=1, res=0)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:118
#2  0x000000000042311a in _decisionTreeTrainer::TrainTarget::compute_yw (this=0x1757d70, reverse_index=0x1757b30, b=28, e=25, yw=0x17588b0, num_yw=1)
    at D:/Users/nekit/Downloads/fast_rgf/src/forest/training_target.h:268
#3  0x0000000000550c56 in _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR::map_range(int, int, int) (this=0xf6ef70, tid=7, b=28,
    e=25) at D:\Users\nekit\Downloads\fast_rgf\src\forest\dtree_trainer.cpp:550
#4  0x0000000000444249 in rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool) (this=0xf6efb0, mr=..., begin=0, end=25, tid=7, nthreads=8,
    run_range=true) at D:/Users/nekit/Downloads/fast_rgf/include/utils.h:211
#5  0x00000000005317e7 in std::__invoke_impl<void, void (rgf::MapReduceRunner::*)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>, int, int, int, int, bool>(std::__invoke_memfun_deref, void (rgf::MapReduceRunner::*&&)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*&&, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>&&, int&&, int&&, int&&, int&&, bool&&) (__f=
    @0x1757f70: (void (rgf::MapReduceRunner::*)(rgf::MapReduceRunner * const, _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::Tree_YW_MR &, int, int, int, int, bool)) 0x4441c0 <rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool)>, __t=@0x1757f68: 0xf6efb0, __args#0=...,
    __args#1=@0x1757f58: 0, __args#2=@0x1757f54: 25, __args#3=@0x1757f50: 7, __args#4=@0x1757f4c: 8, __args#5=@0x1757f48: true)
    at D:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/bits/invoke.h:73
#6  0x0000000000549005 in std::__invoke<void (rgf::MapReduceRunner::*)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>, int, int, int, int, bool>(void (rgf::MapReduceRunner::*&&)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*&&, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_de---Type <return> to continue, or q <return> to quit---
cisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>&&, int&&, int&&, int&&, int&&, bool&&) (__fn=
    @0x1757f70: (void (rgf::MapReduceRunner::*)(rgf::MapReduceRunner * const, _decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::Tree_YW_MR &, int, int, int, int, bool)) 0x4441c0 <rgf::MapReduceRunner::single_thread_map_reduce<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR>(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR&, int, int, int, int, bool)>, __args#0=@0x1757f68: 0xf6efb0,
    __args#1=..., __args#2=@0x1757f58: 0, __args#3=@0x1757f54: 25, __args#4=@0x1757f50: 7, __args#5=@0x1757f4c: 8, __args#6=@0x1757f48: true)
    at D:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/bits/invoke.h:95
#7  0x00000000004fb989 in std::thread::_Invoker<std::tuple<void (rgf::MapReduceRunner::*)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>, int, int, int, int, bool> >::_M_invoke<0ull, 1ull, 2ull, 3ull, 4ull, 5ull, 6ull, 7ull>(std::_Index_tuple<0ull, 1ull, 2ull, 3ull, 4ull, 5ull, 6ull, 7ull>) (this=0x1757f48)
    at D:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/thread:234
#8  0x00000000004fb9e7 in std::thread::_Invoker<std::tuple<void (rgf::MapReduceRunner::*)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>, int, int, int, int, bool> >::operator()() (this=0x1757f48)
    at D:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/thread:243
#9  0x00000000004f68dc in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgf::MapReduceRunner::*)(_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)&), rgf::MapReduceRunner*, std::reference_wrapper<_decisionTreeTrainer::TreeToIndex<unsigned short, int, unsigned char>::update_predictions(_decisionTreeTrainer::TrainTarget&, rgf::DecisionTree<unsigned short, int, unsigned char>::TrainParam&, rgf::Timer&, rgf::Timer&)::Tree_YW_MR(int, int, int, int, bool)>, int, int, int, int, bool> > >::_M_run() (this=0x1757f40)
    at D:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/thread:186
#10 0x000000000055329f in execute_native_thread_routine ()
#11 0x0000000064944a94 in pthread_create_wrapper () from D:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\opt\bin\libwinpthread-1.dll
#12 0x00007ff9228cb2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#13 0x00007ff9228cb38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#14 0x00007ff922938364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#15 0x00007ff924707091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

[fukatani]

yw_LS_add(my_yw[j],my_w?my_w[i]:1.0,my_r[i]);

Any variable is not nullptr at here?

For example, could you add code for dumping as follows?

printf("%f\n", my_yw[j]);  // dump my_yw[j]
yw_LS_add(my_yw[j],my_w?my_w[i]:1.0,my_r[i]);

With my expectation, it will crash with printf function.

[fukatani]

Or

assert(j < num_yw);
printf("%f\n", my_yw[j]);  // dump my_yw[j]
yw_LS_add(my_yw[j],my_w?my_w[i]:1.0,my_r[i]);

If j>= num_yw, the code referencing my_yw[j] will crash.

[StrikerRUS]

@fukatani Seems you're right!

      if (loss==TrainLoss::LS) {
	for (i=0; i<size; i++) {
	  j= my_i[i];
	  assert(j < num_yw);
	  printf("%f\n", my_yw[j]);  // dump my_yw[j]
	  yw_LS_add(my_yw[j],my_w?my_w[i]:1.0,my_r[i]);
	}
      }

results in

D:\Users\nekit\Downloads\fast_rgf\examples\ex2>"../../bin/forest_train" -config="inputs/config" trn.x-file="inputs/housing.train" trn.x-file_format="y.x" trn.target=REAL model.save="outputs/model-rgf"

reading options from configuration file <inputs/config>

 using up to 8 openmp threads
 the number of threads is set to 8, which is the maximum number of logical hardware threads including hyperthreads
 the optimal number of threads is often the number of physical cores that may be smaller than 8
 for example, to achieve better performance, you may try to set the number of threads to 4

loading training data ...
  trn.target=REAL
  trn.x-file_format=y.x
  trn.x-file=inputs/housing.train
  trn.y-file=
  trn.w-file=
loading time: wall time=0.0032088 seconds; cpu time=0.003 seconds.
discretizing training data ...
  discretize.dense.min_bucket_weights=5.000000
  discretize.dense.max_buckets=250
  discretize.dense.lamL2=10
  discretize.sparse.min_bucket_weights=5.000000
  discretize.sparse.max_buckets=200
  discretize.sparse.max_features=80000
  discretize.sparse.min_occrrences=5
  discretize.sparse.missing_type=MIN
  discretize.sparse.lamL2=2.000000
discritizer training time: wall time=0.0011295 seconds; cpu time=0.001 seconds.


training decision forest ...
  dtree.loss=LS
  dtree.max_level=6
  dtree.max_nodes=50
  dtree.new_tree_gain_ratio=1.0
  dtree.min_sample=5
  dtree.lamL1=10
  dtree.lamL2=1000
  forest.opt=rgf
  forest.ntrees=1000
  forest.eval_frequency=50
  forest.save_frequency=0


  training data size= 25 with 0 dense features and 1 sparse feature groups


build tree     1/ 10000.0000Assertion failed!

Program: D:\Users\nekit\Downloads\fast_rgf\bin\forest_train.exe
File: D:\Users\nekit\Downloads\fast_rgf\src\forest\training_target.h, Line 268

Expression: j < num_yw

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
00
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000

[fukatani]

Does assert passes in other OS?
If so what variable is different with Mac OS?

[StrikerRUS]

Yes, on macOS all asserts passed.

Caution! Huge log.

macOS log.txt