change sql image to markdown by cyongli · Pull Request #7 · apache/doris

cyongli · 2017-08-15T01:21:16Z

No description provided.

* support nullable Change-Id: I7b222d0e6ad2d3ac3e763534a74a696dcd2d8a5e * fix core when serde nullable column Change-Id: I6253da1c51ff8e07bc2008e7df3e47dc604fe358 * format Change-Id: I49ad7c83e66f7a157de46efef829b80ba72be0f4 Co-authored-by: yangzhengguo01 <yangzhengguo01@baidu.com>

[test] Trino with_clause string_functions math_functions case

``` ==12681==ERROR: LeakSanitizer: detected memory leaks Direct leak of 128 byte(s) in 1 object(s) allocated from: #0 0x562b93d0fc57 in operator new(unsigned long) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x16e02c57) #1 0x562b955fd176 in doris::ArrowWorkFlowTest::init_runtime_state() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:92 #2 0x562b955fcf58 in doris::ArrowWorkFlowTest::init() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:87 #3 0x562b9560ba7d in doris::ArrowWorkFlowTest::SetUp() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:61 #4 0x562b9f38263c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #5 0x562b9f37ca78 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #6 0x562b9f3540e1 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224470e1) #7 0x562b9f354b7f in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22447b7f) apache#8 0x562b9f35543f in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x2244843f) apache#9 0x562b9f364aff in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22457aff) apache#10 0x562b9f3833ed in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (Impl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224763ed) apache#11 0x562b9f37d932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), e/ut_build_ASAN/test/doris_be_test+0x22470932) apache#12 0x562b9f3632f4 in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224562f4) apache#13 0x562b95369a7c in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490 apache#14 0x562b95366048 in main /root/doris/be/test/testutil/run_all_tests.cpp:43 apache#15 0x7f40dcbb6554 in __libc_start_main (/lib64/libc.so.6+0x22554) Direct leak of 104 byte(s) in 1 object(s) allocated from: #0 0x562b93d0fc57 in operator new(unsigned long) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x16e02c57) #1 0x562b955fd213 in doris::ArrowWorkFlowTest::init_runtime_state() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:93 #2 0x562b955fcf58 in doris::ArrowWorkFlowTest::init() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:87 #3 0x562b9560ba7d in doris::ArrowWorkFlowTest::SetUp() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:61 #4 0x562b9f38263c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #5 0x562b9f37ca78 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #6 0x562b9f3540e1 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224470e1) #7 0x562b9f354b7f in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22447b7f) apache#8 0x562b9f35543f in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x2244843f) apache#9 0x562b9f364aff in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22457aff) apache#10 0x562b9f3833ed in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (Impl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224763ed) apache#11 0x562b9f37d932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), e/ut_build_ASAN/test/doris_be_test+0x22470932) apache#12 0x562b9f3632f4 in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224562f4) apache#13 0x562b95369a7c in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490 apache#14 0x562b95366048 in main /root/doris/be/test/testutil/run_all_tests.cpp:43 apache#15 0x7f40dcbb6554 in __libc_start_main (/lib64/libc.so.6+0x22554) SUMMARY: AddressSanitizer: 232 byte(s) leaked in 2 allocation(s). ```

## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 #2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 #3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 apache#4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 apache#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 apache#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 #2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 #3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 apache#4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 apache#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 apache#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 apache#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 apache#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 apache#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 apache#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```

…38935) ``` 1 __pthread_kill_internal (signo=6, threadid=140314099000896) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140314099000896, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fa1e2e8e476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fa1e2e747f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x000055f77d0e71ea in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #6 0x000055f77d0e5956 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #7 0x000055f77d0e59c1 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 #8 0x000055f76ffc5fbe in __clang_call_terminate () #9 0x000055f77667ef44 in doris::Defer<doris::vectorized::FunctionVariantElement::get_element_column(doris::vectorized::ColumnObject const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*)::{lambda()#1}>::~Defer() (this=0x7f9d6c06a900) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/defer_op.h:37 ```

chaoyli added 3 commits August 15, 2017 09:11

change picture to word

321a1d9

change picture to word

d893b8a

add create partition sql

0f5bc58

morningman merged commit 65c0d95 into apache:master Aug 15, 2017

wangbo mentioned this pull request Sep 30, 2019

v2 segment support string encode(#1766) #1816

Merged

wangbo mentioned this pull request Oct 11, 2019

Segment v2 use string's real length(#1943) #1944

Merged

wangbo mentioned this pull request Oct 22, 2019

Segment v2 stream load core dump #2037

Closed

zhangy5 mentioned this pull request Oct 22, 2019

Segment V2 core dump when schema change #2038

Closed

zhangy5 mentioned this pull request Nov 6, 2019

Read Doris data through spark error，when date column is null #2138

Closed

HangyuanLiu mentioned this pull request Nov 13, 2019

BE crashed during the upgrade #2189

Closed

kangpinghuang mentioned this pull request Nov 14, 2019

too many open files when pressure test #2196

Closed

jmk1011 mentioned this pull request Dec 23, 2019

be cored when select BITMAP_UNION after schema change #2529

Closed

worker24h mentioned this pull request Feb 17, 2020

Invalid memory address in doris::memory_copy #2919

Closed

marising mentioned this pull request Jun 1, 2020

[Bug]If the key column of the table is a double / float type, it will crash when checksum #3735

Closed

morningman mentioned this pull request Jun 23, 2020

[Bug] BE crash occasionally #3929

Closed

ZhangYu0123 mentioned this pull request Sep 3, 2020

[BUG] Fix segment group add zone map bug when schema change. #4527

Closed

caiconghui mentioned this pull request Mar 3, 2021

[Ut] run topn_function_test failed in be unit test #5450

Closed

xinghuayu007 mentioned this pull request Apr 16, 2021

[Coredump] data sink node close causes coredump #5664

Closed

xqinghu mentioned this pull request May 20, 2021

Bitmap过大导致Be宕掉 #5849

Closed

xinyiZzz mentioned this pull request Dec 16, 2021

[Bug] When high concurrency, BE crash reports MissingRequiredFields in SendRpcResponse #7412

Closed

3 tasks

morrySnow pushed a commit to morrySnow/incubator-doris that referenced this pull request Mar 16, 2022

Merge pull request apache#7 from dataroaring/trino_case

7926ef4

[test] Trino with_clause string_functions math_functions case

caiconghui mentioned this pull request Mar 21, 2022

[Bug] lru cache ut failed #8571

Closed

3 tasks

This was referenced Mar 30, 2022

[Bug] asan report use after free due to bad deconstruct order in IndexChannel #8762

Closed

[Bug] tpcds sf1000 core dump #8815

Closed

xinyiZzz mentioned this pull request Apr 25, 2022

[Bug] Status::ConstructErrorStatus core dump #9218

Closed

3 tasks

xinyiZzz mentioned this pull request May 5, 2022

[Bug] Compaction Rowset::end_version null pointer #9371

Closed

3 tasks

ycsongtcs mentioned this pull request Aug 1, 2024

benchmark_tool报错 #38714

Closed

3 tasks

biohazard4321 mentioned this pull request Aug 12, 2024

[fix](cooldown)No need to rdlock inside get_cooldown_tablets, there's enough rdlock inside tablet internal function calls. #39211

Merged

Vallishp mentioned this pull request Aug 17, 2024

[Bug](http-api) fix core dump on API check_rpc_channel coz exec_env not initialized #39519

Closed

3 tasks

yiguolei mentioned this pull request Aug 19, 2024

[bugfix](core) runtime state is destroyed when spill task is run #39566

Merged

Vallishp mentioned this pull request Sep 2, 2024

[fix] _cur_reader can be null in exception cases in VFileScanner #40273

Merged

Gabriel39 mentioned this pull request Oct 12, 2024

[fix](pipeline) Prevent re-scheduling a task at the same time #41743

Merged

sollhui mentioned this pull request Dec 16, 2024

[fix](csv reader) fix core dump when parsing csv with enclose #45485

Merged

16 tasks

yiguolei mentioned this pull request Jan 15, 2025

[bugfix](memtable) arena is freed early and will cause use after free #46997

Merged

16 tasks

This was referenced Feb 21, 2025

[fix](scheduler) Fix invalid access after freed #48168

Merged

[fix](schema scan) Fix invalid pointer access #48313

Merged

[fix](schema scan) Fix invalid pointer access #48370

Merged

Gabriel39 mentioned this pull request Mar 31, 2025

[fix](shutdown) Fix coredump due to shutdown in the wrong order #49700

Merged

16 tasks

zddr mentioned this pull request May 14, 2025

branch-2.1: [enhance](mtmv)add debug log for insert into plan #50884

Closed

yiguolei mentioned this pull request Jun 16, 2025

[be](gracefulexit) should stop async result writer when graceful exit #51743

Merged

16 tasks

This was referenced Jul 7, 2025

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

Merged

[fix](httpserver) Fix lsan check error when stop libevent server #53081

Merged

Gabriel39 mentioned this pull request Jul 21, 2025

[fix](cloud) Fix lifecycle in lambda function #53605

Merged

16 tasks

GoGoWen mentioned this pull request Aug 11, 2025

[BUG] Filter not pushdown to ScanNode #54577

Closed

yiguolei mentioned this pull request Aug 14, 2025

[NoMergeLatest](exit) call stop before brpc server stop to stop queries and allow brpc exist gracefully #54781

Closed

16 tasks

Gabriel39 mentioned this pull request Sep 25, 2025

[fix](debug) Fix illegal access to runtime states #56439

Merged

16 tasks

msridhar78 mentioned this pull request Oct 1, 2025

Added bounds check in ColumnVector::insert_indices_from() #56696

Closed

16 tasks

dh-cloud mentioned this pull request Nov 5, 2025

[Bug] inconsistent results between select distinct and count(distinct) on multi-column combinations #57707

Closed

3 tasks

Gabriel39 mentioned this pull request Dec 17, 2025

[fix](predicate) Fix use-after-free caused by string predicate #59098

Merged

16 tasks

morrySnow mentioned this pull request Mar 26, 2026

[feat](join) support outer join reorder in dphyper #61146

Merged

16 tasks

Gabriel39 mentioned this pull request Apr 15, 2026

[fix](column) Fix incorrect for-loop #62517

Merged

16 tasks

morningman mentioned this pull request Jul 23, 2026

[fix](catalog) catalog-SPI abstraction-mismatch review: paimon time-travel crash, partition/schema fidelity, SPI convergence & dead-code cleanup #65970

Merged

15 tasks

felix199103 mentioned this pull request Aug 21, 2026

[Bug][Cloud] RScan_normal ThreadPool idle workers never shrink; OS threads grow far beyond max_threads=512 #66997

Open

3 tasks

foxtail463 mentioned this pull request Sep 1, 2026

[fix](binlog) Keep allocated LSN map alive during group flush cleanup #67384

Closed

Navigation Menu

Uh oh!

change sql image to markdown - #7

change sql image to markdown#7
morningman merged 3 commits into
apache:masterfrom
cyongli:master

cyongli commented Aug 15, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cyongli commented Aug 15, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants