Skip to content

Add 'TooManyRequestsException' error code to known list in CoreErrorsMapper - #9

Merged
alexey-milovidov merged 1 commit into
ClickHouse:masterfrom
ianton-ru:yandex_s3_throttling
Sep 21, 2021
Merged

Add 'TooManyRequestsException' error code to known list in CoreErrorsMapper#9
alexey-milovidov merged 1 commit into
ClickHouse:masterfrom
ianton-ru:yandex_s3_throttling

Conversation

@ianton-ru

@ianton-ru ianton-ru commented Sep 9, 2021

Copy link
Copy Markdown

Description of changes:

Error code 'TooManyRequestsException' is received from Yandex Object Storage when request throttled.

Change turns on retry stategy in this case.

Without this change ClickHouse catched exception:

2021.08.31 18:43:00.806374 [ 19151 ] {3e96657c-f8e8-4139-bc1d-5bee08ac6367} <Error> executeQuery: Code: 499, e.displayText() = DB::Exception: 100: Unable to parse ExceptionName: TooManyRequestsException Message: Please reduce your request rate. (version 21.7.8.58 (official build)) (from [::ffff:192.168.2.10]:45906) (in query: CREATE TABLE mdb_system.read_sli_part (`shard` String) ENGINE = MergeTree PARTITION BY tuple() ORDER BY shard SETTINGS index_granularity = 8192 ), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8d321da in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
1. void DB::throwIfError<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error>(Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error>&) @ 0xf69aef0 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
2. DB::DiskS3::checkObjectExists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xf6986b1 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
3. DB::DiskS3::findLastRevision() @ 0xf697a2f in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
4. DB::DiskS3::startup() @ 0xf69107b in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
5. std::__1::shared_ptr<DB::IDisk> std::__1::__function::__policy_invoker<std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>)>::__call_impl<std::__1::__function::__default_alloc_func<DB::registerDiskS3(DB::DiskFactory&)::$_1, std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>)> >(std::__1::__function::__policy_storage const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>&&) (.llvm.14105140438014115018) @ 0xf6859fe in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
6. DB::DiskFactory::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>) const @ 0xf66aea9 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
7. DB::DiskSelector::DiskSelector(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>) @ 0xf666fbf in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
8. void std::__1::allocator<DB::DiskSelector>::construct<DB::DiskSelector, Poco::Util::AbstractConfiguration const&, char const* const&, std::__1::shared_ptr<DB::Context const> >(DB::DiskSelector*, Poco::Util::AbstractConfiguration const&, char const* const&, std::__1::shared_ptr<DB::Context const>&&) @ 0xf75d13f in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
9. DB::Context::getDiskSelector(std::__1::lock_guard<std::__1::mutex>&) const @ 0xf7391eb in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
10. DB::Context::getStoragePolicySelector(std::__1::lock_guard<std::__1::mutex>&) const @ 0xf725dbd in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
11. DB::Context::getStoragePolicy(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xf739349 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
12. DB::MergeTreeData::getStoragePolicy() const @ 0x10377fb5 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
13. DB::MergeTreeData::getRelativeDataPathsWithDisks() const @ 0x103778ee in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
14. DB::MergeTreeData::MergeTreeData(DB::StorageID const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::StorageInMemoryMetadata const&, std::__1::shared_ptr<DB::Context>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreeData::MergingParams const&, std::__1::unique_ptr<DB::MergeTreeSettings, std::__1::default_delete<DB::MergeTreeSettings> >, bool, bool, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0x1036de9e in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
15. DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::StorageInMemoryMetadata const&, bool, std::__1::shared_ptr<DB::Context>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreeData::MergingParams const&, std::__1::unique_ptr<DB::MergeTreeSettings, std::__1::default_delete<DB::MergeTreeSettings> >, bool) @ 0x100b8080 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
16. DB::create(DB::StorageFactory::Arguments const&) @ 0x1059eb87 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
17. DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x10032261 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
18. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&) @ 0xf871201 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
19. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0xf86d3c3 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
20. DB::InterpreterCreateQuery::execute() @ 0xf8734fc in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
21. DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) @ 0xfe1d133 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
22. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::__1::shared_ptr<DB::Context>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::optional<DB::FormatSettings> const&) @ 0xfe20bfd in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
23. DB::HTTPHandler::processQuery(std::__1::shared_ptr<DB::Context>, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0x106342ed in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
24. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x10637a49 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
25. DB::HTTPServerConnection::run() @ 0x106b93d0 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
26. Poco::Net::TCPServerConnection::start() @ 0x1338918f in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
27. Poco::Net::TCPServerDispatcher::run() @ 0x1338ac1a in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
28. Poco::PooledThread::run() @ 0x134bda99 in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
29. Poco::ThreadImpl::runnableEntry(void*) @ 0x134b9d2a in /usr/lib/debug/.build-id/95/f91f78b10ac6f0e542a4c4f8d7b8162144de39.debug
30. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
31. clone @ 0x12171f in /lib/x86_64-linux-gnu/libc-2.27.so

@ianton-ru
Add 'TooManyRequestsException' error code to known list in CoreErrors…
b8ff2ca 
…Mapper.

This code is received from Yandex Object Storage when request throttled.

Change turns on retry stategy in this case.
@alexey-milovidov alexey-milovidov self-assigned this Sep 21, 2021
@alexey-milovidov
alexey-milovidov merged commit 00b0360 into ClickHouse:master Sep 21, 2021
@ianton-ru ianton-ru mentioned this pull request Oct 25, 2021
Merged
excitoon pushed a commit to excitoon-favorites/aws-sdk-cpp that referenced this pull request May 22, 2022
@alexey-milovidov @excitoon
Merge pull request ClickHouse#9 from ianton-ru/yandex_s3_throttling
deefe7e 
Add 'TooManyRequestsException' error code to known list in CoreErrorsMapper
excitoon pushed a commit to excitoon-favorites/aws-sdk-cpp that referenced this pull request Jun 9, 2022
@alexey-milovidov @excitoon
Merge pull request ClickHouse#9 from ianton-ru/yandex_s3_throttling
d279d16 
Add 'TooManyRequestsException' error code to known list in CoreErrorsMapper
excitoon added a commit to excitoon-favorites/ClickHouse that referenced this pull request Jun 20, 2022
@excitoon
Tests for ClickHouse/aws-sdk-cpp#4, ClickHouse/aws-sdk-cpp#5 and Clic…
ef63369 
…kHouse/aws-sdk-cpp#9.
alexey-milovidov pushed a commit that referenced this pull request Jun 23, 2022
@excitoon
Revert "Merge pull request #9 from ianton-ru/yandex_s3_throttling"
7fbfee1 
This reverts commit 00b0360, reversing
changes made to 06aa875.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@alexey-milovidov alexey-milovidov

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ianton-ru @alexey-milovidov