Long SQL text and external data

[bgranvea]

I have a long SQL request (>16000 characters) which uses external data and I get a "400 bad request" response from server.

I see that the SQL text is put in URI in this case and that it is related to this problem:

(ClickHouseStatementImpl)

            // write sql in query params when there is external data
            // as it is impossible to pass both external data and sql in body
            // TODO move sql to request body when it is supported in clickhouse
            uri = buildRequestUri(
                    sql,
                    externalData,
                    additionalClickHouseDBParams,
                    additionalRequestParams,
                    ignoreDatabase
            );

Is there any progress regarding this issue? Is there a workaround?

[zhicwu]

Yes, it should be put into request body. This is same issue as #607.

Unfortunately I currently don't have a workaround for this. Let me fix this in these two days.

[zhicwu]

I can reproduce the issue but moving query to request body didn't work when external data is used. Perhaps you can load external data into a temporary table and then issue a query without external data?

[bgranvea]

I found a way to reduce the size of my query so it is not a problem anymore.

I was using temporary tables before but external data are really more comfortable: no need to create a session, create a table, insert data.

My initial question was more about this comment:

// write sql in query params when there is external data
// as it is impossible to pass both external data and sql in body
// TODO move sql to request body when it is supported in clickhouse

it implies that there is a limitation in Clickhouse HTTP protocol. Do you know if it is still relevant or if someone is working on it?

alternatively, if you tell me that gRPC will be used in a few months, there is no need to fix this.

Thanks.

[zhicwu]

Sorry I was wrong. The limitation seems no longer exist. The fix has been merged into develop branch.

Also I'm curious about usage of external data. I agree that it's more convenient compare to temp table, but where is the data from? If it's dumped from another database, why not use JDBC bridge or so to issue a distributed query? We used to have offline data in flat files but now most of them are imported into database tables.