thrift-sasl 0.4.2 invalid query handle #28

Maxsparrow · 2021-01-14T15:07:41Z

Hi, we're experiencing an issue with thrift-sasl 0.4.2 on Python 3.7, where queries are erroring with "Invalid query handle", but only if they return more than around 50 rows.

We've tried a variety of queries and they all succeed if they return only a few rows, and all fail for over 100 rows. The larger queries fail whether we use fetchone or fetchall.

Queries look normal in the Impala logs and the Impala daemon GUI at port 25000. The logs say the user authenticated and there are no errors, and the GUI says the query is in state 'FINISHED'.

This is a kerberized Impala cluster. Everything works fine if we use thrift-sasl 0.2.1.

pip freeze output:

bitarray==1.6.1
Cython==0.29.21
impyla==0.16.3
kerberos==1.3.1
ply==3.11
sasl==0.2.1
six==1.15.0
thrift==0.13.0
thrift-sasl==0.4.2
thriftpy2==0.4.12

Example with impyla debug logging enabled:

>>> query = """SHOW DATABASES"""
>>> conn = connect(host='our-impala-host-name', auth_mechanism='GSSAPI', timeout=20, use_ssl=False, ca_cert=None, ldap_user=None, ldap_password=None)
>>> cursor = conn.cursor()
>>> cursor.execute(query)
>>> f = cursor.fetchall()
Connecting to HiveServer2 our-impala-host-name:21050 with GSSAPI authentication mechanism
get_socket: host=our-impala-host-name port=21050 use_ssl=False ca_cert=None
sock=<thriftpy2.transport.socket.TSocket object at 0x7fd7334c5110>
get_transport: socket=<thriftpy2.transport.socket.TSocket object at 0x7fd7334c5110> host=our-impala-host-name kerberos_service_name=impala auth_mechanism=GSSAPI user=None password=fuggetaboutit
transport=<thrift_sasl.TSaslClientTransport object at 0x7fd733599310> protocol=<thriftpy2.protocol.binary.TBinaryProtocol object at 0x7fd733178390> service=<thriftpy2.thrift.TClient object at 0x7fd7335b2610>
HiveServer2Connection(service=<impala.hiveserver2.HS2Service object at 0x7fd73351f510>, default_db=None)
Getting a cursor (Impala session)
.cursor(): getting new session_handle
OpenSession: req=TOpenSessionReq(client_protocol=5, username='root', password=None, configuration=None)
Attempting to open transport (tries_left=3)
Transport opened
OpenSession: resp=TOpenSessionResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), serverProtocolVersion=5, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(guid=b'4\x9d\xb4(\x05WJ\xe1\xa6\\[\xcf\xd6/\x9c\x1b', secret=b';C\xdf\xf0\xb5\x98C~\xb5{\xe8\xb2\xd0#V|')), configuration={'ABORT_ON_DEFAULT_LIMIT_EXCEEDED': '0', 'ABORT_ON_ERROR': '0', 'ALLOW_UNSUPPORTED_FORMATS': '0', 'APPX_COUNT_DISTINCT': '0', 'BATCH_SIZE': '0', 'BUFFER_POOL_LIMIT': '', 'COMPRESSION_CODEC': '', 'COMPUTE_STATS_MIN_SAMPLE_SIZE': '1073741824', 'DEBUG_ACTION': '', 'DECIMAL_V2': '0', 'DEFAULT_JOIN_DISTRIBUTION_MODE': '0', 'DEFAULT_ORDER_BY_LIMIT': '-1', 'DEFAULT_SPILLABLE_BUFFER_SIZE': '2097152', 'DISABLE_CACHED_READS': '0', 'DISABLE_CODEGEN': '0', 'DISABLE_CODEGEN_ROWS_THRESHOLD': '50000', 'DISABLE_OUTERMOST_TOPN': '0', 'DISABLE_ROW_RUNTIME_FILTERING': '0', 'DISABLE_STREAMING_PREAGGREGATIONS': '0', 'DISABLE_UNSAFE_SPILLS': '0', 'ENABLE_EXPR_REWRITES': '1', 'EXEC_SINGLE_NODE_ROWS_THRESHOLD': '100', 'EXEC_TIME_LIMIT_S': '0', 'EXPLAIN_LEVEL': '1', 'HBASE_CACHE_BLOCKS': '1', 'HBASE_CACHING': '1000', 'IDLE_SESSION_TIMEOUT': '7200', 'MAX_ERRORS': '100', 'MAX_IO_BUFFERS': '0', 'MAX_NUM_RUNTIME_FILTERS': '10', 'MAX_ROW_SIZE': '524288', 'MAX_SCAN_RANGE_LENGTH': '0', 'MEM_LIMIT': '0', 'MIN_SPILLABLE_BUFFER_SIZE': '65536', 'MT_DOP': '', 'NUM_NODES': '0', 'NUM_SCANNER_THREADS': '30', 'OPTIMIZE_PARTITION_KEY_SCANS': '0', 'PARQUET_ANNOTATE_STRINGS_UTF8': '0', 'PARQUET_ARRAY_RESOLUTION': '2', 'PARQUET_DICTIONARY_FILTERING': '1', 'PARQUET_FALLBACK_SCHEMA_RESOLUTION': '1', 'PARQUET_FILE_SIZE': '0', 'PARQUET_READ_STATISTICS': '1', 'PREFETCH_MODE': '1', 'QUERY_TIMEOUT_S': '0', 'REPLICA_PREFERENCE': '0', 'REQUEST_POOL': '', 'RESERVATION_REQUEST_TIMEOUT': '', 'RM_INITIAL_MEM': '0', 'RUNTIME_BLOOM_FILTER_SIZE': '1048576', 'RUNTIME_FILTER_MAX_SIZE': '16777216', 'RUNTIME_FILTER_MIN_SIZE': '1048576', 'RUNTIME_FILTER_MODE': '2', 'RUNTIME_FILTER_WAIT_TIME_MS': '0', 'S3_SKIP_INSERT_STAGING': '1', 'SCAN_NODE_CODEGEN_THRESHOLD': '1800000', 'SCHEDULE_RANDOM_REPLICA': '0', 'SCRATCH_LIMIT': '-1', 'SEQ_COMPRESSION_MODE': '', 'SHUFFLE_DISTINCT_EXPRS': '1', 'STRICT_MODE': '0', 'SYNC_DDL': '0', 'V_CPU_CORES': '', 'http_addr': 'a-host:25000'})
HiveServer2Cursor(service=<impala.hiveserver2.HS2Service object at 0x7fd73351f510>, session_handle=TSessionHandle(sessionId=THandleIdentifier(guid=b'4\x9d\xb4(\x05WJ\xe1\xa6\\[\xcf\xd6/\x9c\x1b', secret=b';C\xdf\xf0\xb5\x98C~\xb5{\xe8\xb2\xd0#V|')), default_config={'ABORT_ON_DEFAULT_LIMIT_EXCEEDED': '0', 'ABORT_ON_ERROR': '0', 'ALLOW_UNSUPPORTED_FORMATS': '0', 'APPX_COUNT_DISTINCT': '0', 'BATCH_SIZE': '0', 'BUFFER_POOL_LIMIT': '', 'COMPRESSION_CODEC': '', 'COMPUTE_STATS_MIN_SAMPLE_SIZE': '1073741824', 'DEBUG_ACTION': '', 'DECIMAL_V2': '0', 'DEFAULT_JOIN_DISTRIBUTION_MODE': '0', 'DEFAULT_ORDER_BY_LIMIT': '-1', 'DEFAULT_SPILLABLE_BUFFER_SIZE': '2097152', 'DISABLE_CACHED_READS': '0', 'DISABLE_CODEGEN': '0', 'DISABLE_CODEGEN_ROWS_THRESHOLD': '50000', 'DISABLE_OUTERMOST_TOPN': '0', 'DISABLE_ROW_RUNTIME_FILTERING': '0', 'DISABLE_STREAMING_PREAGGREGATIONS': '0', 'DISABLE_UNSAFE_SPILLS': '0', 'ENABLE_EXPR_REWRITES': '1', 'EXEC_SINGLE_NODE_ROWS_THRESHOLD': '100', 'EXEC_TIME_LIMIT_S': '0', 'EXPLAIN_LEVEL': '1', 'HBASE_CACHE_BLOCKS': '1', 'HBASE_CACHING': '1000', 'IDLE_SESSION_TIMEOUT': '7200', 'MAX_ERRORS': '100', 'MAX_IO_BUFFERS': '0', 'MAX_NUM_RUNTIME_FILTERS': '10', 'MAX_ROW_SIZE': '524288', 'MAX_SCAN_RANGE_LENGTH': '0', 'MEM_LIMIT': '0', 'MIN_SPILLABLE_BUFFER_SIZE': '65536', 'MT_DOP': '', 'NUM_NODES': '0', 'NUM_SCANNER_THREADS': '30', 'OPTIMIZE_PARTITION_KEY_SCANS': '0', 'PARQUET_ANNOTATE_STRINGS_UTF8': '0', 'PARQUET_ARRAY_RESOLUTION': '2', 'PARQUET_DICTIONARY_FILTERING': '1', 'PARQUET_FALLBACK_SCHEMA_RESOLUTION': '1', 'PARQUET_FILE_SIZE': '0', 'PARQUET_READ_STATISTICS': '1', 'PREFETCH_MODE': '1', 'QUERY_TIMEOUT_S': '0', 'REPLICA_PREFERENCE': '0', 'REQUEST_POOL': '', 'RESERVATION_REQUEST_TIMEOUT': '', 'RM_INITIAL_MEM': '0', 'RUNTIME_BLOOM_FILTER_SIZE': '1048576', 'RUNTIME_FILTER_MAX_SIZE': '16777216', 'RUNTIME_FILTER_MIN_SIZE': '1048576', 'RUNTIME_FILTER_MODE': '2', 'RUNTIME_FILTER_WAIT_TIME_MS': '0', 'S3_SKIP_INSERT_STAGING': '1', 'SCAN_NODE_CODEGEN_THRESHOLD': '1800000', 'SCHEDULE_RANDOM_REPLICA': '0', 'SCRATCH_LIMIT': '-1', 'SEQ_COMPRESSION_MODE': '', 'SHUFFLE_DISTINCT_EXPRS': '1', 'STRICT_MODE': '0', 'SYNC_DDL': '0', 'V_CPU_CORES': '', 'http_addr': 'a-host:25000'}, hs2_protocol_version=5)
Executing query SHOW DATABASES
_execute_async: self._buffer=Batch() self._description=None self._last_operation_active=False self._last_operation=None
_reset_state: Resetting cursor state
_execute_async: self._buffer=Batch() self._description=None self._last_operation_active=False self._last_operation=None
ExecuteStatement: req=TExecuteStatementReq(sessionHandle=TSessionHandle(sessionId=THandleIdentifier(guid=b'4\x9d\xb4(\x05WJ\xe1\xa6\\[\xcf\xd6/\x9c\x1b', secret=b';C\xdf\xf0\xb5\x98C~\xb5{\xe8\xb2\xd0#V|')), statement='SHOW DATABASES', confOverlay=None, runAsync=True)
Attempting to open transport (tries_left=3)
Transport opened
ExecuteStatement: resp=TExecuteStatementResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None))
_execute_async: self._buffer=Batch() self._description=None self._last_operation_active=True self._last_operation=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None)
Waiting for query to finish
GetOperationStatus: req=TGetOperationStatusReq(operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None))
Attempting to open transport (tries_left=3)
Transport opened
GetOperationStatus: resp=TGetOperationStatusResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), operationState=2, sqlState=None, errorCode=None, errorMessage=None, hasResultSet=None)
_wait_to_finish: waited 0.0387423038482666 seconds so far
Query finished
GetOperationStatus: req=TGetOperationStatusReq(operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None))
Attempting to open transport (tries_left=3)
Transport opened
GetOperationStatus: resp=TGetOperationStatusResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), operationState=2, sqlState=None, errorCode=None, errorMessage=None, hasResultSet=None)
_wait_to_finish: waited 0.03879117965698242 seconds so far
Fetching all result rows
__next__: buffer empty and op is active => fetching more data
description=None has_result_set=True => getting schema
GetResultSetMetadata: req=TGetResultSetMetadataReq(operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None))
Attempting to open transport (tries_left=3)
Transport opened
GetResultSetMetadata: resp=TGetResultSetMetadataResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), schema=TTableSchema(columns=[TColumnDesc(columnName='name', typeDesc=TTypeDesc(types=[TTypeEntry(primitiveEntry=TPrimitiveTypeEntry(type=7, typeQualifiers=None), arrayEntry=None, mapEntry=None, structEntry=None, unionEntry=None, userDefinedTypeEntry=None)]), position=0, comment=None), TColumnDesc(columnName='comment', typeDesc=TTypeDesc(types=[TTypeEntry(primitiveEntry=TPrimitiveTypeEntry(type=7, typeQualifiers=None), arrayEntry=None, mapEntry=None, structEntry=None, unionEntry=None, userDefinedTypeEntry=None)]), position=1, comment=None)]))
get_result_schema: schema=[('name', 'STRING', None, None, None, None, None), ('comment', 'STRING', None, None, None, None, None)]
FetchResults: req=TFetchResultsReq(operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None), orientation=0, maxRows=1024)
Attempting to open transport (tries_left=3)
Transport opened
Failed to open transport (tries_left=3)
Traceback (most recent call last):
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 1010, in _execute
    return func(request)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/thrift.py", line 219, in _req
    return self._recv(_api)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/thrift.py", line 238, in _recv
    result.read(self._iprot)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/thrift.py", line 160, in read
    iprot.read_struct(self)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 387, in read_struct
    return read_struct(self.trans, obj, self.decode_response)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 289, in read_val
    read_struct(inbuf, obj, decode_response)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 289, in read_val
    read_struct(inbuf, obj, decode_response)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 256, in read_val
    result.append(read_val(inbuf, v_type, v_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 289, in read_val
    read_struct(inbuf, obj, decode_response)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 289, in read_val
    read_struct(inbuf, obj, decode_response)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 256, in read_val
    result.append(read_val(inbuf, v_type, v_spec, decode_response))
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/protocol/binary.py", line 229, in read_val
    sz = unpack_i32(inbuf.read(4))
  File "/opt/venv/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 173, in read
    self._read_frame()
  File "/opt/venv/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 177, in _read_frame
    header = self._trans_read_all(4)
  File "/opt/venv/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 198, in _trans_read_all
    return read_all(sz)
  File "/opt/venv/lib/python3.7/site-packages/thriftpy2/transport/socket.py", line 110, in read
    buff = self.sock.recv(sz)
socket.timeout: timed out
Closing transport (tries_left=3)
Attempting to open transport (tries_left=2)
Transport opened
FetchResults: resp=TFetchResultsResp(status=TStatus(statusCode=3, infoMessages=None, sqlState='HY000', errorCode=None, errorMessage='Invalid query handle: 3e4ba0e9367430f9:924f5d8d00000000\n'), hasMoreRows=None, results=None)
Traceback (most recent call last):
  File "simple_impyla_test.py", line 24, in <module>
    f = cursor.fetchall()
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 536, in fetchall
    return list(self)
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 584, in __next__
    convert_types=self.convert_types)
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 1266, in fetch
    resp = self._rpc('FetchResults', req)
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 995, in _rpc
    err_if_rpc_not_ok(response)
  File "/opt/venv/lib/python3.7/site-packages/impala/hiveserver2.py", line 749, in err_if_rpc_not_ok
    raise HiveServer2Error(resp.status.errorMessage)
impala.error.HiveServer2Error: Invalid query handle: 3e4ba0e9367430f9:924f5d8d00000000

Closing active operation
_reset_state: Resetting cursor state
CloseOperation: req=TCloseOperationReq(operationHandle=TOperationHandle(operationId=THandleIdentifier(guid=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92', secret=b'\xf90t6\xe9\xa0K>\x00\x00\x00\x00\x8d]O\x92'), operationType=0, hasResultSet=True, modifiedRowCount=None))
Attempting to open transport (tries_left=3)
Transport opened
CloseOperation: resp=TCloseOperationResp(status=TStatus(statusCode=3, infoMessages=None, sqlState='HY000', errorCode=None, errorMessage='Invalid query handle: 3e4ba0e9367430f9:924f5d8d00000000'))
CloseSession: req=TCloseSessionReq(sessionHandle=TSessionHandle(sessionId=THandleIdentifier(guid=b'4\x9d\xb4(\x05WJ\xe1\xa6\\[\xcf\xd6/\x9c\x1b', secret=b';C\xdf\xf0\xb5\x98C~\xb5{\xe8\xb2\xd0#V|')))
Attempting to open transport (tries_left=3)
Transport opened
CloseSession: resp=TCloseSessionResp(status=TStatus(statusCode=3, infoMessages=None, sqlState='HY000', errorCode=None, errorMessage='Invalid session id: e14a570528b49d34:1b9c2fd6cf5b5ca6\n'))

The text was updated successfully, but these errors were encountered:

Rosadosa · 2021-02-08T16:27:44Z

We have the same issue but thought it was related to thriftpy2, see my comment to that package

neilgluent · 2021-03-02T12:03:22Z

I have this same issue when connecting to Impala via LDAP or Kerberos and have reproduced it both by querying rows and fetching an Impala profile. When querying rows ~30KB was enough data to cause a hang. For a profile it's taking a little more data to cause the hang.

Package versions:

$ python3 -m pip list
Package     Version
----------- -------
bitarray    1.7.0
impyla      0.16.3
pip         21.0.1
ply         3.11
sasl        0.2.1
setuptools  28.8.0
six         1.15.0
thrift      0.13.0
thrift-sasl 0.4.2
thriftpy2   0.4.14

Test case setup:

CREATE DATABASE thrift_hang_db;
CREATE TABLE thrift_hang_db.thift_hang_test
AS
SELECT 1 AS id, RPAD(CAST(RAND() AS STRING),64,'w') AS string_data
UNION ALL
SELECT 2 AS id, RPAD(CAST(RAND() AS STRING),64,'x') AS string_data
UNION ALL
SELECT 3 AS id, RPAD(CAST(RAND() AS STRING),64,'y') AS string_data
UNION ALL
SELECT 4 AS id, RPAD(CAST(RAND() AS STRING),64,'z') AS string_data;

Test script to run a query that generates an Impala profile of 55KB in size via LDAP connection:

import impala.dbapi as hs2
import logging
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
#root_logger.addHandler(logging.StreamHandler())

MYDB = 'thrift_hang_db'
MYTABLE = 'thift_hang_test'
MYHOST = 'ip-xx-yy-z-zzz'
MYUSER = 'xxx'
MYPASS = 'yyy'

def query_test_table(conn):
    cursor = conn.cursor()
    try:
        sql = f"""SELECT SUM(`id`) AS c0, MAX(LENGTH(`string_data`)) AS c1 FROM `{MYDB}`.`{MYTABLE}` WHERE id = 1
                  UNION ALL
                  SELECT SUM(`id`) AS c0, MAX(LENGTH(`string_data`)) AS c1 FROM `{MYDB}`.`{MYTABLE}` WHERE id = 2
                  UNION ALL
                  SELECT SUM(`id`) AS c0, MAX(LENGTH(`string_data`)) AS c1 FROM `{MYDB}`.`{MYTABLE}` WHERE id = 3
               """
        cursor.execute(sql)
        rows = cursor.fetchall()
        print('row count:', len(rows))
        profile = cursor.get_profile()
        print('profile size:', len(profile))
    finally:
        cursor.close()


conn = hs2.connect(host=MYHOST, port=21050, auth_mechanism='PLAIN', user=MYUSER, password=MYPASS)

try:
    query_test_table(conn)
finally:
    conn.close()

When I run the script it hangs after showing the number of rows, i.e. when fetching the profile:

$ python3 ~/testimpyla-ldap.py
row count: 3
Ctrl^C
Traceback (most recent call last):
  File "/home/oracle/testimpyla-ldap.py", line 34, in <module>
    query_test_table(conn)
  File "/home/oracle/testimpyla-ldap.py", line 25, in query_test_table
    profile = cursor.get_profile()
  File "/home/oracle/.venv/lib/python3.6/site-packages/impala/hiveserver2.py", line 609, in get_profile
    return self._last_operation.get_profile(profile_format=profile_format)
  File "/home/oracle/.venv/lib/python3.6/site-packages/impala/hiveserver2.py", line 1241, in get_profile
    resp = self._rpc('GetRuntimeProfile', req)
  File "/home/oracle/.venv/lib/python3.6/site-packages/impala/hiveserver2.py", line 993, in _rpc
    response = self._execute(func_name, request)
  File "/home/oracle/.venv/lib/python3.6/site-packages/impala/hiveserver2.py", line 1010, in _execute
    return func(request)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/thrift.py", line 219, in _req
    return self._recv(_api)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/thrift.py", line 238, in _recv
    result.read(self._iprot)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/thrift.py", line 160, in read
    iprot.read_struct(self)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/protocol/binary.py", line 387, in read_struct
    return read_struct(self.trans, obj, self.decode_response)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/protocol/binary.py", line 316, in read_struct
    read_val(inbuf, f_type, f_container_spec, decode_response))
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/protocol/binary.py", line 289, in read_val
    read_struct(inbuf, obj, decode_response)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/protocol/binary.py", line 295, in read_struct
    f_type, fid = read_field_begin(inbuf)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/protocol/binary.py", line 190, in read_field_begin
    f_type = unpack_i8(inbuf.read(1))
  File "/home/oracle/.venv/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 175, in read
    self._read_frame()
  File "/home/oracle/.venv/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 179, in _read_frame
    header = self._trans_read_all(4)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 201, in _trans_read_all
    return read_all(sz)
  File "/home/oracle/.venv/lib/python3.6/site-packages/thriftpy2/transport/socket.py", line 110, in read
    buff = self.sock.recv(sz)
KeyboardInterrupt

I have tested a couple if workarounds and both resolve the problem.

Downgrade thift_sasl==0.2.1 as mentioned on this issue
Manually re-key commit in PR 32: Make sure frames are fully buffered when used together with thriftpy … #32

I don't understand the implications of the commit but also don't like downgrading thrift_sasl from a 2020 version to a 2016 version

Maxsparrow · 2021-03-19T15:21:41Z

Bumping this - can we get an update from a contributor?

It looks like the fix in #32 works, and it would help us a lot if it was released.

Rosadosa mentioned this issue Feb 8, 2021

0.4.14 maximum query size Thriftpy/thriftpy2#162

Open

neilgluent mentioned this issue Mar 2, 2021

Frames are not fully buffered when used with thriftpy / thriftpy2 #31

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

thrift-sasl 0.4.2 invalid query handle #28

thrift-sasl 0.4.2 invalid query handle #28

Maxsparrow commented Jan 14, 2021

Rosadosa commented Feb 8, 2021

neilgluent commented Mar 2, 2021

Maxsparrow commented Mar 19, 2021

thrift-sasl 0.4.2 invalid query handle #28

thrift-sasl 0.4.2 invalid query handle #28

Comments

Maxsparrow commented Jan 14, 2021

Rosadosa commented Feb 8, 2021

neilgluent commented Mar 2, 2021

Maxsparrow commented Mar 19, 2021