[2023-10-24 14:25:17,413: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:25:47,993: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:26:18,564: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:26:49,118: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:27:19,722: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:27:50,714: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py:686: RuntimeWarning: Exception raised outside body: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.'): Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook raise RuntimeError('Maximum retries occured') RuntimeError: Maximum retries occured During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") mysql.connector.errors.OperationalError: MySQL Connection not available. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 494, in trace_task I, R, state, retval = on_error(task_request, exc) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 399, in on_error R = I.handle_error_state( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 178, in handle_error_state return { File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 230, in handle_failure task.backend.mark_as_failure( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure self.store_result(task_id, exc, state, File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 22, in store_result raise e File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 16, in store_result api_account = session.query(connection.ApiAccount).filter_by(api_key=api_key).first() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2743, in first return self.limit(1)._iter().first() # type: ignore File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2842, in _iter result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2262, in execute return self._execute_internal( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement result = conn.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ret = self._execute_context( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1816, in _execute_context self._handle_dbapi_exception( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available. [SQL: SELECT api_account.id AS api_account_id, api_account.api_key AS api_account_api_key FROM api_account WHERE api_account.api_key = %(api_key_1)s LIMIT %(param_1)s] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/20/e3q8) warn(RuntimeWarning( [2023-10-24 14:27:50,871: ERROR/ForkPoolWorker-2] Task scheduler_infomin_solutions.app.trigger_webhook[f1a35ebe-d496-4660-83e7-42cb8a31eb81] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.') Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook raise RuntimeError('Maximum retries occured') RuntimeError: Maximum retries occured During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") mysql.connector.errors.OperationalError: MySQL Connection not available. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 494, in trace_task I, R, state, retval = on_error(task_request, exc) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 399, in on_error R = I.handle_error_state( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 178, in handle_error_state return { File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 230, in handle_failure task.backend.mark_as_failure( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure self.store_result(task_id, exc, state, File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 22, in store_result raise e File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 16, in store_result api_account = session.query(connection.ApiAccount).filter_by(api_key=api_key).first() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2743, in first return self.limit(1)._iter().first() # type: ignore File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2842, in _iter result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2262, in execute return self._execute_internal( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement result = conn.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ret = self._execute_context( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1816, in _execute_context self._handle_dbapi_exception( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available. [SQL: SELECT api_account.id AS api_account_id, api_account.api_key AS api_account_api_key FROM api_account WHERE api_account.api_key = %(api_key_1)s LIMIT %(param_1)s] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/20/e3q8) [2023-10-24 14:27:50,953: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:28:21,522: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:28:52,144: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:29:22,761: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:29:53,301: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '15c2-220-158-159-149.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-10-24 14:30:23,845: ERROR/ForkPoolWorker-2] Task scheduler_infomin_solutions.app.trigger_webhook[2bce8150-e268-47a6-848e-326c41229bf3] raised unexpected: RuntimeError('Maximum retries occured') Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook raise RuntimeError('Maximum retries occured') RuntimeError: Maximum retries occured [2023-12-25 15:21:09,582: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '4f92-2409-40f4-100b-4cde-5901-2ecb-7b73-6f4d.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-12-25 15:21:39,827: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '4f92-2409-40f4-100b-4cde-5901-2ecb-7b73-6f4d.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-12-25 15:22:09,962: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '4f92-2409-40f4-100b-4cde-5901-2ecb-7b73-6f4d.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-12-25 15:22:40,102: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '4f92-2409-40f4-100b-4cde-5901-2ecb-7b73-6f4d.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-12-25 15:23:10,279: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host '4f92-2409-40f4-100b-4cde-5901-2ecb-7b73-6f4d.ngrok-free.app'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2023-12-25 15:23:41,025: ERROR/ForkPoolWorker-2] Task scheduler_infomin_solutions.app.trigger_webhook[e4986ca3-d3e1-47f4-8703-16e0cedbfcb2] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.') Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook raise RuntimeError('Maximum retries occured') RuntimeError: Maximum retries occured During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") mysql.connector.errors.OperationalError: MySQL Connection not available. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 494, in trace_task I, R, state, retval = on_error(task_request, exc) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 399, in on_error R = I.handle_error_state( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 178, in handle_error_state return { File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 230, in handle_failure task.backend.mark_as_failure( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure self.store_result(task_id, exc, state, File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 22, in store_result raise e File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 16, in store_result api_account = session.query(connection.ApiAccount).filter_by(api_key=api_key).first() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2743, in first return self.limit(1)._iter().first() # type: ignore File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2842, in _iter result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2262, in execute return self._execute_internal( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement result = conn.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ret = self._execute_context( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1816, in _execute_context self._handle_dbapi_exception( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available. [SQL: SELECT api_account.id AS api_account_id, api_account.api_key AS api_account_api_key FROM api_account WHERE api_account.api_key = %(api_key_1)s LIMIT %(param_1)s] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/20/e3q8) [2024-06-23 02:51:17,201: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy.infomin.solutions'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2024-06-23 02:51:49,273: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy.infomin.solutions'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2024-06-23 02:52:21,401: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy.infomin.solutions'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2024-06-23 02:52:53,401: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy.infomin.solutions'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2024-06-23 02:53:25,477: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy.infomin.solutions'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [2024-06-23 02:53:57,811: WARNING/ForkPoolWorker-2] /home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py:686: RuntimeWarning: Exception raised outside body: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.'): Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook except: RuntimeError: Maximum retries occured During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") mysql.connector.errors.OperationalError: MySQL Connection not available. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 494, in trace_task I, R, state, retval = on_error(task_request, exc) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 399, in on_error R = I.handle_error_state( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 178, in handle_error_state return { File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 230, in handle_failure task.backend.mark_as_failure( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure self.store_result(task_id, exc, state, File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 22, in store_result raise e File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 16, in store_result api_account = session.query(connection.ApiAccount).filter_by(api_key=api_key).first() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2743, in first return self.limit(1)._iter().first() # type: ignore File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2842, in _iter result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2262, in execute return self._execute_internal( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement result = conn.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ret = self._execute_context( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1816, in _execute_context self._handle_dbapi_exception( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available. [SQL: SELECT api_account.id AS api_account_id, api_account.api_key AS api_account_api_key FROM api_account WHERE api_account.api_key = %(api_key_1)s LIMIT %(param_1)s] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/20/e3q8) warn(RuntimeWarning( [2024-06-23 02:53:57,953: ERROR/ForkPoolWorker-2] Task scheduler_infomin_solutions.app.trigger_webhook[f9fbf8b4-6938-402c-983a-d99e46af9c59] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.') Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/app.py", line 51, in trigger_webhook except: RuntimeError: Maximum retries occured During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") mysql.connector.errors.OperationalError: MySQL Connection not available. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 494, in trace_task I, R, state, retval = on_error(task_request, exc) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 399, in on_error R = I.handle_error_state( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 178, in handle_error_state return { File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/app/trace.py", line 230, in handle_failure task.backend.mark_as_failure( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure self.store_result(task_id, exc, state, File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 22, in store_result raise e File "/home/infominsolutions/scheduler_infomin_solutions/backend.py", line 16, in store_result api_account = session.query(connection.ApiAccount).filter_by(api_key=api_key).first() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2743, in first return self.limit(1)._iter().first() # type: ignore File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2842, in _iter result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2262, in execute return self._execute_internal( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement result = conn.execute( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ret = self._execute_context( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1816, in _execute_context self._handle_dbapi_exception( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1810, in _execute_context context = constructor( File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1398, in _init_compiled self.cursor = self.create_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1722, in create_cursor return self.create_default_cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1728, in create_default_cursor return self._dbapi_connection.cursor() File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1489, in cursor return self.dbapi_connection.cursor(*args, **kwargs) File "/home/infominsolutions/scheduler_infomin_solutions/venv/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 700, in cursor raise OperationalError("MySQL Connection not available.") sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available. [SQL: SELECT api_account.id AS api_account_id, api_account.api_key AS api_account_api_key FROM api_account WHERE api_account.api_key = %(api_key_1)s LIMIT %(param_1)s] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/20/e3q8)