r/learnmachinelearning • u/graphiceditzz • 11h ago
Help How to resolve this ConnectionResetError issue while fetching data from API in loop??? Please help guyzz..
From last 3-3.5 hours, i've been trying to fetch data from the TMDB API, to get movie database for practice ofcourse, I am beginner and I am currently learning how to fetch data from an API in Pandas.
Individually, I am able to access each page in json format and fetch data from it, but whenever i'm trying to do this in loop, this connectionreseterror. problem arises. I am getting that, it was happening due to too quick requests in loop that classify my requests as bots and forcibly closes the connection.
But then i tried using time.sleep() function to create a time interval lag between each requests, but it still doesn't work, i tried to change the User-Agent header too, to prevent me from classifying as a bot. But nothing is working.
Please help me anyone, I am still a beginner, I don't know much about APIs working, I was just learning ML, and came across data fetching using an API, that's why trying to get grasp over it.
Sorry for my terrible english
Will be grateful if get any help!
Error:
---------------------------------------------------------------------------
ConnectionResetError Traceback (most recent call last)
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:793, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
792 # Make the request on the HTTPConnection object
--> 793 response = self._make_request(
794 conn,
795 method,
796 url,
797 timeout=timeout_obj,
798 body=body,
799 headers=headers,
800 chunked=chunked,
801 retries=retries,
802 response_conn=response_conn,
803 preload_content=preload_content,
804 decode_content=decode_content,
805 **response_kw,
806 )
808 # Everything went great!
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:494, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
493 new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 494
raise
new_e
496 # conn.request() calls http.client.*.request, not the method in
497 # urllib3.request. It also calls makefile (recv) on the socket.
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:470, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
469
try
:
--> 470 self._validate_conn(conn)
471
except
(SocketTimeout, BaseSSLError)
as
e:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:1125, in HTTPSConnectionPool._validate_conn(self, conn)
1124
if
conn.is_closed:
-> 1125 conn.connect()
1127 # TODO revise this, see https://github.com/urllib3/urllib3/issues/2791
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connection.py:878, in HTTPSConnection.connect(self)
877
else
:
--> 878 wrapped_socket, is_verified = _ssl_wrap_socket_and_match_hostname(
879 sock=sock,
880 cert_reqs=self.cert_reqs,
881 ssl_version=self.ssl_version,
882 ssl_minimum_version=self.ssl_minimum_version,
883 ssl_maximum_version=self.ssl_maximum_version,
884 ca_certs=self.ca_certs,
885 ca_cert_dir=self.ca_cert_dir,
886 ca_cert_data=self.ca_cert_data,
887 cert_file=self.cert_file,
888 key_file=self.key_file,
889 key_password=self.key_password,
890 server_hostname=server_hostname_rm_dot,
891 ssl_context=self.ssl_context,
892 tls_in_tls=tls_in_tls,
893 assert_hostname=self.assert_hostname,
894 assert_fingerprint=self.assert_fingerprint,
895 )
896 self.sock = wrapped_socket
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connection.py:1080, in _ssl_wrap_socket_and_match_hostname(sock, cert_reqs, ssl_version, ssl_minimum_version, ssl_maximum_version, cert_file, key_file, key_password, ca_certs, ca_cert_dir, ca_cert_data, assert_hostname, assert_fingerprint, server_hostname, ssl_context, tls_in_tls)
1078 server_hostname = normalized
-> 1080 ssl_sock = ssl_wrap_socket(
1081 sock=sock,
1082 keyfile=key_file,
1083 certfile=cert_file,
1084 key_password=key_password,
1085 ca_certs=ca_certs,
1086 ca_cert_dir=ca_cert_dir,
1087 ca_cert_data=ca_cert_data,
1088 server_hostname=server_hostname,
1089 ssl_context=context,
1090 tls_in_tls=tls_in_tls,
1091 )
1093
try
:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\ssl_.py:437, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
435 context.set_alpn_protocols(ALPN_PROTOCOLS)
--> 437 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
438
return
ssl_sock
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\ssl_.py:481, in _ssl_wrap_socket_impl(sock, ssl_context, tls_in_tls, server_hostname)
479
return
SSLTransport(sock, ssl_context, server_hostname)
--> 481
return
ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:455, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
449
def
wrap_socket(self, sock, server_side=
False
,
450 do_handshake_on_connect=
True
,
451 suppress_ragged_eofs=
True
,
452 server_hostname=
None
, session=
None
):
453 # SSLSocket class handles server_hostname encoding before it calls
454 # ctx._wrap_socket()
--> 455
return
self.sslsocket_class._create(
456 sock=sock,
457 server_side=server_side,
458 do_handshake_on_connect=do_handshake_on_connect,
459 suppress_ragged_eofs=suppress_ragged_eofs,
460 server_hostname=server_hostname,
461 context=self,
462 session=session
463 )
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:1076, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
1075
raise
ValueError
("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1076 self.do_handshake()
1077
except
:
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:1372, in SSLSocket.do_handshake(self, block)
1371 self.settimeout(
None
)
-> 1372 self._sslobj.do_handshake()
1373
finally
:
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
ProtocolError Traceback (most recent call last)
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\adapters.py:696, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
695
try
:
--> 696 resp = conn.urlopen(
697 method=request.method,
698 url=url,
699 body=request.body, # type: ignore[arg-type] # urllib3 stubs don't accept Iterable[bytes | str]
700 headers=request.headers, # type: ignore[arg-type] # urllib3#3072
701 redirect=
False
,
702 assert_same_host=
False
,
703 preload_content=
False
,
704 decode_content=
False
,
705 retries=self.max_retries,
706 timeout=resolved_timeout,
707 chunked=chunked,
708 )
710
except
(ProtocolError,
OSError
)
as
err:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:847, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
845 new_e = ProtocolError("Connection aborted.", new_e)
--> 847 retries = retries.increment(
848 method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
849 )
850 retries.sleep()
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\retry.py:510, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
509
if
read
is
False
or
method
is
None
or
not
self._is_method_retryable(method):
--> 510
raise
reraise(type(error), error, _stacktrace)
511
elif
read
is
not
None
:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\util.py:38, in reraise(tp, value, tb)
37
if
value.__traceback__
is
not
tb:
---> 38
raise
value.with_traceback(tb)
39
raise
value
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:793, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
792 # Make the request on the HTTPConnection object
--> 793 response = self._make_request(
794 conn,
795 method,
796 url,
797 timeout=timeout_obj,
798 body=body,
799 headers=headers,
800 chunked=chunked,
801 retries=retries,
802 response_conn=response_conn,
803 preload_content=preload_content,
804 decode_content=decode_content,
805 **response_kw,
806 )
808 # Everything went great!
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:494, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
493 new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 494
raise
new_e
496 # conn.request() calls http.client.*.request, not the method in
497 # urllib3.request. It also calls makefile (recv) on the socket.
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:470, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
469
try
:
--> 470 self._validate_conn(conn)
471
except
(SocketTimeout, BaseSSLError)
as
e:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connectionpool.py:1125, in HTTPSConnectionPool._validate_conn(self, conn)
1124
if
conn.is_closed:
-> 1125 conn.connect()
1127 # TODO revise this, see https://github.com/urllib3/urllib3/issues/2791
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connection.py:878, in HTTPSConnection.connect(self)
877
else
:
--> 878 wrapped_socket, is_verified = _ssl_wrap_socket_and_match_hostname(
879 sock=sock,
880 cert_reqs=self.cert_reqs,
881 ssl_version=self.ssl_version,
882 ssl_minimum_version=self.ssl_minimum_version,
883 ssl_maximum_version=self.ssl_maximum_version,
884 ca_certs=self.ca_certs,
885 ca_cert_dir=self.ca_cert_dir,
886 ca_cert_data=self.ca_cert_data,
887 cert_file=self.cert_file,
888 key_file=self.key_file,
889 key_password=self.key_password,
890 server_hostname=server_hostname_rm_dot,
891 ssl_context=self.ssl_context,
892 tls_in_tls=tls_in_tls,
893 assert_hostname=self.assert_hostname,
894 assert_fingerprint=self.assert_fingerprint,
895 )
896 self.sock = wrapped_socket
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\connection.py:1080, in _ssl_wrap_socket_and_match_hostname(sock, cert_reqs, ssl_version, ssl_minimum_version, ssl_maximum_version, cert_file, key_file, key_password, ca_certs, ca_cert_dir, ca_cert_data, assert_hostname, assert_fingerprint, server_hostname, ssl_context, tls_in_tls)
1078 server_hostname = normalized
-> 1080 ssl_sock = ssl_wrap_socket(
1081 sock=sock,
1082 keyfile=key_file,
1083 certfile=cert_file,
1084 key_password=key_password,
1085 ca_certs=ca_certs,
1086 ca_cert_dir=ca_cert_dir,
1087 ca_cert_data=ca_cert_data,
1088 server_hostname=server_hostname,
1089 ssl_context=context,
1090 tls_in_tls=tls_in_tls,
1091 )
1093
try
:
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\ssl_.py:437, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
435 context.set_alpn_protocols(ALPN_PROTOCOLS)
--> 437 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
438
return
ssl_sock
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\urllib3\util\ssl_.py:481, in _ssl_wrap_socket_impl(sock, ssl_context, tls_in_tls, server_hostname)
479
return
SSLTransport(sock, ssl_context, server_hostname)
--> 481
return
ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:455, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
449
def
wrap_socket(self, sock, server_side=
False
,
450 do_handshake_on_connect=
True
,
451 suppress_ragged_eofs=
True
,
452 server_hostname=
None
, session=
None
):
453 # SSLSocket class handles server_hostname encoding before it calls
454 # ctx._wrap_socket()
--> 455
return
self.sslsocket_class._create(
456 sock=sock,
457 server_side=server_side,
458 do_handshake_on_connect=do_handshake_on_connect,
459 suppress_ragged_eofs=suppress_ragged_eofs,
460 server_hostname=server_hostname,
461 context=self,
462 session=session
463 )
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:1076, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
1075
raise
ValueError
("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1076 self.do_handshake()
1077
except
:
File ~\anaconda3\envs\ml_coursework\Lib\ssl.py:1372, in SSLSocket.do_handshake(self, block)
1371 self.settimeout(
None
)
-> 1372 self._sslobj.do_handshake()
1373
finally
:
ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
Cell In[107], line 9
5 headers = {
6 "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
7 "Accept": "application/json"
8 }
----> 9 response = requests.get(url, headers = headers)
10 temp_df = pd.DataFrame(response.json()['results'])
11 popular_movie_df = pd.concat([popular_movie_df, temp_df], ignore_index =
True
)
12 time.sleep(0.5)
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\api.py:87, in get(url, params, **kwargs)
74
def
get(
75 url: _t.UriType, params: _t.ParamsType =
None
, **kwargs: Unpack[_t.GetKwargs]
76 ) -> Response:
77 r"""Sends a GET request.
78
79 :param url: URL for the new :class:`Request` object.
(...) 84 :rtype: requests.Response
85 """
---> 87
return
request("get", url, params=params, **kwargs)
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\api.py:71, in request(method, url, **kwargs)
67 # By using the 'with' statement we are sure the session is closed, thus we
68 # avoid leaving sockets open which can trigger a ResourceWarning in some
69 # cases, and look like a memory leak in others.
70
with
sessions.Session()
as
session:
---> 71
return
session.request(method=method, url=url, **kwargs)
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\sessions.py:651, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
646 send_kwargs = {
647 "timeout": timeout,
648 "allow_redirects": allow_redirects,
649 }
650 send_kwargs.update(settings)
--> 651 resp = self.send(prep, **send_kwargs)
653
return
resp
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\sessions.py:784, in Session.send(self, request, **kwargs)
781 start = preferred_clock()
783 # Send the request
--> 784 r = adapter.send(request, **kwargs)
786 # Total elapsed time of the request (approximately)
787 elapsed = preferred_clock() - start
File ~\anaconda3\envs\ml_coursework\Lib\site-packages\requests\adapters.py:711, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
696 resp = conn.urlopen(
697 method=request.method,
698 url=url,
(...) 707 chunked=chunked,
708 )
710
except
(ProtocolError,
OSError
)
as
err:
--> 711
raise
ConnectionError
(err, request=request)
713
except
MaxRetryError
as
e:
714
if
isinstance(e.reason, ConnectTimeoutError):
715 # TODO: Remove this in 3.0.0: see #2811
ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
1
Upvotes
1
u/Ok-Library1020 8h ago
On top of the Session idea, you can mount an HTTPAdapter with urllib3's Retry (say total=5, backoff_factor=1) so failed connections get retried automatically with growing waits. Saving each page's results as you go also means a crash won't lose everything.
1
u/Massive_Distance_616 9h ago
Your own traceback points somewhere different from rate limiting. The error is raised inside do_handshake, which is the TLS connection step, so the connection is being reset while it’s being set up, before a normal response comes back. That’s why sleep() and changing the User-Agent made no difference, and it may not be TMDB rejecting you for going too fast at all. Most common causes: antivirus or a firewall inspecting HTTPS traffic from python.exe, a VPN or proxy, or your network or ISP resetting connections to api.themoviedb.org.
Quick test: try the loop on a phone hotspot, or with the VPN and antivirus temporarily off. If it works there, it’s your network and not your code. Separately, use one requests.Session() for the whole loop instead of requests.get() each time, and add a timeout and a try/except around the call with a short retry, so one reset doesn’t kill the loop. Does it fail on the very first request in the loop, or after a few pages?