크롤링 서버 403 에러

Question

안녕하세요!크롤링 서버를 구축하던중 로컬환경에서는 정상적으로 작동되던 작업이 배포환경에서는 403 에러가 발생하는 문제를 해결하지 못해서 질문하게 되었습니다.바쁘시겠지만 한번 살펴보시고 피드백 부탁드리겠습니다 ㅜㅜ며칠동안 해결을 못해서 너무 답답하네요 ㅜㅜ환경GKE 사용Java, Spring boot 3.0.7seleniumError[http-nio-8084-exec-1] [2023-08-05 23:40:28,741] [ERROR] [SolvedCrawling.java:136] - ERROR: The request could not be satisfied

403 ERROR

The request could not be satisfied.

Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront) Request ID: -ok_4ED_rixCpeCLsp6ytvEtEjyMwZSMUb2VxVa10USNMDizAGzXbg==

CodeRestTemplate 설정@Bean public RestTemplate restTemplate() { RestTemplate restTemplate; try { restTemplate = new RestTemplate(clientHttpRequestFactory()); }catch (Exception e){ restTemplate = new RestTemplate(); } restTemplate.setInterceptors(Collections.singletonList( (request, body, execution) -> { HttpHeaders headers = request.getHeaders(); headers.setContentType(APPLICATION_JSON); headers.setAccept(Collections.singletonList(APPLICATION_JSON)); headers.add(HttpHeaders.ACCEPT, "application/json"); headers.add(HttpHeaders.USER_AGENT, "Mozilla/5.0"); return execution.execute(request, body); } )); return restTemplate; } private HttpComponentsClientHttpRequestFactory clientHttpRequestFactory() { return new HttpComponentsClientHttpRequestFactory(); } API 요청public String getSubject(int problemId) throws Exception{ String jsonString = null; try { jsonString = restTemplate.getForObject("" + problemId, String.class); }catch (Exception e){ e.printStackTrace(); throw new HttpResponseException("문제 정보를 가져오는데 실패했습니다."); } JSONParser jsonParser = new JSONParser(); Object jsonObject = null; try { jsonObject = jsonParser.parse(jsonString); } catch (ParseException e) { e.printStackTrace(); } JSONObject jsonBody = (JSONObject) jsonObject; return jsonBody.get("titleKo").toString(); } 크롤링public BaekJoonDto profileCrawling(String baekjoonId) throws IOException, InterruptedException { WebDriver driver = setDriver(); sleep(1000); driver.get(SOLVED_BASE_URL + SOLVED_PROFILE + baekjoonId); By solvedListBy = By.xpath("//*[@id=\"__next\"]/div[3]/div/div[6]/div[3]/div/table/tbody"); sleep(1000); try { wait(driver, solvedListBy); } catch (TimeoutException | NoSuchElementException e) { log.error("{}", driver.getCurrentUrl()); log.error("{}", driver.getPageSource()); throw new CrawlingException("사용자 정보를 찾을 수 없습니다. 다시 시도해 주세요"); } WebElement elements = driver.findElement(solvedListBy); WebElement webElement = elements.findElement(By.className("css-1ojb0xa")); int bronze = getUserSolvedCount(webElement, By.xpath("//*[@id=\"__next\"]/div[3]/div/div[6]/div[3]/div/table/tbody/tr[1]/td[2]/b")); driver.quit(); return new BaekJoonDto(bronze); driver settingprivate WebDriver setDriver() throws IOException, InterruptedException { String os = System.getProperty("os.name").toLowerCase(); if (os.contains("win")) { System.setProperty("webdriver.chrome.driver", "drivers/chromedriver_win.exe"); } else if (os.contains("mac")) { Process process = Runtime.getRuntime().exec("xattr -d com.apple.quarantine drivers/chromedriver_mac"); process.waitFor(); System.setProperty("webdriver.chrome.driver", "drivers/chromedriver_mac"); } else if (os.contains("linux")) { System.setProperty("webdriver.chrome.driver", "/usr/bin/chromedriver-linux64/chromedriver"); } ChromeOptions chromeOptions = new ChromeOptions(); chromeOptions.addArguments("--disk-cache-size=0"); chromeOptions.addArguments("--media-cache-size=0"); chromeOptions.addArguments("--headless=new"); chromeOptions.setHeadless(true); chromeOptions.addArguments("--no-sandbox"); chromeOptions.addArguments("--disable-dev-shm-usage"); chromeOptions.addArguments("--disable-gpu"); chromeOptions.addArguments("--remote-allow-origins=*"); chromeOptions.addArguments("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537"); // binary 는 확인해야함 local 에서만 // chromeOptions.setBinary("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"); local //. 배포환경 binary chromeOptions.setBinary("/usr/bin/google-chrome"); return new ChromeDriver(chromeOptions); }

kotliner · Answer

403이면 보통 크롤링 대상 서버에서 ip나 모종의 조건으로 제한을 거는 경우입니다. 해당 사이트와 정상적으로 계약을 맺고 데이터를 받아가시거나 혹은 별개의 ip를 제공하는 프록시 업체를 이용하셔야합니다.

크롤링 서버 403 에러

1개의 답변

Q&A 목록