> For the complete documentation index, see [llms.txt](https://docs.predicthq.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.predicthq.com/getting-started/core-concepts/guide-to-protecting-predicthq-data.md). # Guide to protecting PredictHQ data It's important to protect PredictHQ data from unauthorized access and usage. This guide provides ideas on how to protect PredictHQ’s data from unauthorized usage due to what is known as web scraping or screen scraping, specifically when being used in public facing websites. Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites, which automated tools or bots typically perform. Our terms require customers to protect against unauthorized use of our data including by these techniques so carefully consider how you're exposing PredictHQ data and what protections you have in-place to protect it. ## Technical deterrents and protection You must use any reasonable endeavors to prevent unauthorized access to, or use of PredictHQ Data and, in the event of any such unauthorized access or use, promptly notify PredictHQ. Reasonable endeavors to prevent web scraping may include any of the following (**but are not limited to**): ### Require authentication to view data Users must log in, or be approved before using the application. This may help detect legitimate users from automated scripts. It also allows more effective monitoring for (and the ability to take action against) any unwanted activity. ### Application design Effective application design can make it difficult to scrape information or easier to detect scraping. This includes techniques such as limiting the amount of data returned per search, restricting the area of the search, or requiring pagination of results. ### IP address monitoring, limiting, or blocking Ability to monitor, alert, and report on website activity by IP address allows for the detection of sudden increases in traffic, outliers, or bad actors. Tracking of an IP address allows for the ability to use a blocklist if needed. ### Use of commercial software to protect public-facing content from bots Many companies offer fully featured solutions to prevent scraping by bots or automated scripts. The major cloud providers (AWS, Azure, and GCP) offer Web Application Firewall (WAF) solutions. In addition to this there are several stand alone solutions, for example Cloudflare or Fastly. These solutions all provide services that include regularly updated blocklists, automatic bot detection, and bot prevention. ### Traffic monitoring Monitoring of website traffic through capture and analysis of access logs or similar allows for trend monitoring, the configuration of alerts, and early detection of suspicious activity such as increased traffic volumes. ### Use of captcha challenge-response tests, in particular reCaptcha Captcha or reCaptcha solutions that attempt to identify legitimate human users can help prevent bot usage. Together with IP tracking, the use of a captcha can be triggered only after certain thresholds are met, or for repeated infringements. ### Correct use of the Robots Exclusion Standards (robots.txt file) Websites can declare if crawling is allowed or not in the robots.txt file and allow partial access, limit the crawl rate, specify the optimal time to crawl, and more. This can be used to prevent web crawlers from scraping data ### Protect or disable any publicly available APIs Ensure you implement proper security and access controls for any publicly accessible APIs that your website uses, so that only legitimate usage has access. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.predicthq.com/getting-started/core-concepts/guide-to-protecting-predicthq-data.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.