Avoiding duplicates when retrieving data from the InsecureWeb API

When using the InsecureWeb API to obtain security breach data and executing the API call multiple times, data may be duplicated in the target database.

To avoid this, we recommend two strategies:

1. Use UPSERT.

One effective way is to use MySQL's UPSERT function, which allows you to update or insert data depending on whether it already exists or not. This automatically avoids duplicates.

See UPSERT documentation: https://www.techbeamers.com/mysql-upsert/

2. Filter by dates

Another option is to apply a date filter in the API call, for example to get only breaches from the last 24 hours. This way, assuming that the process is executed daily, duplicate results are avoided.