Avoiding duplicates when retrieving data from the InsecureWeb API

Created by Juan Manuel Libera Frómeta, Modified on Fri, 18 Aug 2023 at 07:45 PM by Juan Manuel Libera Frómeta

When using the InsecureWeb API to obtain security breach data and executing the API call multiple times, data may be duplicated in the target database.

To avoid this, we recommend two strategies:

1. Use UPSERT.

One effective way is to use MySQL's UPSERT function, which allows you to update or insert data depending on whether it already exists or not. This automatically avoids duplicates.

See UPSERT documentation: https://www.techbeamers.com/mysql-upsert/

2. Filter by dates

Another option is to apply a date filter in the API call, for example to get only breaches from the last 24 hours. This way, assuming that the process is executed daily, duplicate results are avoided.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article