Initial version of the app had bunch of issues. I used task queue lib (bullMQ), to keep every single ping as a separate entry. For any given url, I put about 8000 entry to the queue.
Surprisngly, the memory was not the issue, but the general speed, and some IO errors, which I couldn‘t track down.
I came up with some obvious solution: delegate it to the mature tool, which is at the same time famous for the speed - ffuf - written in Go and mutlithreaded.
Stay tuned, more tests will be integrated soon!