The GitHub Actions build itself takes about ~15 seconds to run. It is generally expected for your site to update around that duration, and take no more than a minute.
Which environment is used to build and deploy the site?
The Actions workflow file publish.yml is configured to pull the latest docker image; you can adjust it to pull from a (unchanging) tag instead. See here for a list of available tags which correspond to neuron versions; generally you would pick the latest tag and put it in publish.yml. Then, your site will continue use the same neuron version, regardless of upstream development changes.