Are there any ideas for the future of RandomX, where it might be improved? Say, via new CPU hardware standards?

Or is it the expectation that the algorithm is 99% finished?

new instructions and platforms would bring ways to optimize current implementation

"new" is relative. no point in adding a new feature that less than 99% of deployed systems support.

mostly the instructions the RandomX VM uses are basic CPU instructions. everything supports them and nobody can further optimize them, CPUs have been doing them for decades.

modern CPU development has given up on optimizing them individually, and now just goes for wider parallelism. E.g. vectorizing with AVX.

scratchpad and dataset sizes can be tweaked in the future to reflect newer CPUs better

maybe program size will be increased too if CPU speed increases much faster than memory latency

AVX itself is not a suitable target for randomX, not just because not all CPUs have it, but because not all CPUs even have the same version of it when they do have it. and the data types require particular alignment in memory, which constrains how randomly you can generate them

number and ratio of main memory reads/writes can be tweaked if newer CPUs have drastically different memory interface characteristics

they already do

RandomX was initially tweaked for Zen1, and Zen4 is 50% faster per thread (CPU-only tasks)

but memory latency is the same compared to Zen1