-
Rucknium[m]
I find a lot of the statistical work in Moser et al. (2018) to be extremely hand-wavy. What I read is "we plotted the distributions and they looked similar to the naked eye, so they are probably pretty similar" 🧐
-
coinstudent2048[
Rucknium: I have very poor understanding of stats right now, hence I have stupid questions: Is there process/es of 'fitting' 1) discrete and 2) continuous prob distribution to let's say a histogram? I just need topic name and some resources to study. I kind of need it not cryptocurrency related. Thanks!
-
Rucknium[m]
coinstudent2048: Yes. This is called estimation of a probability density function (PDF) if the values are continuous and probability mass function (PMF) if the values are discrete. The most widely-used way to do this is through Maximum Likelihood Estimation (MLE). You may run into MLE as applied to "regression", but that is different. The idea is to guess the type of probability distribution, e.g. normal, uniform, exponential,
-
Rucknium[m]
gamma, Poisson, etc., and then determine the parameters for that type of distribution that would best fit the data. So for normal distribution, MLE would find the best mean and variance of a normal distribution that best fits the data. The advantage of MLE is that if the distribution of your data is truly the chosen family, e.g. if you choose to use a Poisson distribution for MLE and the actual underlying data is actually
-
Rucknium[m]
Poisson, then your estimate will have minimum variance, i.e. your approximation error will typically reach the Cramér-Rao lower bound. The bad news is if you choose incorrectly, e.g. choose normal distribution when the data is actually uniform, then your MLE estimates could be very inaccurate. You won't know you've chosen wrongly unless you do further tests. MLE is in a class of estimators known as parametric estimators since
-
Rucknium[m]
at the end of the estimation procedure you get a small set of parameters, i.e. numbers, to look at and work with. For a normal distribution, again, the parameters would be mean and variance....
-
Rucknium[m]
There is another class of PDF estimators called nonparametric estimators. Kernel density estimation is one of the most popular nonparametric estimators for PDFs. It is sort of like a more sophisticated histogram, but rather than having discrete bins as in a histogram, instead you have a rolling smoothing function. The advantage of kernel density estimation is that you don't have to guess a particular probability family, so you
-
Rucknium[m]
cannot accidentally guess wrong and be way off the mark as in MLE. Some disadvantages: Since it is nonparametric, you do not get a few numbers to work with at the end of the estimation. Basically, you get a graph to look at. This makes the result of the estimation not very interpretable at times. Also, you have a choice of k, which is basically the bin width. Most software has sensible defaults. However, changing k has a
-
Rucknium[m]
bias-variance tradeoff. Finally, to get good estimates for kernel density estimation, you need a lot more observations than for MLE.
-
Rucknium[m]
Does that get you started?
-
Rucknium[m]
By observations I mean sample size
-
coinstudent2048[
Yes! I already heard of some of those keywords, and there are some that are new to me. I copied your answers in a text file. Many thanks!
-
merope
Rucknium: if I may pick your brain as well: is there a variant/analogue of Principal Component Analysis that works on binary data? Basically I'm looking at block nonces, and I'm looking for any patterns/correlations between each individual bit
-
Rucknium[m]
endor00: PCA is almost never used in economics, so I am not familiar with a PCA method that works on binary data. There could be another class of techniques that I am familiar with that could accomplish what you are trying to do.
-
Rucknium[m]
Maybe any of the various clustering techniques. Or just calculate a correlation matrix to start?
-
nioc
fyi, I cannot see any comment from endor00 here on IRC, may need to be voiced
-
coinstudent2048[
endor00 doesn't see Rucknium's answers yet...
-
nioc
yes matrix person :)
-
nioc
here on IRC endor00 does not exist
-
nioc
I assume that he would need to be voiced
-
selsta
nioc: no more voice required here
-
nioc
thx
-
nioc
haven't seen endor for weeks
-
selsta
weird
-
merope
Maybe it's because I'm merope on irc
-
merope
Not sure how it shows up with the different nickname
-
nioc
oh, I should have looked closer
-
merope
<Rucknium[m]> "Maybe any of the various cluster" <- I'll look into it, thanks!
-
nioc
my bad
-
nioc
merope> @endor00:matrix.org
-
nioc
^^ what I see
-
-
Rucknium[m]
endor00: I would be happy to take a look at results to aid interpretation.
-
merope
Sorry for the confusion - I registered the irc account before I knew about the possibility to link them, so I ended up with two different usernames
-
merope
Rucknium I'm out atm, I'll send some pics here (or in your DM if it's too much spam here) when I'm in front of the computer again
-
merope
Oh, actually - a slightly outdated pic
imgur.com/a/ZzveBeY
-
merope
I wrote some c++ to (attempt to) do some apriori analysis and try to extract some nonce patterns
-
merope
I'm getting some results, but I'm kinda struggling with their interpretation (or wether they're even correct and meaningful at all)
-
Rucknium[m]
So the 20th-24th bits are much more likely to be zero?
-
merope
So it would seem
-
merope
But the question is: are they zero because there's an actual bias in the algo, or just because nobody ever picks them?
-
selsta
sech1: ^^
-
merope
But figuring out the causality is quite the complicated mess
-
merope
There are a whole bunch of things that I wanna try throwing at the data to see what sticks, it's just a matter of finding the time to learn how they work
-
Rucknium[m]
Wouldn't naive mining software just count the nonce upward, incrementally +1, so that it is less likely that the software will ever need to reach those later bits to increment upward, since they will have already found a block by then?
-
merope
That's the basic assumption
-
merope
But then there are bits 25-32
-
merope
Which are typically due to mining proxies and the way they split the work among their clients
-
merope
But again it brings the question of intrinsic biases and causality
-
Rucknium[m]
I guess one use of the data could be to try to sniff out an unknown miners who are using custom nonce-choosing algorithms.
-
merope
Also, there's the weird dip around bits 11-15 which I have no idea how to explain, and does not fit the model of naively incrementing from 0
-
merope
Yeah, that would be one application
-
merope
What I would really like to have is a metric ton of data about the valid shares submitted to the pools (block template, target diff, nonce, resulting hash)
-
merope
Because atm I can only look at the winning block nonces on the main chain, but it would be nice to use the target diff as a parameter and see if it has any influence
-
Rucknium[m]
What is the ultimate research question? Or is it just mostly exploratory for now?
-
merope
Partly exploratory fun, partly looking for any hidden biases in RandomX's input
-
merope
(And using the broad research question as a starting point to learn data analysis)
-
merope
Either I find something, which would have an actual impact on RandomX and the mining landscape, or I find nothing, and RandomX passes another test and can be considered that much "safer" - and in both cases I've learned stuff along the way
-
merope
Plus, the same methodology could be applied to any other coin with any other mining algo
-
sech1
merope bits 24-31 are used by Nicehash and xmrig-proxy to split work
-
sech1
25-32 in your notation
-
sech1
xmrig checks numbers in increasing order until it finds a solution, so bits 21-24 are almost always 0: solution is found before it gets to such large numbers
-
sech1
bump at bit 16 (actually bit 15 if count them properly) on your graph is because xmrig splits search space in chunks of the size 2^15 (32768).
-
sech1
each thread increments nonce by 1 until it finds a solution, and each thread starts from a multiple of 32768
-
sech1
from what I can see, per-bit frequencies match what xmrig should produce
-
merope
Yep, I've tried playing with xmrig's nonce selection too (though it's hard to get any statistically significant results when all you have is 1.6 kH/s)
-
merope
Bit 15 isn't actually a bump though, but rather it goes (almost) back to 50%
-
merope
(Look at the overall bit frequencies in the third plot in my link)
-
sech1
50% means it's changed a lot, so it counts as a bump
-
sech1
anything above 50% shouldn't happen normally
-
merope
Shouldn't be below 50% either though - should it?
-
sech1
it should
-
sech1
-
sech1
as long as numbers run through several orders of magnitude (2^N in this case), it applies
-
merope
Hmmm
-
sech1
why would you expect bit 23, for example, to be at 50% if it's almost always 0?
-
sech1
because solution is found before it gets to 2^23
-
merope
As I mentioned above, it does explain the massive decrease in frequency of the higher bits
-
merope
But it doesn't explain why bits 15-16 are way higher than 13-14
-
sech1
I explained. "each thread starts from a multiple of 32768"
-
sech1
so they're changed a lot more often on multi-core CPUs
-
merope
Yeah, but the search is still incremental after that
-
sech1
incremental, but bit 15 is already set in each thread
-
sech1
8-core CPU starts search from values 0, 32768, ..., 229376
-
sech1
and it can find a solution randomly on each core, so bit 15 will be random
-
merope
But that would mean that the individual threads are way to slow to "fully finish" the search in each chunk before a new template shows up
-
sech1
when thread exhausts 32768 nonce values, it takes a new multiple of 32768
-
merope
...which at ~600 H/s/thread would actually take 55 seconds per chunk, so it kinda makes sense actually
-
sech1
each runs typically runs at 500-900 h/s depending on CPU, so it has enough time to exhaust it
-
merope
Huh
-
sech1
plus some blocks are really slow, much slower than 2 minutes
-
merope
Hadn't considered the effect of the chunk size on the nonce distribution, interesting
-
merope
When I tried messing with the nonce selection in xmrig, I reduced the chunk size to 2048 to get more granular control of the nonce pattern
-
merope
Is there an actual performance benefit to splitting the search in chunks of 2^15, vs any other number?
-
sech1
every new chunk requires synchronization between threads, so there is a benefit