00:25:01 <gingeropolous> it got sigkill. wtf
00:28:19 <gingeropolous> so i had 52 lines of stuff like this: [Thread 0x7ffff4dafa00 (LWP 104520) exited]
00:28:32 <gingeropolous> Program terminated with signal SIGKILL, Killed.
00:28:32 <gingeropolous> The program no longer exists.
00:28:38 <gingeropolous> did it get OOM killed?
00:29:54 <gingeropolous> ima try running it with --max-concurrency 4
00:30:11 <gingeropolous> it *is* a 32 thread cpu with only 16 gb of ram
02:32:25 <upp> im kind of in a perpetual lifelong state of shock at the programming convention of just pretending that there is infinite memory, and if your program exceeds what memory the system *actually* has, then this is totally fine, we should just let the computer crash and catch fire and force the user to pull the plug and start over, and all of that is totally fine too
02:32:44 <upp> WTF kind of convention is that?
02:33:55 <upp> computers have existed for how many decades now and we are still doing that
02:40:35 <gingeropolous> its v0.17.3.0_
02:40:44 <gingeropolous> compiled on ubuntu 20
02:41:00 <gingeropolous> wtf. "the program no longer exists" again
02:41:39 <gingeropolous> this time there were only 23 threads
02:42:21 <upp> my chain was a few years old and i synced fine recently, i guess you have to start at 0 to get this crash?
02:42:40 <gingeropolous> dude i got daemons running all over the place. this is the first time i've seen this
02:43:00 <gingeropolous> i mean, i've got xmrig mining monero on all threads at the same time
02:43:09 <gingeropolous> but this isn't any different than any other setup
02:43:23 <gingeropolous> ugh, i know there's a way to log memory usage instead of just starting at top
02:44:07 <gingeropolous> well there's always a bash loop with free -h.... 
02:44:37 <gingeropolous> oh i guess i'll google it
02:44:58 <upp> i think you can track a specific process with top?
02:47:27 <gingeropolous> yeah but im trying to dump it to a log file so i can crockpot it
02:55:45 <gingeropolous> i got it
02:56:52 <Yin-Dawg> what was the issue?
02:57:19 <gingeropolous> i mean i figured out my log situation to watch the memory :)_
02:58:04 <gingeropolous> my guess is there's a memory leak somewhere. if i can get this to reproduce quickly ima try a git bisect
02:58:13 <gingeropolous> because what are SSDs there for but to shred
02:58:27 <Yin-Dawg> LOLs
02:58:30 <Mochi102> SShreD
09:44:30 <selsta> gingeropolous: right, sigkill seems like the OS is killing the process and not a crash
09:45:06 <selsta> check with htop if you have enough resources, but currently there is no known memory leak in v0.17.3.0
13:13:15 <gingeropolous> yeah. so according to my bash loop of ps, monerod got up to 50% ram utilization and was then killed
13:16:36 <gingeropolous> im trying it on a different box
13:16:42 <gingeropolous> with the same amount of ram
13:29:12 <gingeropolous> alright, 0.17.2.3 up to 49.4%
13:33:57 <gingeropolous> 58.6 ....
13:34:57 <gingeropolous> i mean im running log level 3. i guess i could test if high logging is responsible but ...
13:35:08 <gingeropolous> and killed
13:35:17 <gingeropolous> on v0.17.2.3 . wtf
13:36:31 <gingeropolous> or i guess i stop the mining process. yeah lets do that
13:42:49 <hyc> how much RAM is in that box? did you allocate 2GB of hugepages for the miner?
14:49:14 <selsta> I currently run 5 monero daemons + tor and i2p with 6gb ram usage
15:01:06 <hyc> hugepages are a separate memory allocation. they're not pageable, they're not interchangeable with other memory.
15:01:28 <hyc> so allocating them is like removing 2GB from the machine, they're unavailable for any other use.
17:38:55 <gingeropolous> nope, wasn't the mining either
17:38:59 <gingeropolous> 16 gb in the box
17:39:32 <gingeropolous> i honestly just think its been a while since ive tried a fresh sync
17:39:56 <gingeropolous> ok, got killed on another box with 16 gb ram
17:40:02 <gingeropolous> ubuntu 20
17:41:40 <gingeropolous> huh, shot up to 76.4% usage before it got killed. perhaps i need to log memory usage more than every 10 seconds
17:47:26 <gingeropolous> or maybe because i usually run with systemd the memory gets managed better? but then again, thats always with an old chain, never a fresh sync
17:59:48 <gingeropolous> "your node is 7.9 years behind" ..... oh time
18:06:37 <rbrunner> Feeling old already? :)
18:14:20 <moneromooo> You can try applying wfaressuissia's serialization patch. It cuts down a lot of memory usage.
18:21:02 <gingeropolous> i mean, im generally not affected. but noobs doing fresh syncs on your average computer might be encountering this
18:43:39 <gingeropolous> wow. so it seems it was the log level
18:43:55 <hyc> maybe the logger itself has a memleak
18:44:16 <gingeropolous> log level 3 and 4 cause ever increasing mem usage
18:44:47 <hyc> would be nice if we could maybe do some sort of annual rollup txns so we would no longer need the preceding year's worth of blocks
18:45:01 <gingeropolous> they were in screen sessions. i wonder if it happens with it being daemonized sans screen
18:45:13 <hyc> no idea how that would work since we still need th txos
18:45:45 <hyc> if your screen session has a ridiculously large scrollback history setting, that could've been it
18:45:59 <hyc> but then you should've seen the memory being attributed to screen, not to monerod
18:46:18 <gingeropolous> its default scrollback. didn't know i could mod that :)
18:46:28 <hyc> you can mod everything
18:46:57 <hyc> ctrl-A : scrollback 10000
18:47:34 <gingeropolous> lulz. the one box is on wifi and the data stream is screwing up other wifi things in the house
18:47:54 <hyc> I always switch away to an idle shell
18:48:16 <hyc> so the terminal contents doesn't spew across the network until \I want to take a look
18:50:00 <gingeropolous> gotta close the windows so you don't let the wifi out
18:51:42 <hyc> i only use organic free range wifi
19:11:39 <rbrunner> "organic free range wifi" ?
19:13:45 <moneromooo> Organic: does not leach harmful chemicals in the air where you can breathe it in; free: as in software; range: because wifi if you're within half a meter from your router sucks.
19:14:49 <moneromooo> Maybe you're missing "free range", it means "allowed to roam" for poultry. I guess it might be a UK thing only.
19:15:28 <rbrunner> I am mostly puzzled about that "organic" part.
19:16:08 <moneromooo> Following a set of rules that basically say "don't be a jerk to the environment".
19:16:32 * moneromooo might have missed a joke here
19:16:50 <rbrunner> Ah, I think I finally got the joke. It's about "don't let the wifi out" from ginger
19:17:33 <rbrunner> hyc does not have a problem there, this wifi is indeed always roaming free, and even organic on top.
19:25:21 <ndorf> free range is a thing in the US, too
19:30:08 <moneromooo> With a name like that, I'd expect something to do with long rifles in the US ^_^
19:30:27 <ndorf> haha, that too!
20:00:40 <gingeropolous> ok, well the wifi one got to 51% with default logs
20:01:06 <gingeropolous> and once i stopped sync my music works again.