Discussion of NodeBB Updates
-
@pie_flavor said in Discussion of NodeBB Updates:
@Tsaukpaetra said in Discussion of NodeBB Updates:
the most
Posts have a max char count of 32767.
Sure, but it only counts as one no matter how many times you repeat it in the post. Good try though!
-
@pie_flavor I wonder what happens to the Reply box if I popcorn this.
Edit: Dang, bug not invoked.
-
@pie_flavor said in Discussion of NodeBB Updates:
Dang, bug not invoked.
That's the TDWTF community in a nutshell for you.
-
@ben_lubar said in Discussion of NodeBB Updates:
@ben_lubar said in NodeBB Updates:
Fix for the IP address hashing thing coming soon.
Note to @administators: one of the server's cores will be used by PostgreSQL for a while the SHA1 hashes are being "reversed". Don't stop it.
It shouldn't affect forum performance by any noticeable amount while it runs.
Ok, the temporary table has 1197436 rows. That's 1197436 hashes. I'll reply again when it finishes with the number of rows in the mapping table (which might be higher due to new guest IPs or lower due to the garbage data that's in there).
Got bored, wrote a script to insert IPs based on a MongoDB backup from before the "does this help with GDPR" thing.
INSERT 0 1163148
-
Removed
unknown
and999.999.999.999
(last visited2016-07-07T15:09:52.165Z
and2016-03-24T23:19:55.475Z
respectively).
-
There were 7 IP addresses that weren't hashed in the database, but they also existed in hashed form with more recent timestamps so I deleted the non-hashed records.
Downloaded a list of remaining unknown hashes (32111 of them) and am currently running this program on them:
package main import ( "bufio" "crypto/sha1" "encoding/binary" "encoding/hex" "encoding/json" "flag" "fmt" "io/ioutil" "log" "net" "os" "runtime" "sort" "strings" "sync" "time" ) func usage() { log.Printf("Usage of %q:", os.Args[0]) flag.PrintDefaults() log.Println() log.Println("-i, -o, and -secret are required.") os.Exit(2) } var secret string func main() { log.SetFlags(0) flag.Usage = usage flagInput := flag.String("i", "", "input (JSON) file") flagOutput := flag.String("o", "", "output (JSON) file") flagSQL := flag.String("sql", "", "output (SQL) file") flag.StringVar(&secret, "secret", "", "NodeBB secret from config.json") flag.Parse() if *flagInput == "" || *flagOutput == "" || secret == "" || flag.NArg() != 0 { flag.Usage() panic("unreachable") } b, err := ioutil.ReadFile(*flagInput) if err != nil { log.Fatalln("failed to read input file:", err) } var unknown []struct { Hash string `json:"hash"` LastVisited time.Time `json:"last_visited"` } err = json.Unmarshal(b, &unknown) if err != nil { log.Fatalln("failed to unmarshal input file:", err) } var known map[string]string var knownLock sync.Mutex b, err = ioutil.ReadFile(*flagOutput) if err == nil { err = json.Unmarshal(b, &known) if err != nil { log.Fatalln("failed to unmarshal output file:", err) } } else if os.IsNotExist(err) { log.Println("Output file does not exist. Assuming no IPs are known.") known = make(map[string]string) } else { log.Fatalln("failed to read output file:", err) } for i := 0; i < len(unknown); i++ { if ip, ok := known[unknown[i].Hash]; ok { log.Printf("already known: %s = %q (last visit %v)", unknown[i].Hash, ip, unknown[i].LastVisited) unknown = append(unknown[:i], unknown[i+1:]...) i-- } } fmt.Printf("Known: %d - Unknown: %d\n", len(known), len(unknown)) want := make(map[[sha1.Size]byte]time.Time, len(unknown)) for _, u := range unknown { h, err := hex.DecodeString(u.Hash) if err != nil || len(h) != sha1.Size { if err == nil { err = fmt.Errorf("expected %d bytes for SHA1, but hex decoded to %d bytes", sha1.Size, len(h)) } log.Printf("skipping bad hash %q: %v", u.Hash, err) continue } var fixed [sha1.Size]byte copy(fixed[:], h) want[fixed] = u.LastVisited } if len(unknown) != 0 { // basic brute force (IPv4) var wg sync.WaitGroup workers := runtime.GOMAXPROCS(0) wg.Add(workers) var found [][sha1.Size]byte for i := 0; i < workers; i++ { go func(start, inc uint64) { defer wg.Done() for i := start; i < 1<<32; i += inc { ip := make(net.IP, 4) binary.BigEndian.PutUint32(ip, uint32(i)) hash := doHash(ip.String()) if t, ok := want[hash]; ok { knownLock.Lock() log.Printf("found new ip: %x = %q (last visit %v)", hash, ip, t) known[hex.EncodeToString(hash[:])] = ip.String() found = append(found, hash) knownLock.Unlock() } } }(uint64(i), uint64(workers)) } wg.Wait() for _, h := range found { delete(want, h) } log.Printf("Found: %d", len(found)) } b, err = json.Marshal(known) if err != nil { log.Fatalln("failed to marshal output file:", err) } err = ioutil.WriteFile(*flagOutput, b, 0644) if err != nil { log.Fatalln("failed to write output file:", err) } if *flagSQL != "" { hashes := make([]string, 0, len(known)) for h := range known { hashes = append(hashes, h) } sort.Strings(hashes) f, err := os.Create(*flagSQL) if err != nil { log.Fatalln("failed to open SQL file:", err) } defer func() { if err := f.Close(); err != nil { log.Fatalln("failed to close SQL file:", err) } }() w := bufio.NewWriter(f) defer func() { if err := w.Flush(); err != nil { log.Fatalln("failed to flush SQL file:", err) } }() check := func(_ int, err error) { if err != nil { log.Fatalln("failed to write SQL file:", err) } } check(w.WriteString("INSERT INTO wtdwtf_real_ip (ip, hash) VALUES\n")) for i, h := range hashes { term := ",\n" if i == len(hashes)-1 { term = "\nON CONFLICT DO NOTHING;\n" } check(fmt.Fprintf(w, "(inet '%s', E'\\\\x%s')%s", known[h], strings.ToUpper(h), term)) } } } func doHash(input string) [sha1.Size]byte { return sha1.Sum([]byte(input + secret)) }
The program is spewing out log lines like there's no tomorrow.
Edit: Made it peg all cores instead of just one. See edit history for previous version.
Edit 2: Made it able to write SQL output.
-
LGTM
nodebb=# select * from legacy_zset full outer join wtdwtf_real_ip on encode(hash, 'hex') = value where _key = 'ip:recent' and (ip is null or value is null); _key | value | score | type | ip | hash ------+-------+-------+------+----+------ (0 rows)
-
@ben_lubar So what exactly is the goal here? Are you trying to reverse the hashing on the IP addresses? If so, why? Just to see how easily it can be done?
-
@anotherusername said in Discussion of NodeBB Updates:
@ben_lubar So what exactly is the goal here? Are you trying to reverse the hashing on the IP addresses? If so, why? Just to see how easily it can be done?
I have completed the process.
The reason is that the admin panel page that shows which users are on instances that were recently restarted crashes the instance it's on if it gets SHA1 hashes instead of IPv4 addresses.
-
@ben_lubar aren't these hashes useful only to detect spam and sockpuppets? Why is it that big?
-
@sockpuppet7 said in Discussion of NodeBB Updates:
@ben_lubar aren't these hashes useful only to detect spam and sockpuppets? Why is it that big?
Obviously they're not "only" useful for one thing.
-
@sockpuppet7 said in Discussion of NodeBB Updates:
@ben_lubar aren't these hashes useful only to detect spam and sockpuppets? Why is it that big?
The "users with this IP address" index is not hashed.
-
@ben_lubar said in Discussion of NodeBB Updates:
The reason is that the admin panel page that shows which users are on instances that were recently restarted crashes the instance it's on if it gets SHA1 hashes instead of IPv4 addresses.
Or... You could have fixed the admin panel page?
EDIT: Assuming it works by collecting the IPv4 addresses on an instance and then checking them against the ip addresses of all users, couldn't it have just hashed the IPv4 addresses first and checked the hash against the hashed ip addresses of all users?
-
@ben_lubar said in Discussion of NodeBB Updates:
The system will continue to work because it records new IP hashes automatically.
GDPR Crime?
-
@CreatedToDislikeThis said in Discussion of NodeBB Updates:
@ben_lubar said in Discussion of NodeBB Updates:
The reason is that the admin panel page that shows which users are on instances that were recently restarted crashes the instance it's on if it gets SHA1 hashes instead of IPv4 addresses.
Or... You could have fixed the admin panel page?
Exactly how would I have made this code return the correct answers for salted SHA1 hashes?
-
@ben_lubar MOAR HASHES!
-
Status: Is now more hungry for hash.
-
@ben_lubar said in NodeBB Updates:
Thanks for the warning... Not.
Personally, I would have waited for Saturday and then had the slapstick "666-day uptime was too evil" joke to make.
-
@ben_lubar said in NodeBB Updates:
What?! You mean you had to reboot on Linux?! Inconceivable! I've been assured that this is never the case.
-
@heterodox said in Discussion of NodeBB Updates:
@ben_lubar said in NodeBB Updates:
What?! You mean you had to reboot on Linux?! Inconceivable! I've been assured that this is never the case.
And for something that, as far as the link he provided is leading me to believe is seemingly telling me, is not an issue with our host.
Unless, of course, we happen to be sharing out the forums' server out to anonymous parties running arbitrary code on them. Which, to my knowledge, is not happening.
-
@heterodox said in Discussion of NodeBB Updates:
@ben_lubar said in NodeBB Updates:
What?! You mean you had to reboot on Linux?! Inconceivable! I've been assured that this is never the case.
It's 100% Intel's fault.
-
@Tsaukpaetra said in Discussion of NodeBB Updates:
And for something that, as far as the link he provided is leading me to believe is seemingly telling me, is not an issue with our host.
Unless, of course, we happen to be sharing out the forums' server out to anonymous parties running arbitrary code on them. Which, to my knowledge, is not happening.You don't get to make the call whether to take a security update or not. Your complaint is rightfully riddled with ambivalent phrases because you're not in the position to assess the risk.
-
@heterodox said in Discussion of NodeBB Updates:
@ben_lubar said in NodeBB Updates:
What?! You mean you had to reboot on Linux?! Inconceivable! I've been assured that this is never the case.
He had to. The screensaver had locked up the machine.
-
@heterodox said in Discussion of NodeBB Updates:
@Tsaukpaetra said in Discussion of NodeBB Updates:
And for something that, as far as the link he provided is leading me to believe is seemingly telling me, is not an issue with our host.
Unless, of course, we happen to be sharing out the forums' server out to anonymous parties running arbitrary code on them. Which, to my knowledge, is not happening.You don't get to make the call whether to take a security update or not. Your complaint is rightfully riddled with ambivalent phrases because you're not in the position to assess the risk.
So... I did the needful as a luser? :D
-
@ben_lubar said in NodeBB Updates:
Is this (the link) meant to be a notification, or an opportunity for some preening and some major, and extensive, rotator-cuff injuries?
CVE-2018-3615 (also known as Foreshadow), CVE-2018-3620, and CVE-2018-3646
Raoul Strackx, Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and researchers from Intel discovered that
-
I'm compressing some old database backups on the server, but it shouldn't affect performance because the compressor has been instructed to be very nice.
-
@ben_lubar said in NodeBB Updates:
I had to look up what a Xerus is, but it's just a dumb squirrel, so now we're on Cyborg Castor canadensis.
Edit: The reboot will happen at 03:00 UTC. There will be a brief downtime, but this isn't Windows, so expect to have the forum back within a minute or two.
I can't hack in the reply-to-id thing, so have a fake attempt.
-
@Tsaukpaetra oi, I was going to do that!
-
@ben_lubar said in NodeBB Updates:
is because the SSH daemon refused to start. with a valid configuration file.
Does it not check the configuration before altering the status of the service?
IIRC nginx throws a fit and won't even stop if the conf is bad...
-
@ben_lubar said in NodeBB Updates:
Update scheduled for 15:00 UTC
- Fixed "in reply to" button on dark themes by explicitly setting the foreground color.
- Added
nodebb-plugin-pwned-passwords
(not yet enabled). - The WTDWTF forum customizations are now available under a new license.
So, good news... the "Popcorn Button Duplicated reply count" bug seems to be fixed for me.
Bad news, I couldn't see the Popcorn button (on any posts in the Updates thread) until I dropped down the reply count of a previous post...
I saw this:
Though once I expanded the previous post's reply, the popcorn button showed up for both the previous posts and the new one.
... Though maybe it was something cache related, going back to the thread has the popcorn button again now, and the counts are correct
-
Status: did some database maintenance that locked the entire database for a few minutes but apparently nobody noticed?
-
@ben_lubar
Oh, believe me, I noticed :pitchfork:
-
Is my commit message too short?
-
@ben_lubar I've written a couple as long.
-
@ben_lubar said in NodeBB Updates:
- HTML comments are allowed in posts
-
@boomzilla THANKS, OBAMA
-
-
@ben_lubar said in NodeBB Updates:
Update scheduled for 19:30 UTC
- [X] Fix this garbage
- HTML comments are allowed in posts
Too bad the HTML comments can't be displayed with CSS like the @Gribnit-style comments can, though...
-
@ben_lubar said in NodeBB Updates:
- HTML comments are allowed in posts
-
-
-
-
@anotherusername hey, it worked! But my userscript didn't.
-
@anotherusername <!--
dunno why it didnt work
edit: it needed a hard reset
edit2: but why it didn't work now?
-->
-
so what are checkboxes like now?
[X] Test
[ ] Test 2oh good, removed entirely
-
@sockpuppet7 said in Discussion of NodeBB Updates:
Because you posted it before
@ben_lubar said in NodeBB Updates:
It's a time-the-post-was-baked thing.
Old posts didn't get rebaked.Well I guess they did, but they didn't display properly until a hard refresh (while posts that were posted after 19:30 UTC did display properly).
-
@sockpuppet7 said in Discussion of NodeBB Updates:
@anotherusername <!--
dunno why it didnt work
edit: it needed a hard reset
edit2: but why it didn't work now?
-->
I think the paragraphizing is breaking that, but I'm not sure why...
-
@anotherusername said in Discussion of NodeBB Updates:
@sockpuppet7 said in Discussion of NodeBB Updates:
@anotherusername <!--
dunno why it didnt work
edit: it needed a hard reset
edit2: but why it didn't work now?
-->
I think the paragraphizing is breaking that, but I'm not sure why...
I believe it's because the Markdown spec requires multiline HTML elements to be the first thing in a line.
-
@ben_lubar Why did you get rid of checkboxes?
-
@ben_lubar there, the HTML sanitizer appears to be escaping the
<!--
and-->
sequences, probably due to the fact that they're "in different paragraphs". Even though actually, they're not --<p>This is a paragraph <!-- with a comment, where </p><p> should be ignored --> and this is still the same paragraph</p>
should be rendered as one single paragraph with a comment in it. And it is rendered as one single paragraph with a comment in it, if the post is written in HTML instead of markdown...
works properly, while
@anotherusername <!--
dunno why it didnt work
edit: it needed a hard reset
edit2: but why it didn't work now?
-->
does not.
Is the Markdown processor doing something funky like calling the HTML sanitizer for each paragraph separately?