Discourse search requests

Arantor

Because They. Know. Best.

Haven't you been paying attention this whole time? :P

sam

@abarker said:

You're situation probably isn't in the 0.01% of cases where hijacking the browser is ok.

I don't know about this...

You navigate to a topic at post 1
You scroll down to post 200
We unload based on total height, once you navigate down 10 screens we start unloading (but keep scrollbar for that area)
End result is that say posts 1-80 are not in the DOM
You search in page
We only light up stuff in posts 80-200
To add insult to injury unless you disabled spam history as you scroll we clear results.

The experience is less than spectacular.

abarker

@sam said:

I don't know about this...

You navigate to a topic at post 1

You scroll down to post 200

We unload based on total height, once you navigate down 10 screens we start unloading (but keep scrollbar for that area)

End result is that say posts 1-80 are not in the DOM

You search in page

We only light up stuff in posts 80-200

To add insult to injury unless you disabled spam history as you scroll we clear results.

The experience is less than spectacular.

I understand where you're coming from, so let me add a corollary to that rule I proposed: If you find that you absolutely must hijack a browser function, emulate it as closely as possible. This will prevent user confusion and make the hijacking less apparent, thus minimizing user frustration.

sam

Hi, I already agree on that :) give me a chance to fix it.

abarker

Yeah, based on your earlier comments, I figured that. Mainly posted for the benefit of others.

sam

I just fixed it so we use a trivial ILIKE for in topic search, this means that you will now get search hits on the topic if you search for stuff like me a or this this

Not quite perfect, but much improved.

TODO:

Tokenise the cooked with a trivial tokenizer (at the moment you will not get hits on expanded oneboxes)
Strip spaces fnd useless chars from tokenised output
Better UI for navigating the results

boomzilla

@sam said:

Better UI for navigating the results

+1 +1 +1 +1

abarker

It is improved, but the original scenario I mentioned is still an issue. To reiterate, so you don't have to re-read the topic:

Searching for "sacrificed" or "sacrific" includes I'm A Grumpy Cat: An open letter to Alex in the results.
Searching for "sacrifice" does not include I'm A Grumpy Cat: An open letter to Alex in the results. There isn't even an option to "Show more results".

As far as I can figure, that's got to be due to some sort of custom search function.

sam

OK, so we confirm it works on topic search

However is broken for general search. I am not overly comfortable with our fulltext search config.

sam

Just to confirm, this is the rubbish snowball stemmer:

discourse_development=# select * from to_tsvector('english', 'sacrificed');
 to_tsvector 
-------------
 'sacrif':1
(1 row)

discourse_development=# select * from to_tsvector('english', 'sacrifice');
 to_tsvector  
--------------
 'sacrific':1
(1 row)

discourse_development=# select * from to_tsvector('english', 'sacrific');
 to_tsvector 
-------------
 'sacrif':1
(1 row)

I am seeing if I can add in hunspell which is far less crazy:

discourse_development=# select * from to_tsvector('hunspell_english', 'sacrificed');
  to_tsvector  
---------------
 'sacrifice':1
(1 row)

discourse_development=# select * from to_tsvector('hunspell_english', 'sacrifice');
 to_tsvector 
-------------
  'sacrifice':1
(1 row)

For added bonus hunspell would do wonderful stuff like this:

discourse_development=# select * from to_tsvector('hunspell_english', 'tester');
     to_tsvector     
---------------------
 'test':1 'tester':1
(1 row)

Which our current stemmer does not, so searching for "test" will still score "tester" high.

ben_lubar

For something that advertises "Search that actually works" on the front page of its website, I'm surprised you didn't find this sooner.

sam

To be honest snowball is the least of the problems our search system has, we need a way to see results in a region larger than 100px*100px , far more of a huge problem. Also, zero support for Chinese / Japanese Korean is a big problem.

Search is hard.

M_Adams

@sam said:

add in hunspell which is far less crazy

@ben_lubar said:

For something that advertises "Search that actually works" on the front page of its website, I'm surprised you didn't find this sooner.

Hunspell is so much better than snoball that there is really no comparison, especially for tsearch in Postgresql… Not sure how you'd deal with Chinese / Japanese / Korean et al…

dkf

@sam said:

For added bonus hunspell would do wonderful stuff like this:

[… 'test' and 'tester' …]

Which our current stemmer does not, so searching for "test" will still score "tester" high.

Would it also map testy and testes to the same root? That would seem… appropriate…

dkf

(And agh, quoting code seems hard.)

abarker

@dkf said:

Would it also map testy and testes to the same root? That would seem… appropriate…

Well, at least for this group.

abarker

That appears to provide a significant improvement.