-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(query): Read just the latest value for scalar types #8966
base: main
Are you sure you want to change the base?
Conversation
53ec545
to
3a2f444
Compare
8e0dc79
to
191f210
Compare
191f210
to
2a07e8b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added inline comments
posting/lists.go
Outdated
|
||
pl, err := getPostings() | ||
if err == badger.ErrKeyNotFound { | ||
err = nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return empty posting with nil error
worker/task.go
Outdated
@@ -376,6 +376,9 @@ func (qs *queryState) handleValuePostings(ctx context.Context, args funcArgs) er | |||
|
|||
outputs := make([]*pb.Result, numGo) | |||
listType := schema.State().IsList(q.Attr) | |||
hasLang := schema.State().HasLang(q.Attr) | |||
getMultiplePosting := q.DoCount || q.ExpandAll || listType || hasLang | |||
//getMultiplePosting := true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove commented line
@@ -195,6 +195,59 @@ func (lc *LocalCache) getInternal(key []byte, readFromDisk bool) (*List, error) | |||
return lc.SetIfAbsent(skey, pl), nil | |||
} | |||
|
|||
func (lc *LocalCache) GetSinglePosting(key []byte) (*pb.PostingList, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since its a public function could you add a description of what this function's doing/why its necessary? basically what you explained yesterday in text format would be helpful for future reference
posting/lists.go
Outdated
return pl, nil | ||
} | ||
} | ||
lc.RUnlock() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not defer unlock till the end of the function?
d572cae
to
29cb753
Compare
worker/task.go
Outdated
@@ -376,6 +376,9 @@ func (qs *queryState) handleValuePostings(ctx context.Context, args funcArgs) er | |||
|
|||
outputs := make([]*pb.Result, numGo) | |||
listType := schema.State().IsList(q.Attr) | |||
hasLang := schema.State().HasLang(q.Attr) | |||
getMultiplePosting := q.DoCount || q.ExpandAll || listType || hasLang |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add some explanation
posting/list_test.go
Outdated
require.NoError(t, err) | ||
checkValue(t, ol, string(k.Postings[0].Value), uint64(j)) | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: remove newline
80102bb
to
e893711
Compare
The base branch was changed.
e893711
to
0ce037c
Compare
Description: Right now, when we read some data, we read the entire posting list. With this we populate List structure in memory, which has two layers. These two layers are then merge sorted. All of this takes a lot of time. In this diff, we are updating the query processor to just read the latest value in case of scalar types.
LDBC test for IC09.
Before: 35 seconds
After: 26 seconds