Skip to content

Conversation

@jfontan
Copy link
Contributor

@jfontan jfontan commented Jun 13, 2018

After each retry it waited a fixed amount of time. This caused it to retry at least once per message, that is, a 50 ms extra sleep per job.

Now it blocks for a configurable time if no messages are arriving. No retry is needed and jobs are processes as fast as they arrive.

Previously it processed 20 jobs per second. Now it processes 1220 jobs per second.

Needs #4
Fixes #5

@jfontan jfontan requested a review from a team June 13, 2018 14:47
}

func (i *JobIter) nextNonBlocking() (*queue.Job, error) {
func (i *JobIter) nextWithTimeout(timeout time.Duration) (*queue.Job, error) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can make this function more generic (if it makes sense), e.g.:

func (i *JobIter) nextWithTimeout(timeout time.Duration, fn func(*queue.Job) error) error { } 
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextWithTimeout is a version of Next (https://github.com/src-d/go-queue/blob/master/amqp/amqp.go#L492-L500) that does not block forever. It's only used in republish buried function so I don't see the benefit of adding a function parameter.

I suppose the best thing to do is to change Next to be able to timeout instead of adding a private method. Maybe adding the timeout value when constructing JobIter. It's not done like this as I didn't want to do bigger changes to the Queue library.

jfontan added 2 commits June 13, 2018 18:10
After each retry it waited a fixed amount of time. This caused it to retry at least once per message, that is, a 50 ms extra sleep per job. Now it blocks for a configurable time if no messages are arriving. No retry is needed and jobs are processes as fast as they arrive. Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
@jfontan jfontan force-pushed the improvement/use-blocking-call-for-republish branch from 43530c2 to 762e6f4 Compare June 13, 2018 16:11
@jfontan jfontan merged commit eb1b2b0 into src-d:master Jun 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants