master client: retry connecting to etcd #3649
Conversation
| c.ch = make(chan record, c.bufSize) | ||
| // FIXME: connection is created asyncrosly in monitorMaster go routine, | ||
| // ensure the connection is ready for use before calling c.addClient. | ||
| time.Sleep(time.Second) |
There was a problem hiding this comment.
Deleting because I think without this line the program should work (c.conn.Call will block until a connection is established). If deleting this line does not work, I would be happy to revert this line and send another PR to fix it.
There was a problem hiding this comment.
Creating connection seems already moved to NewClient, these lines should be removed.
go/master/client.go Outdated
| } | ||
| } | ||
| | ||
| func retry(f func() error, dur time.Duration, count int) error { |
go/master/client.go Outdated
| if err != nil { | ||
| if count > 0 { | ||
| time.Sleep(dur) | ||
| return retry(f, dur, count-1) |
There was a problem hiding this comment.
I thought should be waiting forever if etcd is not started, this is also what we do in paddle cloud starting up scripts.
There was a problem hiding this comment.
I see, make sense, will change to wait forever.
I got this error when trying Paddle cloud with fault tolerant on (log below). Anyway, it's better to handle this in the master code as well.
➜ paddle git:(develop) ✗ paddlecloud logs -n 1000 b ==========================b-trainer-brwn0========================== time="2017-08-23T23:42:29Z" level=info msg="Waiting for ps desired registered ..." ... # a lot of same lines time="2017-08-24T01:05:47Z" level=info msg="Waiting for ps desired registered ..." ==========================b-trainer-lbs8m========================== label selector: paddle-job-master=b, desired: 1 label selector: paddle-job=b, desired: 1 Starting training job: /pfs/dlnel/home/helinwang@baidu.com/jobs/b, num_gradient_servers: 1, trainer_id: 0, version: I0823 20:36:07.662369 34 Util.cpp:166] commandline: --num_gradient_servers=1 --ports_num_for_sparse=1 --use_gpu=0 --trainer_id=0 --trainer_count=1 --num_passes=1 --ports_num=1 --port=7164 I0823 20:36:07.668710 34 GradientMachine.cpp:85] Initing parameters.. I0823 20:36:07.668742 34 GradientMachine.cpp:92] Init parameters done. panic: dial tcp 10.1.93.6:2379: getsockopt: connection refused goroutine 17 [running, locked to thread]: main.paddle_new_etcd_master_client(0x7faf728eebd4, 0x5, 0x40, 0x1c400000008) /paddle/build/go/src/github.com/PaddlePaddle/Paddle/go/master/c/client.go:79 +0x132 main._cgoexpwrap_f2aa1382a54e_paddle_new_etcd_master_client(0x7faf728eebd4, 0x5, 0x40, 0xd74167732b8bc700) command-line-arguments/_obj/_cgo_gotypes.go:100 +0x41 Aborted (core dumped) job returned 134...setting pod return message... =============================== termination log wroted... There was a problem hiding this comment.
Thanks! Done. Now master server will wait etcd forever.
3 participants
No description provided.