go - How to use a goroutine pool -


i want use go downloading stock price spreadsheets yahoo finance. i'll making http request every stock in own goroutine. have list of around 2500 symbols, instead of making 2500 requests in parallel, i'd prefer making 250 @ time. in java i'd create thread pool , reuse threads , when free. trying find similar, goroutine pool, if will, unable find resources. i'd appreciate if can tell me how accomplish task @ hand or point me resources same. thanks!

the simplest way, suppose, create 250 goroutines , pass them channel can use pass links main goroutine child ones, listening channel.

when links passed goroutines, close channel , goroutines finish jobs.

to secure main goroutine finished before children process data, can use sync.waitgroup.

here code illustrate (not final working version shows point) told above:

func worker(linkchan chan string, wg *sync.waitgroup) {    // decreasing internal counter wait-group goroutine finishes    defer wg.done()     url := range linkchan {      // analyze value , job here    } }  func main() {     lch := make(chan string)     wg := new(sync.waitgroup)      // adding routines workgroup , running     := 0; < 250; i++ {         wg.add(1)         go worker(lch, wg)     }      // processing links spreading them `free` goroutines     _, link := range yourlinksslice {         lch <- link     }      // closing channel (waiting in goroutines won't continue more)     close(lch)      // waiting goroutines finish (otherwise die main routine dies)     wg.wait() } 

Comments

Popular posts from this blog

c# - Send Image in Json : 400 Bad request -

jquery - Fancybox - apply a function to several elements -

An easy way to program an Android keyboard layout app -