3 回答

TA貢獻(xiàn)1848條經(jīng)驗(yàn) 獲得超2個(gè)贊
這里的第一個(gè)問題是在發(fā)生錯(cuò)誤的情況下您不會(huì)返回響應(yīng),因此len(responses) == len(urls)很可能永遠(yuǎn)不會(huì)匹配,從而迫使您的循環(huán)永遠(yuǎn)繼續(xù)下去。
首先sync.WaitGroup為并發(fā)請(qǐng)求添加一個(gè)
var wg sync.WaitGroup
ch := make(chan *HttpResponse)
responses := []*HttpResponse{}
for _, url := range urls {
wg.Add(1)
go func(url string) {
defer wg.Done()
然后你可以覆蓋響應(yīng),直到所有未完成的 goroutine 都完成
go func() {
wg.Wait()
close(ch)
}()
for r := range ch {
fmt.Printf("%s was fetched\n", r.url)
responses = append(responses, r)
}
return responses
然后,您必須決定如何處理響應(yīng),您是要在并發(fā)調(diào)用中讀取它們,還是返回它們的正文未讀。由于如果您想重用連接,您將始終Body.Close()嘗試使用主體,并且由于您已經(jīng)推遲了,因此目前需要在同一個(gè)函數(shù)調(diào)用中發(fā)生。您可以更改httpResponse類型以使其成為可能,或者將 替換為resp.Body包含響應(yīng)的緩沖區(qū)。
最后,您將希望為客戶端設(shè)置某種超時(shí)(可能使用 a Context),并對(duì)發(fā)出的并發(fā)請(qǐng)求數(shù)進(jìn)行限制。

TA貢獻(xiàn)1878條經(jīng)驗(yàn) 獲得超4個(gè)贊
問題是您在沒有寫入通道的情況下返回錯(cuò)誤??茨愕膇f err != nil { return }說法。因?yàn)槟悴粚懶沤o頻道,所以len(responses) == len(urls)聲明永遠(yuǎn)不會(huì)是真的。
go func(url string) {
fmt.Printf("Fetching %s \n", url)
resp, err := http.Get("http://" + url)
if err != nil {
fmt.Printf("Failed to fetch %s\n", err)
return
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
fmt.Printf("HTTP Response Status : %v", resp.StatusCode)
bodyBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
fmt.Printf("HTTP Response Content Length : %v\n", len(bodyString))
}
ch <- &HttpResponse{url, resp, err}
}(url)

TA貢獻(xiàn)1868條經(jīng)驗(yàn) 獲得超4個(gè)贊
您可以使用以下庫:
Requests:一個(gè) Go 庫,用于減少發(fā)出 HTTP 請(qǐng)求時(shí)的麻煩(20k/s req)
https://github.com/alessiosavi/Requests
它是為解決to many open files處理并行請(qǐng)求而開發(fā)的。
這個(gè)想法是分配一個(gè)請(qǐng)求列表,而不是使用可配置的“并行”因子發(fā)送它們,該因子允許一次只運(yùn)行“N”個(gè)請(qǐng)求。
初始化請(qǐng)求(你已經(jīng)有一組 url)
// This array will contains the list of request
var reqs []requests.Request
// N is the number of request to run in parallel, in order to avoid "TO MANY OPEN FILES. N have to be lower than ulimit threshold"
var N int = 12
// Create the list of request
for i := 0; i < 1000; i++ {
// In this case, we init 1000 request with same URL,METHOD,BODY,HEADERS
req, err := requests.InitRequest("https://127.0.0.1:5000", "GET", nil, nil, true)
if err != nil {
// Request is not compliant, and will not be add to the list
log.Println("Skipping request [", i, "]. Error: ", err)
} else {
// If no error occurs, we can append the request created to the list of request that we need to send
reqs = append(reqs, *req)
}
}
此時(shí),我們有一個(gè)列表,其中包含必須發(fā)送的請(qǐng)求。讓我們并行發(fā)送它們!
// This array will contains the response from the givens request
var response []datastructure.Response
// send the request using N request to send in parallel
response = requests.ParallelRequest(reqs, N)
// Print the response
for i := range response {
// Dump is a method that print every information related to the response
log.Println("Request [", i, "] -> ", response[i].Dump())
// Or use the data present in the response
log.Println("Headers: ", response[i].Headers)
log.Println("Status code: ", response[i].StatusCode)
log.Println("Time elapsed: ", response[i].Time)
log.Println("Error: ", response[i].Error)
log.Println("Body: ", string(response[i].Body))
}
您可以在存儲(chǔ)庫的示例文件夾中找到示例用法。
- 3 回答
- 0 關(guān)注
- 121 瀏覽
添加回答
舉報(bào)