我嘗試讀取以下格式的大文件：a string key, 200 values separated by comma并將其寫入地圖。我寫了這段代碼：package mainimport ( "bufio" "unsafe" "fmt" "log" "os" "runtime" "strings")func main() { file, err := os.Open("file_address.txt") if err != nil { log.Fatal(err) } defer file.Close() mp := make(map[string]float32) var total_size int64 = 0 scanner := bufio.NewScanner(file) var counter int64 = 0 for scanner.Scan() { counter++ sliced := strings.Split(scanner.Text(), ",") mp[sliced[0]] = 2.2 } if err := scanner.Err(); err != nil { log.Fatal(err) } fmt.Printf("loaded: %d. Took %d Mb of memory.", counter, total_size/1024.0/1024.0) fmt.Println("Loading finished. Now waiting...") var ms runtime.MemStats runtime.ReadMemStats(&ms) fmt.Printf("\n") fmt.Printf("Alloc: %d MB, TotalAlloc: %d MB, Sys: %d MB\n", ms.Alloc/1024/1024, ms.TotalAlloc/1024/1024, ms.Sys/1024/1024) fmt.Printf("Mallocs: %d, Frees: %d\n", ms.Mallocs, ms.Frees) fmt.Printf("HeapAlloc: %d MB, HeapSys: %d MB, HeapIdle: %d MB\n", ms.HeapAlloc/1024/1024, ms.HeapSys/1024/1024, ms.HeapIdle/1024/1024) fmt.Printf("HeapObjects: %d\n", ms.HeapObjects) fmt.Printf("\n")}這是輸出：loaded: 544594. Took 8 Mb of memory.Loading finished. Now waiting...Alloc: 2667 MB, TotalAlloc: 3973 MB, Sys: 2831 MBMallocs: 1108463, Frees: 401665HeapAlloc: 2667 MB, HeapSys: 2687 MB, HeapIdle: 11 MBHeapObjects: 706798Done!雖然密鑰僅占用約 8Mb，但程序占用約 2.7Gb 內(nèi)存！似乎sliced永遠不會從堆中刪除。我嘗試sliced=nil在末尾進行設(shè)置for，但沒有幫助。我讀過，如果我將整個文件加載到內(nèi)存中然后分割它，我可以避免這個問題，但是我必須逐行讀取文件，因為我沒有足夠的內(nèi)存來加載一些較大的文件文件。為什么內(nèi)存被占用了？處理完每一行后如何釋放它？
2 回答
拉丁的傳說
TA貢獻1789條經(jīng)驗 獲得超8個贊
為了高效地使用 CPU 和內(nèi)存，
key := string(bytes.SplitN(scanner.Bytes(), []byte(","), 2)[0])
mp[key] = 2.2
慕仙森
TA貢獻1827條經(jīng)驗 獲得超8個贊
我想我發(fā)現(xiàn)了問題！我對大文件的每一行進行切片。返回的[]string是一個切片，包含原始字符串（文件行）的子字符串?，F(xiàn)在的問題是，每個子串都不是一個新串。Is 只是一個slice，它保留對未切片字符串（文件行！）的引用。我保留了sliced[0]每一行，因此，我保留了對文件每一行的引用。垃圾收集器不會觸及讀取行，因為我仍然引用它。從技術(shù)上講，我讀取文件的所有行并將其保留在內(nèi)存中。
解決方案是將我想要的部分（sliced[0]）復(fù)制到一個新字符串，從而有效地丟失對整行的引用。我是這樣做的：
sliced := strings.Split(scanner.Text(), ",")
key_rune_arr := []rune(sliced[0])
key := string(key_rune_arr) // now key is a copy of sliced[0] without reference to line
mp[key] = 2.2 //instead of mp[sliced[0]] = 2.2
該程序現(xiàn)在變?yōu)椋?/p>
package main
import (
"bufio"
"unsafe"
"fmt"
"log"
"os"
"runtime"
"strings"
)
func main() {
file, err := os.Open("file_address.txt")
if err != nil {
log.Fatal(err)
}
defer file.Close()
mp := make(map[string]float32)
var total_size int64 = 0
scanner := bufio.NewScanner(file)
var counter int64 = 0
for scanner.Scan() {
counter++
sliced := strings.Split(scanner.Text(), ",")
key_rune_arr := []rune(sliced[0])
key := string(key_rune_arr) // now key is a copy of sliced[0] without reference to line
mp[key] = 2.2 //instead of mp[sliced[0]] = 2.2
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("loaded: %d. Took %d Mb of memory.", counter, total_size/1024.0/1024.0)
fmt.Println("Loading finished. Now waiting...")
var ms runtime.MemStats
runtime.ReadMemStats(&ms)
fmt.Printf("\n")
fmt.Printf("Alloc: %d MB, TotalAlloc: %d MB, Sys: %d MB\n",
ms.Alloc/1024/1024, ms.TotalAlloc/1024/1024, ms.Sys/1024/1024)
fmt.Printf("Mallocs: %d, Frees: %d\n",
ms.Mallocs, ms.Frees)
fmt.Printf("HeapAlloc: %d MB, HeapSys: %d MB, HeapIdle: %d MB\n",
ms.HeapAlloc/1024/1024, ms.HeapSys/1024/1024, ms.HeapIdle/1024/1024)
fmt.Printf("HeapObjects: %d\n", ms.HeapObjects)
fmt.Printf("\n")
}
結(jié)果正如我所希望的那樣：
loaded: 544594. Took 8 Mb id memory.Loading finished. Now waiting...
Alloc: 94 MB, TotalAlloc: 3986 MB, Sys: 135 MB
Mallocs: 1653590, Frees: 1108129
HeapAlloc: 94 MB, HeapSys: 127 MB, HeapIdle: 32 MB
HeapObjects: 545461
Done!
- 2 回答
- 0 關(guān)注
- 180 瀏覽
添加回答
舉報
0/150
提交
取消