第七色在线视频,2021少妇久久久久久久久久,亚洲欧洲精品成人久久av18,亚洲国产精品特色大片观看完整版,孙宇晨将参加特朗普的晚宴

為了賬號(hào)安全,請(qǐng)及時(shí)綁定郵箱和手機(jī)立即綁定
已解決430363個(gè)問(wèn)題,去搜搜看,總會(huì)有你想問(wèn)的

Golang - gzipping mongodb find 查詢的游標(biāo)數(shù)據(jù),寫入文件并解壓縮時(shí)出錯(cuò)

Golang - gzipping mongodb find 查詢的游標(biāo)數(shù)據(jù),寫入文件并解壓縮時(shí)出錯(cuò)

Go
開(kāi)心每一天1111 2022-06-01 12:24:39
我正在迭代一個(gè) mongodb 游標(biāo)并將數(shù)據(jù)壓縮并發(fā)送到 S3 對(duì)象。嘗試使用 解壓縮上傳的文件gzip -d時(shí),出現(xiàn)以下錯(cuò)誤,gzip: 9.log.gz: invalid compressed data--crc errorgzip: 9.log.gz: invalid compressed data--length error下面給出了我用于迭代、壓縮、上傳的代碼,// CursorReader struct acts as reader wrapper on top of mongodb cursortype CursorReader struct {    Csr *mongo.Cursor}// Read func reads the data from cursor and puts it into byte arrayfunc (cr *CursorReader) Read(p []byte) (n int, err error) {    dataAvail := cr.Csr.Next(context.TODO())    if !dataAvail {        n = 0        err = io.EOF        if cr.Csr.Close(context.TODO()) != nil {            fmt.Fprintf(os.Stderr, "Error: MongoDB: getting logs: close cursor: %s", err)        }        return    }    var b bytes.Buffer    w := gzip.NewWriter(&b)    w.Write([]byte(cr.Csr.Current.String() + "\n"))    w.Close()    n = copy(p, []byte(b.String()))    err = nil    return}cursor, err := coll.Find(ctx, filter) // runs the find query and returns cursorcsrRdr := new(CursorReader) // creates a new cursorreader instancecsrRdr.Csr = cursor // assigning the find cursor to cursorreader instance_, err = s3Uploader.Upload(&s3manager.UploadInput{  // Uploading the data to s3 in parts    Bucket: aws.String("bucket"),    Key:    aws.String("key")),    Body:   csrRdr, })如果數(shù)據(jù)低,那么我沒(méi)有得到問(wèn)題。但如果數(shù)據(jù)很大,那么我就會(huì)出錯(cuò)。到目前為止我調(diào)試的東西,試圖壓縮 1500 個(gè)文檔,每個(gè)大小為 15MB,得到錯(cuò)誤。即使我嘗試將壓縮后的字節(jié)直接寫入本地文件,但我得到了同樣的錯(cuò)誤。
查看完整描述

1 回答

?
Helenr

TA貢獻(xiàn)1780條經(jīng)驗(yàn) 獲得超4個(gè)贊

問(wèn)題似乎是反復(fù)調(diào)用gzip.NewWriter()infunc(*CursorReader) Read([]byte) (int, error)


您正在gzip.Writer為每個(gè)調(diào)用分配一個(gè)新的Read. gzip壓縮是有狀態(tài)的,因此您只能Writer對(duì)所有操作使用單個(gè)實(shí)例。


解決方案#1

解決您的問(wèn)題的一個(gè)相當(dāng)簡(jiǎn)單的方法是讀取游標(biāo)中的所有行并將其傳遞gzip.Writer并將 gzip 壓縮的內(nèi)容存儲(chǔ)到內(nèi)存緩沖區(qū)中。


var cursor, _ = collection.Find(context.TODO(), filter)

defer cursor.Close(context.TODO())


// prepare a buffer to hold gzipped data

var buffer bytes.Buffer

var gz = gzip.NewWriter(&buffer)

defer gz.Close()


for cursor.Next(context.TODO()) {

    if _, err = io.WriteString(gz, cursor.Current.String()); err != nil {

        // handle error somehow  ˉ\_(ツ)_/ˉ

    }

}


// you can now use buffer as io.Reader

// and it'll contain gzipped data for your serialized rows

_, err = s3.Upload(&s3.UploadInput{

    Bucket: aws.String("..."),

    Key:    aws.String("...")),

    Body:   &buffer, 

})

解決方案#2

另一種解決方案是使用goroutines創(chuàng)建一個(gè)流,按需讀取和壓縮數(shù)據(jù),而不是在內(nèi)存緩沖區(qū)中io.Pipe()。如果您正在讀取的數(shù)據(jù)非常大并且您無(wú)法將所有數(shù)據(jù)都保存在內(nèi)存中,這將非常有用。


var cursor, _ = collection.Find(context.TODO(), filter)

defer cursor.Close(context.TODO())


// create pipe endpoints

reader, writer := io.Pipe()


// note: io.Pipe() returns a synchronous in-memory pipe

// reads and writes block on one another

// make sure to go through docs once.


// now, since reads and writes on a pipe blocks

// we must move to a background goroutine else

// all our writes would block forever

go func() {

    // order of defer here is important

    // see: https://stackoverflow.com/a/24720120/6611700

    // make sure gzip stream is closed before the pipe

    // to ensure data is flushed properly

    defer writer.Close()

    var gz = gzip.NewWriter(writer)

    defer gz.Close()


    for cursor.Next(context.Background()) {

        if _, err = io.WriteString(gz, cursor.Current.String()); err != nil {

            // handle error somehow  ˉ\_(ツ)_/ˉ

        }

    }

}()


// you can now use reader as io.Reader

// and it'll contain gzipped data for your serialized rows

_, err = s3.Upload(&s3.UploadInput{

    Bucket: aws.String("..."),

    Key:    aws.String("...")),

    Body:   reader, 

})


查看完整回答
反對(duì) 回復(fù) 2022-06-01
  • 1 回答
  • 0 關(guān)注
  • 156 瀏覽
慕課專欄
更多

添加回答

舉報(bào)

0/150
提交
取消
微信客服

購(gòu)課補(bǔ)貼
聯(lián)系客服咨詢優(yōu)惠詳情

幫助反饋 APP下載

慕課網(wǎng)APP
您的移動(dòng)學(xué)習(xí)伙伴

公眾號(hào)

掃描二維碼
關(guān)注慕課網(wǎng)微信公眾號(hào)