mirror of
https://github.com/go-gitea/gitea.git
synced 2026-08-03 18:24:29 +00:00
Backport #38753 by @Zettat123 ## Background `MinioStorage.Save` is called with `size = -1` on several paths, including Actions logs, Actions artifacts, repository archives, avatars and attachments. With an unknown size (-1) minio-go assumes a 5TiB object and allocates a single part-sized buffer of 528MiB per upload, regardless of the real payload size, which can exhaust the memory of small instances. Measured against a local S3 stub, five sequential uploads of a 4KiB payload grew RSS by 1041MiB before the fix and by 34MiB after it. ## Fix Pass an explicit 16MiB part size in that case, the same value minio-go uses as minimum part size (https://github.com/minio/minio-go/blob/v7.2.1/constants.go#L28). Uploads with a known size are left untouched, since minio-go already derives a part size proportional to the real object size. ## Note: One behaviour change: with an unknown size the object is now limited to 16MiB * 10000 parts = 156.25GiB (https://github.com/minio/minio-go/blob/v7.2.1/api-put-object-common.go#L112-L116). `putObjectMultipartStreamNoLength` completes the upload once it runs out of parts without checking that the reader was drained, so a stream past that limit is silently truncated rather than rejected. The previous limit was 5TiB. No payload Gitea uploads comes close to either. Parts are also uploaded serially, so a smaller part size means proportionally more round trips for large unknown-size uploads. Co-authored-by: Zettat123 <zettat123@gmail.com> Co-authored-by: silverwind <me@silverwind.io>
This commit is contained in:
@@ -26,6 +26,8 @@ import (
|
||||
|
||||
var _ ObjectStorage = &MinioStorage{}
|
||||
|
||||
const unknownSizePartSize = 1024 * 1024 * 16 // same as minio-go's minPartSize
|
||||
|
||||
type minioObject struct {
|
||||
*minio.Object
|
||||
}
|
||||
@@ -211,6 +213,10 @@ func (m *MinioStorage) Save(path string, r io.Reader, size int64) (int64, error)
|
||||
// * https://www.backblaze.com/b2/docs/s3_compatible_api.html
|
||||
// do not support "x-amz-checksum-algorithm" header, so use legacy MD5 checksum
|
||||
SendContentMd5: m.cfg.ChecksumAlgorithm == "md5",
|
||||
|
||||
// with an unknown size (-1) minio-go assumes a 5TiB object and buffers a 528MiB part for it, even
|
||||
// for a payload of a few KiB, so pin the part size there, a known size derives its own
|
||||
PartSize: util.Iif[uint64](size < 0, unknownSizePartSize, 0),
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
|
||||
Reference in New Issue
Block a user