## Issue
Gitea fails to dump LFS (and other object-storage) files when Azure Blob
Storage is configured as the storage backend. The dump reports:
Failed to dump LFS objects: /file/path: copying contents: file does not
exist
This happens with any non-empty base path (the default for LFS storage),
which is why the user could only work around it by using `--skip-*`
flags.
The root cause is in `AzureBlobStorage.IterateObjects()`: Azure's list
API already returns each blob's name including the configured base path,
but the code was building the read client by running that name through
the base-path-prepending helper a second time. This doubled the base
path (e.g. `gitea-lfs/gitea-lfs/aa/bb/hash`), pointing at a blob that
doesn't exist. `Stat()` still succeeded because it doesn't touch the
network, so the failure only surfaced when the dumper actually tried to
read the object's contents.
## Solution
Add `getBlobClientByFullName()`, which builds a blob client from a name
that is already fully qualified, without re-applying
`buildAzureBlobPath()`. `IterateObjects()` now uses it for names
obtained from Azure's list API. `getBlobClient()` (used by `Open`,
`Stat`, `Delete`, `ServeDirectURL`, which take relative paths) is
unchanged in behavior.
Also add `TestAzureBlobStorageDumpArchive`, a regression test that
drives the real dump path (`IterateObjects` → `Stat` →
`dump.Dumper.AddFileByReader` → `mholt/archives` zip writer) against a
**non-empty** `BasePath`, and verifies the produced archive contains the
object with the correct content. The existing Azure tests use an empty
`BasePath` and never read object content via `IterateObjects`, which is
why they didn't catch this.
Fixes#35476
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Use base64.RawURLEncoding to avoid equal sign
* using the nodejs package they seem to get lost
* Support uploads with unspecified length
* Support uploads with a single named blockid
* without requiring a blockmap
---------
Signed-off-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Follow up #22405Fix#20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
minio storage iterator shows different behavior with local fs iterator.
in local fs storage:
``` go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "prefix/xxx.file"
})
```
in minio storage:
```go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "xxx.file"
})
```
I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Since #23493 has conflicts with latest commits, this PR is my proposal
for fixing #23371
Details are in the comments
And refactor the `modules/options` module, to make it always use
"filepath" to access local files.
Benefits:
* No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))),
"/")` any more (not only one before)
* The function behaviors are clearly defined
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Clean paths when looking in Storage
Ensure paths are clean for minio aswell as local storage.
Use url.Path not RequestURI/EscapedPath in storageHandler.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Lauris BH <lauris@nix.lv>
Mitrahsoft