[cpm, tools]: Ability to override and set custom hosts (like a self hosted instance/git-daemon) #2837

Open
Lizzie wants to merge 3 commits from selfhost0 into master
Member

Signed-off-by: lizzie lizzie@eden-emu.dev

Signed-off-by: lizzie <lizzie@eden-emu.dev>
[cpm]: Ability to override and set custom hosts (like a self hosted instance or whatever)
Some checks failed
eden-license / license-header (pull_request) Successful in 31s
GitHub Actions [CI] Build failed
f140354a46
Signed-off-by: lizzie <lizzie@eden-emu.dev>
@ -14,2 +14,4 @@
"Force system packages for all CPM dependencies (NOT RECOMMENDED)" OFF)
option(CPMUTIL_DEFAULT_HOST
"Sets the default host when 'git_host' isn't defined" "github.com")
Owner

https://cmake.org/cmake/help/latest/command/set.html#cache-legacy

set(CPMUTIL_DEFAULT_HOST github.com CACHE STRING "Sets the default host when no custom Git host isn't defined")

Though I already have this and a few other changes stashed in CPMUtil, so I will push that whenever it's ready

https://cmake.org/cmake/help/latest/command/set.html#cache-legacy `set(CPMUTIL_DEFAULT_HOST github.com CACHE STRING "Sets the default host when no custom Git host isn't defined")` Though I already have this and a few other changes stashed in CPMUtil, so I will push that whenever it's ready
crueter marked this conversation as resolved
tools/README.md Outdated
@ -33,1 +33,4 @@
- [Translation Scripts](./translations)
## Mirror
- [Mirroring scripts](./mirror)
Owner

Prefer newlines after headings, some Markdown viewers need it

Prefer newlines after headings, some Markdown viewers need it
crueter marked this conversation as resolved
@ -0,0 +12,4 @@
echo "1 - Do initial clone/world"
echo "2 - Clone submodules (update)"
echo "3 - Force manual remotes update (please use crontab instead)"
read -p "Selection? [none]: " ANSWER
Owner

interactive CLIs are not the greatest. Instead we should use getopt-like parsing

https://git.crueter.xyz/scripts/windev/src/branch/master/install.sh#L379

interactive CLIs are not the greatest. Instead we should use getopt-like parsing https://git.crueter.xyz/scripts/windev/src/branch/master/install.sh#L379
Owner

Also, this is less relevant for a getopt-like solution, but prefer cat-EOF heredocs unless there are a ton of variables/command outputs

Also, this is less relevant for a getopt-like solution, but prefer cat-EOF heredocs unless there are a ton of variables/command outputs
crueter marked this conversation as resolved
@ -0,0 +20,4 @@
REPOS=$(find "$CPM_DIR" "$CPM_DIR/src" -maxdepth 3 -type f -name 'cpmfile.json' -not -path 'build' \
| sort | uniq | xargs jq -s 'reduce .[] as $item ({}; . * $item)' \
| jq -r 'reduce .[] as $i (""; . + (if $i.git_host == null then "https://github.com" else "https://" + $i.git_host end) + "/" + $i.repo + " ")' \
| tr ' ' '\n' | xargs -I {} echo {})
Owner

tools/cpm/common.sh will aggregate everything for you

`tools/cpm/common.sh` will aggregate everything for you
crueter marked this conversation as resolved
@ -0,0 +31,4 @@
cd $TARGET_ROOT_DIR && find . -maxdepth 2 -type d -name '*.git' -print0 \
| xargs -0 -I {} sh -c ". $CPM_DIR/tools/mirror/common.sh && git_retrieve_gitmodules {}" \
| while IFS= read -r line; do
NAME=$(echo "$line" | awk -F'/' '{print $5}')
Owner

cut is preferred here. Awk is generally reserved for more complex cases, since cut is faster

`cut` is preferred here. Awk is generally reserved for more complex cases, since cut is faster
crueter marked this conversation as resolved
@ -0,0 +38,4 @@
done
sudo chmod 755 $TARGET_ROOT_DIR
elif [ "$ANSWER" = "3" ]; then
echo Forcing git updates
Owner

echo always prefers quotes

echo always prefers quotes
crueter marked this conversation as resolved
better
Some checks failed
eden-license / license-header (pull_request) Successful in 28s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
build.yml / better (pull_request) Failing after 0s
trigger_release.yml / better (pull_request) Failing after 0s
fb08764372
Signed-off-by: lizzie <lizzie@eden-emu.dev>
@ -0,0 +3,4 @@
# SPDX-License-Identifier: GPL-3.0-or-later
# You must run this at the root of the eden git repo
CPM_DIR=$PWD
NPROC=`nproc`
Owner

This will not work on OpenBSD, macOS and a few others

This will not work on OpenBSD, macOS and a few others
crueter marked this conversation as resolved
@ -0,0 +29,4 @@
case "$1" in
--path)
shift
SCM_ROOT_DIR=$1
Owner

This will have unintended behavior if the user specifies path afterwards. Prefer what I did in windev/install.sh; set environment variables to determine which ops to do, then parse them later. This also lets you force ordering and such

This will have unintended behavior if the user specifies path afterwards. Prefer what I did in windev/install.sh; set environment variables to determine which ops to do, then parse them later. This also lets you force ordering and such
crueter marked this conversation as resolved
crueter added this to the 0.0.4 (real) milestone 2025-10-27 08:00:30 +01:00
crueter removed this from the 0.0.4 (real) milestone 2025-10-27 19:01:03 +01:00
Lizzie force-pushed selfhost0 from fb08764372
Some checks failed
eden-license / license-header (pull_request) Successful in 28s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
build.yml / better (pull_request) Failing after 0s
trigger_release.yml / better (pull_request) Failing after 0s
to af65ea9a1e
All checks were successful
eden-license / license-header (pull_request) Successful in 32s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
2025-10-29 07:13:52 +01:00
Compare
Lizzie force-pushed selfhost0 from af65ea9a1e
All checks were successful
eden-license / license-header (pull_request) Successful in 32s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
to 9662b005d8
All checks were successful
eden-license / license-header (pull_request) Successful in 35s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
2025-11-07 04:26:56 +01:00
Compare
Owner

This has the slightly unfortunate side effect of not working exactly as expected, since CPM specifically works with source archives, releases, and tags, not necessarily raw commits from a Git repo--if there is a cgit-like service that has this functionality though, it might work

This has the slightly unfortunate side effect of not working exactly as expected, since CPM specifically works with source archives, releases, and tags, not necessarily raw commits from a Git repo--if there is a cgit-like service that has this functionality though, it might work
Author
Member

@crueter wrote in #2837 (comment):

This has the slightly unfortunate side effect of not working exactly as expected, since CPM specifically works with source archives, releases, and tags, not necessarily raw commits from a Git repo--if there is a cgit-like service that has this functionality though, it might work

git itself can make a headless git server; cgit is just a frontend - I expect the mirror-er to know that, atleast

cgit offers tarballs (for tags only) btw

<cgit-url>/<path-to-repo>/snapshot/<repo name>-<tag>.tar.gz

http://example.com/OpenTTD/OpenTTD.git/snapshot/OpenTTD-15.0-beta3.tar.gz

@crueter wrote in https://git.eden-emu.dev/eden-emu/eden/pulls/2837#issuecomment-10346: > This has the slightly unfortunate side effect of not working exactly as expected, since CPM specifically works with source archives, releases, and tags, not necessarily raw commits from a Git repo--if there is a cgit-like service that has this functionality though, it might work git itself can make a headless git server; cgit is just a frontend - I expect the mirror-er to know that, atleast cgit offers tarballs (for tags only) btw `<cgit-url>/<path-to-repo>/snapshot/<repo name>-<tag>.tar.gz` `http://example.com/OpenTTD/OpenTTD.git/snapshot/OpenTTD-15.0-beta3.tar.gz`
Owner

That isn't sufficient, because we need general "commitish" archives; that is, commits, tags, and (theoretically) branches. Not to mention releases. Gentoo's gitweb has something similar, e.g. https://gitweb.gentoo.org/repo/proj/guru.git/snapshot/guru-278b5cb7e54fb4f0a1169042fa3559c5811da581.tar.gz, but that still isn't touching releases.

That isn't sufficient, because we need general "commitish" archives; that is, commits, tags, and (theoretically) branches. Not to mention releases. Gentoo's gitweb has something similar, e.g. https://gitweb.gentoo.org/repo/proj/guru.git/snapshot/guru-278b5cb7e54fb4f0a1169042fa3559c5811da581.tar.gz, but that still isn't touching releases.
Lizzie force-pushed selfhost0 from 9662b005d8
All checks were successful
eden-license / license-header (pull_request) Successful in 35s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
to 4da9c6303b
All checks were successful
eden-license / license-header (pull_request) Successful in 36s
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
2025-11-10 01:31:05 +01:00
Compare
All checks were successful
eden-license / license-header (pull_request) Successful in 36s
Required
Details
GitHub Actions [CI] Build succeeded
GitHub Releases [CD] Build succeeded – Release published
This pull request doesn't have enough approvals yet. 0 of 2 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin selfhost0:selfhost0
git switch selfhost0
Sign in to join this conversation.
No description provided.