[tools, cmake] refactor: update/hash check scripts, use tags for some more deps, proper CPMUtil separation #2666

Merged
crueter merged 19 commits from update-fmt into master 2025-10-05 03:04:54 +02:00
21 changed files with 370 additions and 100 deletions
Showing only changes of commit 9cf7cd6e50 - Show all commits

21
tools/README.md Normal file
View file

@ -0,0 +1,21 @@
# Tools
Tools for Eden and other subprojects.
## Third-Party
- [CPMUtil Scripts](./cpm)
## Eden
- `shellcheck.sh`: Ensure POSIX compliance (and syntax sanity) for all tools in this directory and subdirectories.
- `llvmpipe-run.sh`: Sets environment variables needed to run any command (or Eden) with llvmpipe.
- `optimize-assets.sh`: Optimize PNG assets with OptiPng.
- `update-cpm.sh`: Updates CPM.cmake to the latest version.
- `update-icons.sh`: Rebuild all icons (macOS, Windows, bitmaps) based on the master SVG file (`dist/dev.eden_emu.eden.svg`)
* Also optimizes the master SVG
* Requires: `png2icns` (libicns), ImageMagick, [`svgo`](https://github.com/svg/svgo)
- `dtrace-tool.sh`
- `lanczos_gen.c`
- `clang-format.sh`: Runs `clang-format` on the entire codebase.
* Requires: clang

View file

@ -1,3 +1,6 @@
#! /bin/sh #! /bin/sh
exec find src -iname *.h -o -iname *.cpp | xargs clang-format-15 -i -style=file:src/.clang-format # SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
exec find src -iname "*.h" -o -iname "*.cpp" | xargs clang-format -i -style=file:src/.clang-format

58
tools/cpm/README.md Normal file
View file

@ -0,0 +1,58 @@
# CPMUtil Tools
These are supplemental shell scripts for CPMUtil aiming to ease maintenance burden for sanity checking, updates, prefetching, formatting, and standard operations done by these shell scripts, all in one common place.
All scripts are POSIX-compliant.
## Meta
These scripts are generally reserved for internal use.
- `common.sh`: Grabs all available cpmfiles and aggregates them together.
* Outputs:
- `PACKAGES`: The aggregated cpmfile
- `LIBS`: The list of individual libraries contained within each cpmfile
- `value`: A function that grabs a key from the `JSON` variable (typically the package key)
- `download.sh`: Utility script to handle downloading of regular and CI packages.
* Generally only used by the fetch scripts.
- `package.sh`: The actual package parser.
* Inputs:
- `package`: The package key
* Outputs:
- Basically everything. You're best off reading the code rather than me poorly explaining it.
- `which.sh`: Find which cpmfile a package is located in.
* Inputs:
- The package key
## Simple Utilities
These scripts don't really have any functionality, they just help you out a bit yknow?
- `format.sh`: Format all cpmfiles (4-space indent is enforced)
* In the future, these scripts will have options for spacing
- `hash.sh`: Determine the hash of a specific package.
* Inputs:
- The repository (e.g. fmtlib/fmt)
- The sha or tag (e.g. v1.0.1)
- `GIT_HOST`: What git host to use (default github.com)
- `USE_TAG`: Set to "true" if the second argument is a tag instead of a sha
- `ARTIFACT`: The artifact to download, if using a tag. Set to null or empty to use the tag source archive instead
* Output: the SHA512 sum of the package
- `url-hash.sh`: Determine the hash of a URL
* Input: the URL
* Output: the SHA512 sum of the URL
## Functional Utilities
These modify the CPM cache or cpmfiles. Each allows you to input all the packages to act on, as well as a `<scriptname>-all.sh` that acts upon all available packages.
For the update and hash scripts, set `UPDATE=true` to update the cpmfile with the new version or hash. Beware: if the hash is `cf83e1357...` that means you got a 404 error!
- `fetch.sh`: Prefetch a package according to its cpmfile definition
* Packages are fetched to the `.cache/cpm` directory by default, following the CPMUtil default.
- `check-updates.sh`: Check a package for available updates
* This only applies to packages that utilize tags.
* If the tag is a format string, the `git_version` is acted upon instead.
* Setting `FORCE=true` will forcefully update every package and its hash, even if they are on the latest version (`UPDATE` must also be true)
- `check-hashes.sh`: Check a package's hash
* This only applies to packages with hardcoded hashes, NOT ones that use hash URLs.

View file

@ -1,15 +1,10 @@
#!/bin/bash -e #!/bin/bash -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
SCRIPT_DIR=$(dirname -- "$0")
# shellcheck disable=SC1091 # shellcheck disable=SC1091
. tools/cpm/common.sh . tools/cpm/common.sh
# shellcheck disable=SC2086 # shellcheck disable=SC2086
"$SCRIPT_DIR"/check-hash.sh $LIBS tools/cpm/check-hash.sh $LIBS

View file

@ -1,8 +1,5 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later

View file

@ -1,15 +1,10 @@
#!/bin/bash -e #!/bin/bash -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
SCRIPT_DIR=$(dirname -- "$0")
# shellcheck disable=SC1091 # shellcheck disable=SC1091
. tools/cpm/common.sh . tools/cpm/common.sh
# shellcheck disable=SC2086 # shellcheck disable=SC2086
"$SCRIPT_DIR"/check-updates.sh $LIBS tools/cpm/check-updates.sh $LIBS

View file

@ -25,6 +25,7 @@ do
[ "$REPO" = null ] && continue [ "$REPO" = null ] && continue
[ "$GIT_HOST" != "github.com" ] && continue # TODO [ "$GIT_HOST" != "github.com" ] && continue # TODO
# shellcheck disable=SC2153
[ "$TAG" = null ] && continue [ "$TAG" = null ] && continue
echo "-- Package $package" echo "-- Package $package"
@ -34,6 +35,7 @@ do
TAGS=$(gh api --method GET "/repos/$REPO/tags") TAGS=$(gh api --method GET "/repos/$REPO/tags")
# filter out some commonly known annoyances # filter out some commonly known annoyances
# TODO add more
TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("vulkan-sdk"; "i") | not)]') TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("vulkan-sdk"; "i") | not)]')
TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("yotta"; "i") | not)]') TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("yotta"; "i") | not)]')
@ -42,7 +44,7 @@ do
TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("beta"; "i") | not)]') TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("beta"; "i") | not)]')
TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("rc"; "i") | not)]') TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("rc"; "i") | not)]')
# thanks fmt # Add package-specific overrides here, e.g. here for fmt:
[ "$package" = fmt ] && TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("v0.11"; "i") | not)]') [ "$package" = fmt ] && TAGS=$(echo "$TAGS" | jq '[.[] | select(.name | test("v0.11"; "i") | not)]')
LATEST=$(echo "$TAGS" | jq -r '.[0].name') LATEST=$(echo "$TAGS" | jq -r '.[0].name')

View file

@ -9,6 +9,9 @@
# shellcheck disable=SC2038 # shellcheck disable=SC2038
# shellcheck disable=SC2016 # shellcheck disable=SC2016
[ -z "$PACKAGES" ] && PACKAGES=$(find . src -maxdepth 3 -name cpmfile.json | xargs jq -s 'reduce .[] as $item ({}; . * $item)') [ -z "$PACKAGES" ] && PACKAGES=$(find . src -maxdepth 3 -name cpmfile.json | xargs jq -s 'reduce .[] as $item ({}; . * $item)')
# For your project you'll want to change the PACKAGES call to include whatever locations you may use (externals, src, etc.)
# Always include .
LIBS=$(echo "$PACKAGES" | jq -j 'keys_unsorted | join(" ")') LIBS=$(echo "$PACKAGES" | jq -j 'keys_unsorted | join(" ")')
export PACKAGES export PACKAGES

View file

@ -1,8 +1,5 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later

View file

@ -1,8 +1,5 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later

View file

@ -1,8 +1,5 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later

View file

@ -1,8 +1,12 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# like common.sh, change for your directories
FILES=$(find . src -maxdepth 3 -name cpmfile.json) FILES=$(find . src -maxdepth 3 -name cpmfile.json)
for file in $FILES; do for file in $FILES; do

View file

@ -1,8 +1,5 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
@ -20,30 +17,28 @@ JSON=$(echo "$PACKAGES" | jq -r ".\"$package\" | select( . != null )")
[ -z "$JSON" ] && echo "!! No cpmfile definition for $package" && exit 1 [ -z "$JSON" ] && echo "!! No cpmfile definition for $package" && exit 1
# unset stuff # unset stuff
unset PACKAGE_NAME export PACKAGE_NAME="null"
unset REPO export REPO="null"
unset CI export CI="null"
unset GIT_HOST export GIT_HOST="null"
unset EXT export EXT="null"
unset NAME export NAME="null"
unset DISABLED export DISABLED="null"
unset TAG export TAG="null"
unset ARTIFACT export ARTIFACT="null"
unset SHA export SHA="null"
unset VERSION export VERSION="null"
unset GIT_VERSION export GIT_VERSION="null"
unset DOWNLOAD export DOWNLOAD="null"
unset URL export URL="null"
unset KEY export KEY="null"
unset HASH export HASH="null"
export ORIGINAL_TAG="null"
unset ORIGINAL_TAG export HAS_REPLACE="null"
unset HAS_REPLACE export VERSION_REPLACE="null"
unset VERSION_REPLACE export HASH_URL="null"
export HASH_SUFFIX="null"
unset HASH_URL export HASH_ALGO="null"
unset HASH_SUFFIX
unset HASH_ALGO
######## ########
# Meta # # Meta #
@ -76,30 +71,13 @@ if [ "$CI" = "true" ]; then
NAME=$(value "name") NAME=$(value "name")
DISABLED=$(echo "$JSON" | jq -j '.disabled_platforms') DISABLED=$(echo "$JSON" | jq -j '.disabled_platforms')
[ "$NAME" = null ] && NAME="$PACKAGE" [ "$NAME" = null ] && NAME="$PACKAGE_NAME"
export EXT export EXT
export NAME export NAME
export DISABLED export DISABLED
export VERSION export VERSION
# slight annoyance
TAG=null
ARTIFACT=null
SHA=null
VERSION=null
GIT_VERSION=null
DOWNLOAD=null
URL=null
KEY=null
HASH=null
ORIGINAL_TAG=null
HAS_REPLACE=null
VERSION_REPLACE=null
HASH_URL=null
HASH_SUFFIX=null
HASH_ALGO=null
return 0 return 0
fi fi

View file

@ -1,10 +1,14 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2025 crueter # SPDX-FileCopyrightText: 2025 crueter
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# check which file a package is in # check which file a package is in
# like common.sh, change for your directories
JSON=$(find . src -maxdepth 3 -name cpmfile.json -exec grep -l "$1" {} \;) JSON=$(find . src -maxdepth 3 -name cpmfile.json -exec grep -l "$1" {} \;)
[ -z "$JSON" ] && echo "!! No cpmfile definition for $1" [ -z "$JSON" ] && echo "!! No cpmfile definition for $1"

View file

@ -1,42 +1,59 @@
#!/usr/local/bin/bash -ex #!/usr/local/bin/bash -ex
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project # SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# Basic script to run dtrace sampling over the program (requires Flamegraph) # Basic script to run dtrace sampling over the program (requires Flamegraph)
# Usage is either running as: ./dtrace-tool.sh pid (then input the pid of the process) # Usage is either running as: ./dtrace-tool.sh pid (then input the pid of the process)
# Or just run directly with: ./dtrace-tool.sh <command> # Or just run directly with: ./dtrace-tool.sh <command>
FLAMEGRAPH_DIR=".." FLAMEGRAPH_DIR=".."
function fail { fail() {
printf '%s\n' "$1" >&2 printf '%s\n' "$1" >&2
exit "${2-1}" exit "${2-1}"
} }
[ -f $FLAMEGRAPH_DIR/FlameGraph/stackcollapse.pl ] || fail 'Where is flamegraph?' [ -f $FLAMEGRAPH_DIR/FlameGraph/stackcollapse.pl ] || fail 'Where is flamegraph?'
#[ which dtrace ] || fail 'Needs DTrace installed' #[ which dtrace ] || fail 'Needs DTrace installed'
read -p "Sampling Hz [800]: " TRACE_CFG_HZ
read -r "Sampling Hz [800]: " TRACE_CFG_HZ
if [ -z "${TRACE_CFG_HZ}" ]; then if [ -z "${TRACE_CFG_HZ}" ]; then
TRACE_CFG_HZ=800 TRACE_CFG_HZ=800
fi fi
read -p "Sampling time [5] sec: " TRACE_CFG_TIME
read -r "Sampling time [5] sec: " TRACE_CFG_TIME
if [ -z "${TRACE_CFG_TIME}" ]; then if [ -z "${TRACE_CFG_TIME}" ]; then
TRACE_CFG_TIME=5 TRACE_CFG_TIME=5
fi fi
TRACE_FILE=dtrace-out.user_stacks TRACE_FILE=dtrace-out.user_stacks
TRACE_FOLD=dtrace-out.fold TRACE_FOLD=dtrace-out.fold
TRACE_SVG=dtrace-out.svg TRACE_SVG=dtrace-out.svg
ps ps
if [[ $1 = 'pid' ]]; then
read -p "PID: " TRACE_CFG_PID if [ "$1" = 'pid' ]; then
sudo echo 'Sudo!' read -r "PID: " TRACE_CFG_PID
sudo echo 'Sudo!'
else else
[[ -f $1 && $1 ]] || fail 'Usage: ./tools/dtrace-profile.sh <path to program>' if [ -f "$1" ] && [ "$1" ]; then
echo "Executing: '$@'" fail 'Usage: ./tools/dtrace-profile.sh <path to program>'
sudo echo 'Sudo!' fi
"$@" &
TRACE_CFG_PID=$! printf "Executing: "
echo "$@"
sudo echo 'Sudo!'
"$@" &
TRACE_CFG_PID=$!
fi fi
TRACE_PROBE="profile-${TRACE_CFG_HZ} /pid == ${TRACE_CFG_PID} && arg1/ { @[ustack()] = count(); } tick-${TRACE_CFG_TIME}s { exit(0); }" TRACE_PROBE="profile-${TRACE_CFG_HZ} /pid == ${TRACE_CFG_PID} && arg1/ { @[ustack()] = count(); } tick-${TRACE_CFG_TIME}s { exit(0); }"
rm -- $TRACE_SVG || echo 'Skip' rm -- $TRACE_SVG || echo 'Skip'
sudo dtrace -x ustackframes=100 -Z -n "$TRACE_PROBE" -o $TRACE_FILE 2>/dev/null || exit sudo dtrace -x ustackframes=100 -Z -n "$TRACE_PROBE" -o $TRACE_FILE 2>/dev/null || exit
perl $FLAMEGRAPH_DIR/FlameGraph/stackcollapse.pl $TRACE_FILE > $TRACE_FOLD || exit perl $FLAMEGRAPH_DIR/FlameGraph/stackcollapse.pl $TRACE_FILE > $TRACE_FOLD || exit
perl $FLAMEGRAPH_DIR/FlameGraph/flamegraph.pl $TRACE_FOLD > $TRACE_SVG || exit perl $FLAMEGRAPH_DIR/FlameGraph/flamegraph.pl $TRACE_FOLD > $TRACE_SVG || exit
sudo chmod 0666 $TRACE_FILE sudo chmod 0666 $TRACE_FILE
rm -- $TRACE_FILE $TRACE_FOLD rm -- $TRACE_FILE $TRACE_FOLD

File diff suppressed because one or more lines are too long

View file

@ -1,6 +1,9 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project # SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# Optimizes assets of eden (requires OptiPng)
# Optimizes assets of Eden (requires OptiPng)
which optipng || exit which optipng || exit
find . -type f -name *.png -exec optipng -o7 {} \; find . -type f -name "*.png" -exec optipng -o7 {} \;

View file

@ -1,8 +0,0 @@
#!/bin/bash -ex
# SPDX-FileCopyrightText: 2024 yuzu Emulator Project
# SPDX-License-Identifier: MIT
git submodule sync
git submodule foreach --recursive git reset --hard
git submodule update --init --recursive

11
tools/shellcheck.sh Executable file
View file

@ -0,0 +1,11 @@
#!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
# fd is slightly faster on NVMe (the syntax sux though)
if command -v fd > /dev/null; then
fd . tools -esh -x shellcheck
else
find tools -name "*.sh" -exec shellcheck -s sh {} \;
fi

View file

@ -1,3 +1,6 @@
#!/bin/sh #!/bin/sh -e
wget -O CMakeModules/CPM.cmake https://github.com/cpm-cmake/CPM.cmake/releases/latest/download/get_cpm.cmake # SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later
wget -O CMakeModules/CPM.cmake https://github.com/cpm-cmake/CPM.cmake/releases/latest/download/CPM.cmake

View file

@ -1,8 +1,11 @@
#!/bin/sh -e #!/bin/sh -e
# SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project # SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# Updates main icons for eden # Updates main icons for eden
which png2icns || [ which yay && yay libicns ] || exit
which png2icns || (which yay && yay libicns) || exit
which magick || exit which magick || exit
export EDEN_SVG_ICO="dist/dev.eden_emu.eden.svg" export EDEN_SVG_ICO="dist/dev.eden_emu.eden.svg"