Technik (Deploy)

Translation deployment

Die aktuelle Fassung steht auf How to Translate FOLIO#Translationdeployment.

Eine veraltete Kopie, die hier nicht aktualisiert wird:

Translations are moved from Lokalise to the /translations/ directories of GitHub repositories by a script that Peter Murray runs every few days. They show up as a pull request on each stripes-* and ui-* repository, are run through the CI tests, and automatically merged. CI tests don't always pass and may require manual restart.
This is a one-way process. Any manual change to locale files on GitHub will be overwritten and are lost when new locale files from Lokalise come in.
https://folio-testing.aws.indexdata.com/ is the first reference environment where new or changed translations appear because it uses the master branch. Other reference environments take longer because they only use released front-end module versions, see Reference Environments for details.
Any FOLIO release like Honeysuckle, Iris, etc. ships with the translations that come with each front-end module. Each front-end module has a different release date (Honeysuckle example). This is the time when translations are taken from module's GitHub translations directory and is different for each front-end module. A new front-end module release is needed to get more recent translation included into a FOLIO release.
However, the sysOp of a FOLIO installation may manually update the translations of their installation by exchanging the files in the /translations/ directory from where the static files are shipped. One option is to take the latest translations from https://folio-testing.aws.indexdata.com/ , use the browser developer tools to find out the current number folio-testing uses, for example https://folio-testing.dev.folio.org/translations/de-1605513539114.json, take the .json file, and rename the .json filename to match the number of the own installation. If the own installation has front-end modules that not exist on folio-testing then merge the folio-testing into the own translation file: jq -s '.[0] * .[1]' de-1601343197440.json de-1605513539114.json > de-1601343197440.json.new

(Planung): Lokalise-Zuständigkeiten und unzugewiesene aus Github aktualisieren

Bei Lokalise ist es sehr aufwendig, herauszufinden, welche neuen Übersetzungsdateien hinzugekommen sind und bei welchen alten Dateien neue Begriffe hinzugekommen oder alte weggefallen sind. Eine Möglichkeit dies zu verbessern, wäre, dass bei "Translation deployment" oben beschriebene Verfahren zu kopieren. Man holt also gelegentlich aus Github die Json-Dateien aus den Translation-Ordnern und guck, was sich seit dem letzten Abholen geändert hat und erzeugt darauf die Liste für Zuständigkeiten., statt sie ziemlich aufwendig von Hand zu pflegen.

Die Scripts unten sind erstmals 2024-07-10 eingestellt und zur Anschauung. Da ist sicher Optimierungspotential.

Schritt 1: Json-Dateien holen

fetch-folio-reps.sh

#!/bin/bash

# Set the GitHub organization name
ORG="folio-org"

# Set the API endpoint
API_URL="https://api.github.com/orgs/$ORG/repos"

# Set the number of items per page (max 100)
PER_PAGE=100

# Initialize page number
PAGE=1

# Initialize an empty array to store filtered repository names
declare -a FILTERED_REPOS

while true; do
    # Make the API request with pagination
    RESPONSE=$(curl -s "${API_URL}?page=${PAGE}&per_page=${PER_PAGE}")

    # Extract repository names from the response, filter, and add to array
    while read -r repo; do
        if [[ $repo == mod-* || $repo == stripes-* || $repo == ui-* ]]; then
            FILTERED_REPOS+=("$repo")
        fi
    done < <(echo "$RESPONSE" | jq -r '.[].name')

    # If less than PER_PAGE repositories are returned, we've reached the end
    if [ $(echo "$RESPONSE" | jq '. | length') -lt $PER_PAGE ]; then
        break
    fi

    # Move to the next page
    ((PAGE++))
done

# Sort the filtered repositories alphabetically
IFS=$'\n' sorted=($(sort <<< "${FILTERED_REPOS[*]}"))
unset IFS

# Print all filtered and sorted repository names
printf '%s\n' "${sorted[@]}"

# Print the total number of filtered repositories
echo "Total filtered repositories: ${#sorted[@]}"

# Create a directory to store the downloaded files
mkdir -p translations

# Download en.json and de.json for each repository
for repo in "${sorted[@]}"; do
    echo "Downloading translations for $repo"

    # Download en.json
    en_url=https://raw.githubusercontent.com/folio-org/$repo/master/translations/$repo/en.json
    de_url=https://raw.githubusercontent.com/folio-org/$repo/master/translations/$repo/de.json
    if curl --output /dev/null --silent --head --fail "$en_url"; then
        # Download en.json
        curl -s "$en_url" -o "translations/${repo}_en.json"
    else
        echo "File does not exist or cannot be accessed."
    fi

    # Download de.json
    if curl --output /dev/null --silent --head --fail "$en_url"; then
        curl -s "$de_url" -o "translations/${repo}_de.json"
        echo "File downloaded successfully."
    else
        echo "File does not exist or cannot be accessed."
    fi
done

echo "Download complete. Files are stored in the 'translations' directory."

Schritt 2: Welche Schlüssel in de.json sind nicht übersetzt (also nur in en.json vorhanden): oder: welche ganzen Zeilen (key+value) sind identisch, d.h. nicht übersetzt

compare-keys.sh

#!/bin/bash

# Directory containing the translation files
TRANSLATIONS_DIR="translations"

# Function to extract keys from a JSON file
extract_keys() {
    jq -r 'keys[]' "$1" | sort
}

# Function to count keys in a JSON file
count_keys() {
    jq 'keys | length' "$1"
}

# Function to compare two sets of keys
compare_keys() {
    local en_file="$1"
    local de_file="$2"
    local repo_name="$3"

    local out_de_missing=""
    local out_de_extra=""
    local out_all_match=""
    local out_untranslated=""

    # Extract keys
    en_keys=$(extract_keys "$en_file")
    de_keys=$(extract_keys "$de_file")

    # Count total entries in English file
    en_total=$(count_keys "$en_file")
    de_total=$(count_keys "$de_file")

    # Find keys in en.json but not in de.json
    missing_in_de=$(comm -23 <(echo "$en_keys") <(echo "$de_keys"))
    missing_count=$(echo "$missing_in_de" | grep -c '^')

    # Find keys in de.json but not in en.json
    extra_in_de=$(comm -13 <(echo "$en_keys") <(echo "$de_keys"))
    extra_count=$(echo "$extra_in_de" | grep -c '^')

    # Find whole matching lines in the de and en files (= not translated)
    untranslated_de=$(comm -12 <(cat "$en_file" | sort | awk '{gsub(/^ +| +$/,"")} {print $0}') <(cat "$de_file" | sort | awk '{gsub(/^ +| +$/,"")} {print  $0}') | grep -v '^[\{\}]$' )
    untranslated_count=$(echo "$untranslated_de" | grep -c '^')

    # Report findings
    out_head=$(echo -e "Comparing keys for $repo_name: Total entries in file EN: $en_total / DE: $de_total\n================================== \n")

    if [ -n "$missing_in_de" ]; then
        out_de_missing=$(echo -e "\n--- Keys in en.json but missing in de.json ($missing_count): ---\n $missing_in_de\n")
    fi

    if [ -n "$extra_in_de" ]; then
        out_de_extra=$(echo -e "\n--- Keys in de.json but not in en.json ($extra_count): ---\n $extra_in_de\n")
    fi

    if [ -z "$missing_in_de" ] && [ -z "$extra_in_de" ]; then
        out_all_match=$(echo -e "\n--- All keys (!) match between en.json and de.json ---\n")
    fi

    if [ "$untranslated_count" -gt "1" ]; then
        out_untranslated=$(echo -e "\n\n--- These strings are not translated to de yet ($untranslated_count): ---\n $untranslated_de \n\n")
    fi

    # output the final result (modify it to your likings)
    echo -e "$out_head $out_de_missing $out_de_extra $out_all_match $out_untranslated\n"
}

# Main script
echo "My Compares" > ./compares.txt
for en_file in "$TRANSLATIONS_DIR"/*_en.json; do
    # Extract repository name from filename
    repo_name=$(basename "$en_file" _en.json)
    de_file="${en_file%_en.json}_de.json"


    # Check if both files exist
    if [ -f "$en_file" ] && [ -f "$de_file" ]; then
        compare_keys "$en_file" "$de_file" "$repo_name" >> ./compares.txt
    else
        echo "Missing translation file for $repo_name" >> ./compares.txt
        echo >> ./compares.txt
    fi
done

Schritt 3: Zuständigkeiten formatiert ausgeben

Die Liste unter Zuständigkeiten nun in 3 Spalten mit den Mods und UIs ausgeben (Übersetzungsdatei, Name Zuständigkeit, Fehlende Übersetzung). Eine CSV mit halb-manuell gepflegter "Übersetzungsdatei, Name Zuständigkeit" muss es natürlich geben. Das fehlt hier noch.

D-A-CH SIG