gh-116622: Add Android testbed (GH-117878)

Add code and config for a minimal Android app, and instructions to build and run it.
Improve Android build instructions in general.
Add a tool subcommand to download the Gradle wrapper (with its binary blob). Android
studio must be downloaded manually (due to the license).
This commit is contained in:
Malcolm Smith 2024-05-01 07:36:45 +01:00 committed by GitHub
parent 21336aa127
commit 2520eed0a5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
19 changed files with 570 additions and 10 deletions

12
.github/CODEOWNERS vendored
View File

@ -243,6 +243,18 @@ Lib/test/support/interpreters/ @ericsnowcurrently
Modules/_xx*interp*module.c @ericsnowcurrently
Lib/test/test_interpreters/ @ericsnowcurrently
# Android
**/*Android* @mhsmith
**/*android* @mhsmith
# iOS (but not termios)
**/iOS* @freakboy3742
**/ios* @freakboy3742
**/*_iOS* @freakboy3742
**/*_ios* @freakboy3742
**/*-iOS* @freakboy3742
**/*-ios* @freakboy3742
# WebAssembly
/Tools/wasm/ @brettcannon

View File

@ -22,12 +22,25 @@ you don't already have the SDK, here's how to install it:
`android-sdk/cmdline-tools/latest`.
* `export ANDROID_HOME=/path/to/android-sdk`
The `android.py` script also requires the following commands to be on the `PATH`:
* `curl`
* `java`
* `tar`
* `unzip`
## Building
Building for Android requires doing a cross-build where you have a "build"
Python to help produce an Android build of CPython. This procedure has been
tested on Linux and macOS.
Python can be built for Android on any POSIX platform supported by the Android
development tools, which currently means Linux or macOS. This involves doing a
cross-build where you use a "build" Python (for your development machine) to
help produce a "host" Python for Android.
First, make sure you have all the usual tools and libraries needed to build
Python for your development machine. The only Android tool you need to install
is the command line tools package above: the build script will download the
rest.
The easiest way to do a build is to use the `android.py` script. You can either
have it perform the entire build process from start to finish in one step, or
@ -43,9 +56,10 @@ The discrete steps for building via `android.py` are:
./android.py make-host HOST
```
To see the possible values of HOST, run `./android.py configure-host --help`.
`HOST` identifies which architecture to build. To see the possible values, run
`./android.py configure-host --help`.
Or to do it all in a single command, run:
To do all steps in a single command, run:
```sh
./android.py build HOST
@ -62,3 +76,22 @@ call. For example, if you want a pydebug build that also caches the results from
```sh
./android.py build HOST -- -C --with-pydebug
```
## Testing
To run the Python test suite on Android:
* Install Android Studio, if you don't already have it.
* Follow the instructions in the previous section to build all supported
architectures.
* Run `./android.py setup-testbed` to download the Gradle wrapper.
* Open the `testbed` directory in Android Studio.
* In the *Device Manager* dock, connect a device or start an emulator.
Then select it from the drop-down list in the toolbar.
* Click the "Run" button in the toolbar.
* The testbed app displays nothing on screen while running. To see its output,
open the [Logcat window](https://developer.android.com/studio/debug/logcat).
To run specific tests, or pass any other arguments to the test suite, edit the
command line in testbed/app/src/main/python/main.py.

View File

@ -7,8 +7,9 @@ import shutil
import subprocess
import sys
import sysconfig
from os.path import relpath
from os.path import basename, relpath
from pathlib import Path
from tempfile import TemporaryDirectory
SCRIPT_NAME = Path(__file__).name
CHECKOUT = Path(__file__).resolve().parent.parent
@ -102,11 +103,17 @@ def unpack_deps(host):
for name_ver in ["bzip2-1.0.8-1", "libffi-3.4.4-2", "openssl-3.0.13-1",
"sqlite-3.45.1-0", "xz-5.4.6-0"]:
filename = f"{name_ver}-{host}.tar.gz"
run(["wget", f"{deps_url}/{name_ver}/{filename}"])
download(f"{deps_url}/{name_ver}/{filename}")
run(["tar", "-xf", filename])
os.remove(filename)
def download(url, target_dir="."):
out_path = f"{target_dir}/{basename(url)}"
run(["curl", "-Lf", "-o", out_path, url])
return out_path
def configure_host_python(context):
host_dir = subdir(context.host, clean=context.clean)
@ -160,6 +167,30 @@ def clean_all(context):
delete_if_exists(CROSS_BUILD_DIR)
# To avoid distributing compiled artifacts without corresponding source code,
# the Gradle wrapper is not included in the CPython repository. Instead, we
# extract it from the Gradle release.
def setup_testbed(context):
ver_long = "8.7.0"
ver_short = ver_long.removesuffix(".0")
testbed_dir = CHECKOUT / "Android/testbed"
for filename in ["gradlew", "gradlew.bat"]:
out_path = download(
f"https://raw.githubusercontent.com/gradle/gradle/v{ver_long}/{filename}",
testbed_dir)
os.chmod(out_path, 0o755)
with TemporaryDirectory(prefix=SCRIPT_NAME) as temp_dir:
os.chdir(temp_dir)
bin_zip = download(
f"https://services.gradle.org/distributions/gradle-{ver_short}-bin.zip")
outer_jar = f"gradle-{ver_short}/lib/plugins/gradle-wrapper-{ver_short}.jar"
run(["unzip", bin_zip, outer_jar])
run(["unzip", "-o", "-d", f"{testbed_dir}/gradle/wrapper", outer_jar,
"gradle-wrapper.jar"])
def main():
parser = argparse.ArgumentParser()
subcommands = parser.add_subparsers(dest="subcommand")
@ -173,8 +204,11 @@ def main():
help="Run `configure` for Android")
make_host = subcommands.add_parser("make-host",
help="Run `make` for Android")
clean = subcommands.add_parser("clean", help="Delete files and directories "
"created by this script")
subcommands.add_parser(
"clean", help="Delete the cross-build directory")
subcommands.add_parser(
"setup-testbed", help="Download the testbed Gradle wrapper")
for subcommand in build, configure_build, configure_host:
subcommand.add_argument(
"--clean", action="store_true", default=False, dest="clean",
@ -194,7 +228,8 @@ def main():
"configure-host": configure_host_python,
"make-host": make_host_python,
"build": build_all,
"clean": clean_all}
"clean": clean_all,
"setup-testbed": setup_testbed}
dispatch[context.subcommand](context)

21
Android/testbed/.gitignore vendored Normal file
View File

@ -0,0 +1,21 @@
# The Gradle wrapper should be downloaded by running `../android.py setup-testbed`.
/gradlew
/gradlew.bat
/gradle/wrapper/gradle-wrapper.jar
*.iml
.gradle
/local.properties
/.idea/caches
/.idea/deploymentTargetDropdown.xml
/.idea/libraries
/.idea/modules.xml
/.idea/workspace.xml
/.idea/navEditor.xml
/.idea/assetWizardSettings.xml
.DS_Store
/build
/captures
.externalNativeBuild
.cxx
local.properties

1
Android/testbed/app/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
/build

View File

@ -0,0 +1,129 @@
import com.android.build.api.variant.*
plugins {
id("com.android.application")
id("org.jetbrains.kotlin.android")
}
val PYTHON_DIR = File(projectDir, "../../..").canonicalPath
val PYTHON_CROSS_DIR = "$PYTHON_DIR/cross-build"
val ABIS = mapOf(
"arm64-v8a" to "aarch64-linux-android",
"x86_64" to "x86_64-linux-android",
)
val PYTHON_VERSION = File("$PYTHON_DIR/Include/patchlevel.h").useLines {
for (line in it) {
val match = """#define PY_VERSION\s+"(\d+\.\d+)""".toRegex().find(line)
if (match != null) {
return@useLines match.groupValues[1]
}
}
throw GradleException("Failed to find Python version")
}
android {
namespace = "org.python.testbed"
compileSdk = 34
defaultConfig {
applicationId = "org.python.testbed"
minSdk = 21
targetSdk = 34
versionCode = 1
versionName = "1.0"
ndk.abiFilters.addAll(ABIS.keys)
externalNativeBuild.cmake.arguments(
"-DPYTHON_CROSS_DIR=$PYTHON_CROSS_DIR",
"-DPYTHON_VERSION=$PYTHON_VERSION")
}
externalNativeBuild.cmake {
path("src/main/c/CMakeLists.txt")
}
// Set this property to something non-empty, otherwise it'll use the default
// list, which ignores asset directories beginning with an underscore.
aaptOptions.ignoreAssetsPattern = ".git"
compileOptions {
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
}
kotlinOptions {
jvmTarget = "1.8"
}
}
dependencies {
implementation("androidx.appcompat:appcompat:1.6.1")
implementation("com.google.android.material:material:1.11.0")
implementation("androidx.constraintlayout:constraintlayout:2.1.4")
}
// Create some custom tasks to copy Python and its standard library from
// elsewhere in the repository.
androidComponents.onVariants { variant ->
generateTask(variant, variant.sources.assets!!) {
into("python") {
for (triplet in ABIS.values) {
for (subDir in listOf("include", "lib")) {
into(subDir) {
from("$PYTHON_CROSS_DIR/$triplet/prefix/$subDir")
include("python$PYTHON_VERSION/**")
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
}
}
into("lib/python$PYTHON_VERSION") {
// Uncomment this to pick up edits from the source directory
// without having to rerun `make install`.
// from("$PYTHON_DIR/Lib")
// duplicatesStrategy = DuplicatesStrategy.INCLUDE
into("site-packages") {
from("$projectDir/src/main/python")
}
}
}
exclude("**/__pycache__")
}
generateTask(variant, variant.sources.jniLibs!!) {
for ((abi, triplet) in ABIS.entries) {
into(abi) {
from("$PYTHON_CROSS_DIR/$triplet/prefix/lib")
include("libpython*.*.so")
include("lib*_python.so")
}
}
}
}
fun generateTask(
variant: ApplicationVariant, directories: SourceDirectories,
configure: GenerateTask.() -> Unit
) {
val taskName = "generate" +
listOf(variant.name, "Python", directories.name)
.map { it.replaceFirstChar(Char::uppercase) }
.joinToString("")
directories.addGeneratedSourceDirectory(
tasks.register<GenerateTask>(taskName) {
into(outputDir)
configure()
},
GenerateTask::outputDir)
}
// addGeneratedSourceDirectory requires the task to have a DirectoryProperty.
abstract class GenerateTask: Sync() {
@get:OutputDirectory
abstract val outputDir: DirectoryProperty
}

View File

@ -0,0 +1,20 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET"/>
<application
android:icon="@drawable/ic_launcher"
android:label="@string/app_name"
android:theme="@style/Theme.Material3.Light.NoActionBar">
<activity
android:name=".MainActivity"
android:exported="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>

View File

@ -0,0 +1,9 @@
cmake_minimum_required(VERSION 3.4.1)
project(testbed)
set(PREFIX_DIR ${PYTHON_CROSS_DIR}/${CMAKE_LIBRARY_ARCHITECTURE}/prefix)
include_directories(${PREFIX_DIR}/include/python${PYTHON_VERSION})
link_directories(${PREFIX_DIR}/lib)
link_libraries(log python${PYTHON_VERSION})
add_library(main_activity SHARED main_activity.c)

View File

@ -0,0 +1,147 @@
#include <android/log.h>
#include <errno.h>
#include <jni.h>
#include <pthread.h>
#include <Python.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
static void throw_runtime_exception(JNIEnv *env, const char *message) {
(*env)->ThrowNew(
env,
(*env)->FindClass(env, "java/lang/RuntimeException"),
message);
}
// --- Stdio redirection ------------------------------------------------------
// Most apps won't need this, because the Python-level sys.stdout and sys.stderr
// are redirected to the Android logcat by Python itself. However, in the
// testbed it's useful to redirect the native streams as well, to debug problems
// in the Python startup or redirection process.
//
// Based on
// https://github.com/beeware/briefcase-android-gradle-template/blob/v0.3.11/%7B%7B%20cookiecutter.safe_formal_name%20%7D%7D/app/src/main/cpp/native-lib.cpp
typedef struct {
FILE *file;
int fd;
android_LogPriority priority;
char *tag;
int pipe[2];
} StreamInfo;
static StreamInfo STREAMS[] = {
{stdout, STDOUT_FILENO, ANDROID_LOG_INFO, "native.stdout", {-1, -1}},
{stderr, STDERR_FILENO, ANDROID_LOG_WARN, "native.stderr", {-1, -1}},
{NULL, -1, ANDROID_LOG_UNKNOWN, NULL, {-1, -1}},
};
// The maximum length of a log message in bytes, including the level marker and
// tag, is defined as LOGGER_ENTRY_MAX_PAYLOAD in
// platform/system/logging/liblog/include/log/log.h. As of API level 30, messages
// longer than this will be be truncated by logcat. This limit has already been
// reduced at least once in the history of Android (from 4076 to 4068 between API
// level 23 and 26), so leave some headroom.
static const int MAX_BYTES_PER_WRITE = 4000;
static void *redirection_thread(void *arg) {
StreamInfo *si = (StreamInfo*)arg;
ssize_t read_size;
char buf[MAX_BYTES_PER_WRITE];
while ((read_size = read(si->pipe[0], buf, sizeof buf - 1)) > 0) {
buf[read_size] = '\0'; /* add null-terminator */
__android_log_write(si->priority, si->tag, buf);
}
return 0;
}
static char *redirect_stream(StreamInfo *si) {
/* make the FILE unbuffered, to ensure messages are never lost */
if (setvbuf(si->file, 0, _IONBF, 0)) {
return "setvbuf";
}
/* create the pipe and redirect the file descriptor */
if (pipe(si->pipe)) {
return "pipe";
}
if (dup2(si->pipe[1], si->fd) == -1) {
return "dup2";
}
/* start the logging thread */
pthread_t thr;
if ((errno = pthread_create(&thr, 0, redirection_thread, si))) {
return "pthread_create";
}
if ((errno = pthread_detach(thr))) {
return "pthread_detach";
}
return 0;
}
JNIEXPORT void JNICALL Java_org_python_testbed_MainActivity_redirectStdioToLogcat(
JNIEnv *env, jobject obj
) {
for (StreamInfo *si = STREAMS; si->file; si++) {
char *error_prefix;
if ((error_prefix = redirect_stream(si))) {
char error_message[1024];
snprintf(error_message, sizeof(error_message),
"%s: %s", error_prefix, strerror(errno));
throw_runtime_exception(env, error_message);
return;
}
}
}
// --- Python intialization ----------------------------------------------------
static PyStatus set_config_string(
JNIEnv *env, PyConfig *config, wchar_t **config_str, jstring value
) {
const char *value_utf8 = (*env)->GetStringUTFChars(env, value, NULL);
PyStatus status = PyConfig_SetBytesString(config, config_str, value_utf8);
(*env)->ReleaseStringUTFChars(env, value, value_utf8);
return status;
}
static void throw_status(JNIEnv *env, PyStatus status) {
throw_runtime_exception(env, status.err_msg ? status.err_msg : "");
}
JNIEXPORT void JNICALL Java_org_python_testbed_MainActivity_runPython(
JNIEnv *env, jobject obj, jstring home, jstring runModule
) {
PyConfig config;
PyStatus status;
PyConfig_InitIsolatedConfig(&config);
status = set_config_string(env, &config, &config.home, home);
if (PyStatus_Exception(status)) {
throw_status(env, status);
return;
}
status = set_config_string(env, &config, &config.run_module, runModule);
if (PyStatus_Exception(status)) {
throw_status(env, status);
return;
}
// Some tests generate SIGPIPE and SIGXFSZ, which should be ignored.
config.install_signal_handlers = 1;
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
throw_status(env, status);
return;
}
Py_RunMain();
}

View File

@ -0,0 +1,61 @@
package org.python.testbed
import android.os.*
import android.system.Os
import android.widget.TextView
import androidx.appcompat.app.*
import java.io.*
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
// Python needs this variable to help it find the temporary directory,
// but Android only sets it on API level 33 and later.
Os.setenv("TMPDIR", cacheDir.toString(), false)
val pythonHome = extractAssets()
System.loadLibrary("main_activity")
redirectStdioToLogcat()
runPython(pythonHome.toString(), "main")
findViewById<TextView>(R.id.tvHello).text = "Python complete"
}
private fun extractAssets() : File {
val pythonHome = File(filesDir, "python")
if (pythonHome.exists() && !pythonHome.deleteRecursively()) {
throw RuntimeException("Failed to delete $pythonHome")
}
extractAssetDir("python", filesDir)
return pythonHome
}
private fun extractAssetDir(path: String, targetDir: File) {
val names = assets.list(path)
?: throw RuntimeException("Failed to list $path")
val targetSubdir = File(targetDir, path)
if (!targetSubdir.mkdirs()) {
throw RuntimeException("Failed to create $targetSubdir")
}
for (name in names) {
val subPath = "$path/$name"
val input: InputStream
try {
input = assets.open(subPath)
} catch (e: FileNotFoundException) {
extractAssetDir(subPath, targetDir)
continue
}
input.use {
File(targetSubdir, name).outputStream().use { output ->
input.copyTo(output)
}
}
}
}
private external fun redirectStdioToLogcat()
private external fun runPython(home: String, runModule: String)
}

View File

@ -0,0 +1,17 @@
import runpy
import signal
import sys
# Some tests use SIGUSR1, but that's blocked by default in an Android app in
# order to make it available to `sigwait` in the "Signal Catcher" thread. That
# thread's functionality is only relevant to the JVM ("forcing GC (no HPROF) and
# profile save"), so disabling it should not weaken the tests.
signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGUSR1])
# To run specific tests, or pass any other arguments to the test suite, edit
# this command line.
sys.argv[1:] = [
"--use", "all,-cpu",
"--verbose3",
]
runpy.run_module("test")

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.0 KiB

View File

@ -0,0 +1,19 @@
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<TextView
android:id="@+id/tvHello"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Hello World!"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>

View File

@ -0,0 +1,3 @@
<resources>
<string name="app_name">Python testbed</string>
</resources>

View File

@ -0,0 +1,5 @@
// Top-level build file where you can add configuration options common to all sub-projects/modules.
plugins {
id("com.android.application") version "8.2.2" apply false
id("org.jetbrains.kotlin.android") version "1.9.22" apply false
}

View File

@ -0,0 +1,23 @@
# Project-wide Gradle settings.
# IDE (e.g. Android Studio) users:
# Gradle settings configured through the IDE *will override*
# any settings specified in this file.
# For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true
# AndroidX package structure to make it clearer which packages are bundled with the
# Android operating system, and which are packaged with your app's APK
# https://developer.android.com/topic/libraries/support-library/androidx-rn
android.useAndroidX=true
# Kotlin code style for this project: "official" or "obsolete":
kotlin.code.style=official
# Enables namespacing of each library's R class so that its R class includes only the
# resources declared in the library itself and none from the library's dependencies,
# thereby reducing the size of the R class for that library
android.nonTransitiveRClass=true

View File

@ -0,0 +1,6 @@
#Mon Feb 19 20:29:06 GMT 2024
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

View File

@ -0,0 +1,18 @@
pluginManagement {
repositories {
google()
mavenCentral()
gradlePluginPortal()
}
}
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
google()
mavenCentral()
}
}
rootProject.name = "Python testbed"
include(":app")

View File

@ -0,0 +1 @@
A testbed project was added to run the test suite on Android.