Skip to content

Python support: transpiled Python on every target, Python servers, Pillow and font-from-data - #6128

Open
shai-almog wants to merge 106 commits into
masterfrom
python-support
Open

shai-almog wants to merge 106 commits into
masterfrom
python-support

Conversation

@shai-almog

Copy link
Copy Markdown
Collaborator

Python support for Codename One: Python sources are transpiled to Java and run on every target, client and server.

What is in it

  • maven/python-transpiler and maven/python-runtime: the transpiler, the runtime and the bundled stdlib and site packages, with ~125 conformance programs whose expected output is real CPython output.
  • Build integration in build-engine (Maven goal, Gradle task, project importer, PyPI resolution, C-extension stubs) and the python.virtualThreadStack build hint / @VirtualThreadStack.
  • Python servers on the backend: FastAPI/Starlette/Flask, WebSockets and streaming responses, BackgroundTasks after the response, SQLAlchemy on SQLite, PostgreSQL and MySQL (CI runs it against both), a GIL that cooperates with virtual threads.
  • Pillow: on servers over a new backend image API (com.codename1.backend.image: raster, JPEG/PNG, TrueType text); on clients over Codename One's Image/ImageIO.
  • Core: Font.createTrueTypeFontFromData / isTrueTypeStreamSupported on JavaSE, Android, iOS/Mac, JavaScript, Windows and Linux; Database column types.
  • Backend: MySQL/PostgreSQL sslmode=verify-ca with a CA file (plus the MySQL spellings).
  • Fix: the core inflater (io/gzip/InfCodes) rejected valid deflate streams whose last code ended a few bits into the final byte (about half of all gzip files, depending on the header mtime). Fuzzed against java.util.zip with 160k streams fed in chunks.
  • Developer guide chapters for all of the above.

Verification

  • python-transpiler/python-runtime tests (145), core-unittests, backend tests; SpotBugs/PMD/Checkstyle clean on the touched modules; cast-semantics, native-signature, C++ linkage, control-character and build-hint gates.
  • End-to-end server runs on the JVM and native, and SQLAlchemy against PostgreSQL 16 and MySQL 8.4.
  • Not verified locally: the JavaScript font path in a browser (TrueTypeFontFromDataTest in the JS suite covers it here), the Windows/Linux C font changes (no toolchain locally) and the iOS private-API scan.

🤖 Generated with Claude Code

shai-almog and others added 30 commits October 27, 2026 17:01
Python 3 sources transpile to Java 17 (maven/python-transpiler) and run on
maven/python-runtime on every Codename One target. Generators, async,
match, dataclasses, the builtins and a stdlib wave are covered by a
conformance suite diffed against CPython on the JVM and natively through
ParparVM. Constructs a translated program cannot run (eval, exec,
compile, __import__, frame hooks) are build errors naming file and line.

ParparVM: a frameless method returning a value after several try blocks
that end on one label ran a single END_TRY and left the other blocks on
the thread's try stack, so a later throw longjmp'd into a returned frame.
The folded return paths now unwind the stack (FramelessTryReturnIntegrationTest).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
from java.<package> import <Class> resolves at build time against the
compile class path; a generated JavaInterop bridge dispatches with
instanceof and direct calls (no reflection), converts values both ways,
adapts Python callables to functional interfaces and exposes Java
collections to len/in/iteration/indexing.

cn1:import-python-project and cn1:transpile-python (Maven), the
transpilePython task (Gradle), and the archetype's python profile build
src/main/python. runMain returns the exit status, honouring SystemExit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vice library

from java.java.* resolves against the class path's own class library
(java-runtime, JavaAPI) when it carries one, so an API the device lacks is
a build error rather than a bridge that fails to link.

python-project-test.sh imports scripts/python-samples/hello-ui (a
Codename One UI written in Python), builds it with Maven and Gradle and
stages every target; python-project.yml runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyHttp is the client the packages are written over: ConnectionRequest in
an application, installable like PyFileSystem. requests (verbs, Session,
auth, params/json/form bodies, cookies, exceptions) and httpx (verbs,
Client, AsyncClient, URL, Headers, HTTPStatusError) are our own Python
over it; http_requests and http_httpx match the real packages' output
against the same test server.

BaseException.__new__ ignores keyword arguments, as CPython's does, so an
exception subclass whose __init__ takes keywords can be raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_type_hints

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lice.indices

The HTTP client is installed by the generated registry only when the
program bundles requests or httpx, so other programs do not link the
network stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ule __dict__

Fixes found while porting attrs and pydantic: an annotation built by a call
(Annotated[int, Field(gt=0)], conint(ge=2)) was stored as None; a class
__delattr__ was bypassed by del; classes had no __hash__ attribute; and
type(name, bases, ns) gave __module__ 'builtins'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_cn1server.serve(dispatch, host, port) bridges a Python dispatch callable
to the backend HttpServer when the backend is on the transpile class path;
scripts/python-backend/build.sh builds such an app as a native binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clean-room site packages flask, werkzeug.exceptions, fastapi, starlette
(responses, requests, testclient, status) and uvicorn. Every app exposes
dispatch(method, path, query, headers, body) -> (status, headers, body);
app.run / uvicorn.run hand it to _cn1server.serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions

inspect.isgeneratorfunction/iscoroutinefunction/isasyncgenfunction now
answer from __code__ as CPython's do; FastAPI's yield dependencies use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper returns

CommonWorkloads ported to Python (scripts/python-bench/common) with a
Java hot-spot variant, and a comparison across CPython, PyPy, GraalPy and
Codename One with peak RSS per workload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mance chapter

The developer guide's Python chapter compares CPython, PyPy, GraalPy and
Codename One on startup, kernels, Flask and the CommonWorkloads suite, with
peak memory in every table, two charts, and a section on moving a measured
hot spot into Java.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IntRanges runs interval analysis over each typed function: loop counters
are bounded by their range, accumulators by trip count times their
per-pass growth, and masked values by the mask. A local whose every
value fits in 64 bits becomes a Java long, and an int expression whose
result fits is computed with wrapping long arithmetic, which equals the
exact result however large the intermediates are. Expression helpers
such as i32/i64 are inlined, module int constants are folded, and >> of
a value masked into [0, 2^64) becomes >>>.

Builtins without a typed lowering (print, sorted, ...) are now called
generically from typed code instead of leaving the whole function
untyped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd //, lowered loop tests

Locals that only ever hold a freshly constructed object whose __init__
copies its arguments into fields, and are only read through fields or
getters, become one local per field. while tests lower their arithmetic
and compares like any statement. for c in str walks code points, with
ord(c) the code point itself. l.extend([v] * n) fills in place. Lone
surrogates index and repr as CPython does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped builtins over BOT, dict clear/len intrinsics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… int results held bare, taken in the slow path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t stores per instance, try-free dict key hashing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-top arithmetic over impure calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct clear; direct dict len

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… box the list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shai-almog and others added 18 commits October 28, 2026 15:27
…picks a server or client engine

The pure-Java raster API is no longer client API: RasterImage, RasterGraphics,
PngCodec, JpegCodec, Resampler, TrueTypeFont and the rest move from core to
vm/backend as com.codename1.backend.image, with their tests, the Aileron
license and NOTICE entry. The guide section moves to Backend-Views.

PIL's _cn1image now asks a PyImageEngine for codecs, resampling and text.
A backend build gets a generated BackendImage over the backend package
(Pillow's exact pixels); a client build gets PyImageNative over Image,
ImageIO, Graphics and the native font. python-runtime no longer references
any image library. The pil_ conformance programs build as a server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…--native derives JDK_8_HOME

pilclient_ builds a Pillow program as a client app (no backend on the class
path), so the transpiler installs PyImageNative, and runs it over
compat-testing's HeadlessImplementation with a JDK-backed ImageIO: PNG and
JPEG decode, convert, resize (with box), thumbnail, PNG/JPEG save round
trips, and load_default()/truetype() text metrics, bbox, length and ink.
Assertions are tolerant, so CPython's output is the expected one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m or bytes

New public API Font.createTrueTypeFontFromData(String, InputStream|byte[]) and
Font.isTrueTypeStreamSupported(), backed by the implementation hook
loadTrueTypeFontFromData(String, byte[]). Deliberately not an overload of
createTrueTypeFont: createTrueTypeFont(name, null) is a documented call that a
second two-argument overload would make ambiguous.

Ports: JavaSE (java.awt.Font.createFont), Android (content-hash cache file,
Typeface.createFromFile), iOS/macOS (new IOSNative.registerFontData native:
CGFontCreateWithDataProvider + CTFontManagerRegisterGraphicsFont, PostScript
name), JavaScript (data: URL through the existing worker/host FontFace path),
Windows and Linux (their existing in-memory loaders). UWP untouched.

Pillow client engine: ImageFont.truetype loads through the new API, falling
back to the native font; load_default stays native (Aileron is backend-only).
Tests: core-unittests FontTest, JavaSE headless load/derive/measure of Roboto,
pilclient_ conformance extended. Docs: CSS fonts section and Python Pillow row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…FNT header

The in-memory loaders answer a system family when they cannot register
the bytes; the API contract is null, so the ports check the SFNT tag
(0x00010000, 'true', 'OTTO', 'ttcf') first. Corrects the stale Linux
header comment that said in-memory loading was not wired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… header

FontFace rejects non-font bytes asynchronously, after the port had already
returned a font; check the SFNT tag up front as Linux/Windows do. Adds
TrueTypeFontFromDataTest to the hellocodenameone device suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s report corrupt bytes

AbstractDBCursor takes a column's storage class before any getter reads it and
answers getColumnType from that per-row cache, since sqlite3_column_type is
undefined after a conversion. Font.createTrueTypeFontFromData keys its cache on
name, length and a hash of the bytes (the JavaScript port's browser family too).
iOS registerFontData answers null when registration fails, accepting an
already-registered name only for the same font. Linux and Windows answer 0
(null) for bytes FreeType/DirectWrite cannot load instead of a system family.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…merge, late stack raise refused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld, red-team fixes

PyLock and the generated websocket Socket no longer use monitors, which block
a virtual thread's host; they spin on atomics through Gil.pauseWaiter. The
handshake has a terminal timed-out state, so a late accept or message does
nothing. Gil.yieldNow hands the lock to a waiter instead of retaking it.
Instance __dict__ keeps non-str keys and syncs in place; dict iteration sees
stores made through the object. hasCustomGetattribute is cached per type.
PyServer logs any failing background task and closes a stream that raised;
non-bytes chunks fail as in Starlette. FastAPI keeps a response's own
background. _cn1dbapi skips comments, dollar quotes and escapes when binding
and detecting queries, and parses time offsets and infinity. hmac rejects
non-ASCII str. copyreg is bundled for stdlib modules that need it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orts whether fonts load from memory

AbstractDBCursor captures a column's type before its first read only when the
cursor opts in (capturesColumnTypeBeforeRead, true for the iOS SQLite cursor);
others are asked lazily. SECursor derives the type from the value isNullAt
already fetched instead of fetching it again. Windows isTrueTypeStreamSupported
asks the new isFontMemoryLoadingSupported native whether IDWriteFactory5 exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sslmode as written

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A start that failed after the listener bound (websocket routes, VT hosts)
never decremented the count, so every later stack raise was refused for the
life of the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_cn1dbapi classifies WITH by the statement after the CTE list, follows
MySQL comment rules (-- needs whitespace, #), maps 24:00:00 as psycopg2 does.
Gil.yieldNow waits up to 2ms for a waiter and waiters poll while a handoff
is pending. Instance __dict__ sync re-reads values by index while the shape
and set fields are unchanged. The websocket Socket raises on accept or send
after a refusal or timeout. Cn1DbApiTest runs the driver's pure functions
under CPython; WebSocketSocketTest compiles the generated Socket against a
fake server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instance __dict__: dict-level writes invalidate the sync stamp, the fast path
checks every key against the expected name, popitem syncs first, a store
happens before the dict changes, and a pure-__slots__ instance has no
__dict__. Gil handoff is a counter cleared in finally. The websocket
handshake timeout is per socket. RETURNING inside a CTE no longer makes a
statement a query. Cn1DbApiTest kills a hung python3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tance

JZlib's InfCodes waited for a full table width of bits before decoding a
literal/length or distance code, so a complete raw deflate stream whose last
code ended a few bits into its final byte answered Z_BUF_ERROR. The compress_
conformance program hit it through zipfile: the archived gzip file carries
its mtime, so whether the entry decompressed depended on the wall clock.

Py.dictCount's one-probe path is now taken only for a plain dict; an
instance's __dict__ view goes through get and setitem so the store reaches
the instance. With the flake explained, an instance keeps one __dict__ view,
so o.__dict__ is o.__dict__ as in CPython.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Typed.len read PyDict.size directly, so len(o.__dict__) after an attribute
write answered the count of the last sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sources)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ench jars, Pillow kerning note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Developer Guide build artifacts are available for download from this workflow run:

Developer Guide quality checks:

  • AsciiDoc linter: No issues found (report)
  • Vale: No alerts found (report)
  • Paragraph capitalization: No paragraph capitalization issues (report)
  • LanguageTool: No grammar matches (report)
  • Image references: No unused images detected (report)

shai-almog and others added 2 commits October 29, 2026 04:10
…real)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shai-almog

shai-almog commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Compared 12 screenshots: 12 matched.
✅ JavaSE simulator integration screenshots matched stored baselines.

shai-almog and others added 2 commits October 29, 2026 04:16
…usion the gate rejects

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Flutter vs Codename One

No benchmark results were produced.

shai-almog and others added 2 commits October 29, 2026 05:45
…DK 17 step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked no-try

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Cloudflare Preview

private static java.util.List<int[]> imports(String text, String pkg, String module, TreeSet<String> names) {
java.util.List<int[]> at = new java.util.ArrayList<int[]>();
String leaf = module.substring(module.lastIndexOf('.') + 1);
Matcher m = FROM.matcher(text);
char type = (char) tar[pos + 156];
int data = pos + 512;
if (type == 'x' || type == 'L') {
String body = new String(tar, data, (int) size, StandardCharsets.UTF_8);
String full = longName != null ? longName : prefix.isEmpty() ? name : prefix + "/" + name;
longName = null;
if (safe(full)) {
byte[] b = new byte[(int) size];
longName = null;
if (safe(full)) {
byte[] b = new byte[(int) size];
System.arraycopy(tar, data, b, 0, (int) size);
…pe-smoke gets headroom

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants