Repository navigation
Python support: transpiled Python on every target, Python servers, Pillow and font-from-data - #6128
Open
shai-almog wants to merge 106 commits into
Open
shai-almog wants to merge 106 commits into
shai-almog wants to merge 106 commits into
Conversation
Python 3 sources transpile to Java 17 (maven/python-transpiler) and run on maven/python-runtime on every Codename One target. Generators, async, match, dataclasses, the builtins and a stdlib wave are covered by a conformance suite diffed against CPython on the JVM and natively through ParparVM. Constructs a translated program cannot run (eval, exec, compile, __import__, frame hooks) are build errors naming file and line. ParparVM: a frameless method returning a value after several try blocks that end on one label ran a single END_TRY and left the other blocks on the thread's try stack, so a later throw longjmp'd into a returned frame. The folded return paths now unwind the stack (FramelessTryReturnIntegrationTest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
from java.<package> import <Class> resolves at build time against the compile class path; a generated JavaInterop bridge dispatches with instanceof and direct calls (no reflection), converts values both ways, adapts Python callables to functional interfaces and exposes Java collections to len/in/iteration/indexing. cn1:import-python-project and cn1:transpile-python (Maven), the transpilePython task (Gradle), and the archetype's python profile build src/main/python. runMain returns the exit status, honouring SystemExit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vice library from java.java.* resolves against the class path's own class library (java-runtime, JavaAPI) when it carries one, so an API the device lacks is a build error rather than a bridge that fails to link. python-project-test.sh imports scripts/python-samples/hello-ui (a Codename One UI written in Python), builds it with Maven and Gradle and stages every target; python-project.yml runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyHttp is the client the packages are written over: ConnectionRequest in an application, installable like PyFileSystem. requests (verbs, Session, auth, params/json/form bodies, cookies, exceptions) and httpx (verbs, Client, AsyncClient, URL, Headers, HTTPStatusError) are our own Python over it; http_requests and http_httpx match the real packages' output against the same test server. BaseException.__new__ ignores keyword arguments, as CPython's does, so an exception subclass whose __init__ takes keywords can be raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_type_hints Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lice.indices The HTTP client is installed by the generated registry only when the program bundles requests or httpx, so other programs do not link the network stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ule __dict__ Fixes found while porting attrs and pydantic: an annotation built by a call (Annotated[int, Field(gt=0)], conint(ge=2)) was stored as None; a class __delattr__ was bypassed by del; classes had no __hash__ attribute; and type(name, bases, ns) gave __module__ 'builtins'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_cn1server.serve(dispatch, host, port) bridges a Python dispatch callable to the backend HttpServer when the backend is on the transpile class path; scripts/python-backend/build.sh builds such an app as a native binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clean-room site packages flask, werkzeug.exceptions, fastapi, starlette (responses, requests, testclient, status) and uvicorn. Every app exposes dispatch(method, path, query, headers, body) -> (status, headers, body); app.run / uvicorn.run hand it to _cn1server.serve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions inspect.isgeneratorfunction/iscoroutinefunction/isasyncgenfunction now answer from __code__ as CPython's do; FastAPI's yield dependencies use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper returns CommonWorkloads ported to Python (scripts/python-bench/common) with a Java hot-spot variant, and a comparison across CPython, PyPy, GraalPy and Codename One with peak RSS per workload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mance chapter The developer guide's Python chapter compares CPython, PyPy, GraalPy and Codename One on startup, kernels, Flask and the CommonWorkloads suite, with peak memory in every table, two charts, and a section on moving a measured hot spot into Java. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IntRanges runs interval analysis over each typed function: loop counters are bounded by their range, accumulators by trip count times their per-pass growth, and masked values by the mask. A local whose every value fits in 64 bits becomes a Java long, and an int expression whose result fits is computed with wrapping long arithmetic, which equals the exact result however large the intermediates are. Expression helpers such as i32/i64 are inlined, module int constants are folded, and >> of a value masked into [0, 2^64) becomes >>>. Builtins without a typed lowering (print, sorted, ...) are now called generically from typed code instead of leaving the whole function untyped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd //, lowered loop tests Locals that only ever hold a freshly constructed object whose __init__ copies its arguments into fields, and are only read through fields or getters, become one local per field. while tests lower their arithmetic and compares like any statement. for c in str walks code points, with ord(c) the code point itself. l.extend([v] * n) fills in place. Lone surrogates index and repr as CPython does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped builtins over BOT, dict clear/len intrinsics Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… int results held bare, taken in the slow path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t stores per instance, try-free dict key hashing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-top arithmetic over impure calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct clear; direct dict len Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… box the list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…picks a server or client engine The pure-Java raster API is no longer client API: RasterImage, RasterGraphics, PngCodec, JpegCodec, Resampler, TrueTypeFont and the rest move from core to vm/backend as com.codename1.backend.image, with their tests, the Aileron license and NOTICE entry. The guide section moves to Backend-Views. PIL's _cn1image now asks a PyImageEngine for codecs, resampling and text. A backend build gets a generated BackendImage over the backend package (Pillow's exact pixels); a client build gets PyImageNative over Image, ImageIO, Graphics and the native font. python-runtime no longer references any image library. The pil_ conformance programs build as a server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…--native derives JDK_8_HOME pilclient_ builds a Pillow program as a client app (no backend on the class path), so the transpiler installs PyImageNative, and runs it over compat-testing's HeadlessImplementation with a JDK-backed ImageIO: PNG and JPEG decode, convert, resize (with box), thumbnail, PNG/JPEG save round trips, and load_default()/truetype() text metrics, bbox, length and ink. Assertions are tolerant, so CPython's output is the expected one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m or bytes New public API Font.createTrueTypeFontFromData(String, InputStream|byte[]) and Font.isTrueTypeStreamSupported(), backed by the implementation hook loadTrueTypeFontFromData(String, byte[]). Deliberately not an overload of createTrueTypeFont: createTrueTypeFont(name, null) is a documented call that a second two-argument overload would make ambiguous. Ports: JavaSE (java.awt.Font.createFont), Android (content-hash cache file, Typeface.createFromFile), iOS/macOS (new IOSNative.registerFontData native: CGFontCreateWithDataProvider + CTFontManagerRegisterGraphicsFont, PostScript name), JavaScript (data: URL through the existing worker/host FontFace path), Windows and Linux (their existing in-memory loaders). UWP untouched. Pillow client engine: ImageFont.truetype loads through the new API, falling back to the native font; load_default stays native (Aileron is backend-only). Tests: core-unittests FontTest, JavaSE headless load/derive/measure of Roboto, pilclient_ conformance extended. Docs: CSS fonts section and Python Pillow row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…FNT header The in-memory loaders answer a system family when they cannot register the bytes; the API contract is null, so the ports check the SFNT tag (0x00010000, 'true', 'OTTO', 'ttcf') first. Corrects the stale Linux header comment that said in-memory loading was not wired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… header FontFace rejects non-font bytes asynchronously, after the port had already returned a font; check the SFNT tag up front as Linux/Windows do. Adds TrueTypeFontFromDataTest to the hellocodenameone device suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s report corrupt bytes AbstractDBCursor takes a column's storage class before any getter reads it and answers getColumnType from that per-row cache, since sqlite3_column_type is undefined after a conversion. Font.createTrueTypeFontFromData keys its cache on name, length and a hash of the bytes (the JavaScript port's browser family too). iOS registerFontData answers null when registration fails, accepting an already-registered name only for the same font. Linux and Windows answer 0 (null) for bytes FreeType/DirectWrite cannot load instead of a system family. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…merge, late stack raise refused Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld, red-team fixes PyLock and the generated websocket Socket no longer use monitors, which block a virtual thread's host; they spin on atomics through Gil.pauseWaiter. The handshake has a terminal timed-out state, so a late accept or message does nothing. Gil.yieldNow hands the lock to a waiter instead of retaking it. Instance __dict__ keeps non-str keys and syncs in place; dict iteration sees stores made through the object. hasCustomGetattribute is cached per type. PyServer logs any failing background task and closes a stream that raised; non-bytes chunks fail as in Starlette. FastAPI keeps a response's own background. _cn1dbapi skips comments, dollar quotes and escapes when binding and detecting queries, and parses time offsets and infinity. hmac rejects non-ASCII str. copyreg is bundled for stdlib modules that need it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orts whether fonts load from memory AbstractDBCursor captures a column's type before its first read only when the cursor opts in (capturesColumnTypeBeforeRead, true for the iOS SQLite cursor); others are asked lazily. SECursor derives the type from the value isNullAt already fetched instead of fetching it again. Windows isTrueTypeStreamSupported asks the new isFontMemoryLoadingSupported native whether IDWriteFactory5 exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sslmode as written Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A start that failed after the listener bound (websocket routes, VT hosts) never decremented the count, so every later stack raise was refused for the life of the process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_cn1dbapi classifies WITH by the statement after the CTE list, follows MySQL comment rules (-- needs whitespace, #), maps 24:00:00 as psycopg2 does. Gil.yieldNow waits up to 2ms for a waiter and waiters poll while a handoff is pending. Instance __dict__ sync re-reads values by index while the shape and set fields are unchanged. The websocket Socket raises on accept or send after a refusal or timeout. Cn1DbApiTest runs the driver's pure functions under CPython; WebSocketSocketTest compiles the generated Socket against a fake server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instance __dict__: dict-level writes invalidate the sync stamp, the fast path checks every key against the expected name, popitem syncs first, a store happens before the dict changes, and a pure-__slots__ instance has no __dict__. Gil handoff is a counter cleared in finally. The websocket handshake timeout is per socket. RETURNING inside a CTE no longer makes a statement a query. Cn1DbApiTest kills a hung python3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tance JZlib's InfCodes waited for a full table width of bits before decoding a literal/length or distance code, so a complete raw deflate stream whose last code ended a few bits into its final byte answered Z_BUF_ERROR. The compress_ conformance program hit it through zipfile: the archived gzip file carries its mtime, so whether the entry decompressed depended on the wall clock. Py.dictCount's one-probe path is now taken only for a plain dict; an instance's __dict__ view goes through get and setitem so the store reaches the instance. With the flake explained, an instance keeps one __dict__ view, so o.__dict__ is o.__dict__ as in CPython. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Typed.len read PyDict.size directly, so len(o.__dict__) after an attribute write answered the count of the last sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sources) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/pr.yml
…ench jars, Pillow kerning note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Developer Guide build artifacts are available for download from this workflow run:
Developer Guide quality checks: |
…real) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Compared 12 screenshots: 12 matched. |
…usion the gate rejects Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Flutter vs Codename OneNo benchmark results were produced. |
…DK 17 step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked no-try Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Cloudflare Preview
|
| private static java.util.List<int[]> imports(String text, String pkg, String module, TreeSet<String> names) { | ||
| java.util.List<int[]> at = new java.util.ArrayList<int[]>(); | ||
| String leaf = module.substring(module.lastIndexOf('.') + 1); | ||
| Matcher m = FROM.matcher(text); |
| char type = (char) tar[pos + 156]; | ||
| int data = pos + 512; | ||
| if (type == 'x' || type == 'L') { | ||
| String body = new String(tar, data, (int) size, StandardCharsets.UTF_8); |
| String full = longName != null ? longName : prefix.isEmpty() ? name : prefix + "/" + name; | ||
| longName = null; | ||
| if (safe(full)) { | ||
| byte[] b = new byte[(int) size]; |
| longName = null; | ||
| if (safe(full)) { | ||
| byte[] b = new byte[(int) size]; | ||
| System.arraycopy(tar, data, b, 0, (int) size); |
…pe-smoke gets headroom Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python support for Codename One: Python sources are transpiled to Java and run on every target, client and server.
What is in it
maven/python-transpilerandmaven/python-runtime: the transpiler, the runtime and the bundled stdlib and site packages, with ~125 conformance programs whose expected output is real CPython output.build-engine(Maven goal, Gradle task, project importer, PyPI resolution, C-extension stubs) and thepython.virtualThreadStackbuild hint /@VirtualThreadStack.com.codename1.backend.image: raster, JPEG/PNG, TrueType text); on clients over Codename One'sImage/ImageIO.Font.createTrueTypeFontFromData/isTrueTypeStreamSupportedon JavaSE, Android, iOS/Mac, JavaScript, Windows and Linux;Databasecolumn types.sslmode=verify-cawith a CA file (plus the MySQL spellings).io/gzip/InfCodes) rejected valid deflate streams whose last code ended a few bits into the final byte (about half of all gzip files, depending on the header mtime). Fuzzed againstjava.util.zipwith 160k streams fed in chunks.Verification
TrueTypeFontFromDataTestin the JS suite covers it here), the Windows/Linux C font changes (no toolchain locally) and the iOS private-API scan.🤖 Generated with Claude Code