Doc updates (#385)

* Documentation updates

* Documentation updates

* Update README.md

* Fix tuple indexing error messages

* Update roadmap, differences

* Update README, FAQ

* Trim newline

* Update README.md

* Update README.md

* Update README.md

* Update roadmap.md

* Update cpp.md

* Update README.md

* Update roadmap.md

* Update README.md

* Fix test

* clang-format

* Fix exporting function named "main"

* Update export test

* Fix paths

* Rename extra/python -> jit

* Update license change date

* Minor docs updates

* Re-add __init__.py

* Update header

* Update gitignore

* Update README.md

---------

Co-authored-by: Ibrahim Numanagić <ibrahimpasa@gmail.com>
Co-authored-by: Ibrahim Numanagić <inumanag@users.noreply.github.com>
pull/392/head v0.16.1
A. R. Shajii 2023-05-23 17:59:26 -04:00 committed by GitHub
parent 5085dae04d
commit 38e08b409a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
25 changed files with 253 additions and 57 deletions

12
.gitignore vendored
View File

@ -58,14 +58,16 @@ Thumbs.db
.mypy_cache
.vscode
.cache
extra/jupyter/share/jupyter/kernels/codon/kernel.json
extra/python/codon/version.py
scratch*.*
_*
.ipynb_checkpoints
# CMake generated files #
#########################
jupyter/share/jupyter/kernels/codon/kernel.json
jit/codon/version.py
# Testing files #
#################
temp/
playground/
scratch*.*
_*

View File

@ -8,7 +8,7 @@ set(CODON_JIT_PYTHON_VERSION "0.1.5")
configure_file("${PROJECT_SOURCE_DIR}/cmake/config.h.in"
"${PROJECT_SOURCE_DIR}/codon/config/config.h")
configure_file("${PROJECT_SOURCE_DIR}/cmake/config.py.in"
"${PROJECT_SOURCE_DIR}/extra/python/codon/version.py")
"${PROJECT_SOURCE_DIR}/jit/codon/version.py")
option(CODON_GPU "build Codon GPU backend" OFF)
@ -465,5 +465,5 @@ install(FILES ${CMAKE_BINARY_DIR}/libomp${CMAKE_SHARED_LIBRARY_SUFFIX} DESTINATI
install(TARGETS codon DESTINATION bin)
install(DIRECTORY ${CMAKE_BINARY_DIR}/include/codon DESTINATION include)
install(DIRECTORY ${CMAKE_SOURCE_DIR}/stdlib DESTINATION lib/codon)
install(DIRECTORY ${CMAKE_SOURCE_DIR}/extra/python/ DESTINATION python)
install(DIRECTORY ${CMAKE_SOURCE_DIR}/jit/ DESTINATION python)
install(DIRECTORY DESTINATION lib/codon/plugins)

View File

@ -14,7 +14,7 @@ Licensed Work: Codon compiler, runtime, and standard library
Additional Use Grant: None
Change Date: 2025-11-01
Change Date: 2026-05-01
Change License: Apache License, Version 2.0

View File

@ -4,15 +4,15 @@
<h3 align="center">
<a href="https://docs.exaloop.io/codon" target="_blank"><b>Docs</b></a>
&nbsp;&#65372;&nbsp;
&nbsp;&#183;&nbsp;
<a href="https://docs.exaloop.io/codon/general/faq" target="_blank"><b>FAQ</b></a>
&nbsp;&#65372;&nbsp;
&nbsp;&#183;&nbsp;
<a href="https://blog.exaloop.io" target="_blank"><b>Blog</b></a>
&nbsp;&#65372;&nbsp;
<a href="https://github.com/exaloop/codon/discussions" target="_blank"><b>Forum</b></a>
&nbsp;&#65372;&nbsp;
&nbsp;&#183;&nbsp;
<a href="https://join.slack.com/t/exaloop/shared_invite/zt-1jusa4kc0-T3rRWrrHDk_iZ1dMS8s0JQ" target="_blank">Chat</a>
&nbsp;&#65372;&nbsp;
&nbsp;&#183;&nbsp;
<a href="https://docs.exaloop.io/codon/general/roadmap" target="_blank">Roadmap</a>
&nbsp;&#183;&nbsp;
<a href="https://exaloop.io/benchmarks" target="_blank">Benchmarks</a>
</h3>
@ -23,10 +23,36 @@
## What is Codon?
Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead.
Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with
(and sometimes better than) that of C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many
times higher still. Codon grew out of the [Seq project](https://github.com/seq-lang/seq).
Codon is a high-performance Python implementation that compiles to native machine code without
any runtime overhead. Typical speedups over vanilla Python are on the order of 10-100x or more, on
a single thread. Codon's performance is typically on par with (and sometimes better than) that of
C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many times
higher still.
*Think of Codon as Python reimagined for static, ahead-of-time compilation, built from the ground
up with best possible performance in mind.*
### Goals
- :bulb: **No learning curve:** Be as close to CPython as possible in terms of syntax, semantics and libraries
- :rocket: **Top-notch performance:** At *least* on par with low-level languages like C, C++ or Rust
- :computer: **Hardware support:** Full, seamless support for multicore programming, multithreading (no GIL!), GPU and more
- :chart_with_upwards_trend: **Optimizations:** Comprehensive optimization framework that can target high-level Python constructs
and libraries
- :battery: **Interoperability:** Full interoperability with Python's ecosystem of packages and libraries
### Non-goals
- :x: *Drop-in replacement for CPython:* Codon is not a drop-in replacement for CPython. There are some
aspects of Python that are not suitable for static compilation — we don't support these in Codon.
There are ways to use Codon in larger Python codebases via its [JIT decorator](https://docs.exaloop.io/codon/interoperability/decorator)
or [Python extension backend](https://docs.exaloop.io/codon/interoperability/pyext). Codon also supports
calling any Python module via its [Python interoperability](https://docs.exaloop.io/codon/interoperability/python).
See also [*"Differences with Python"*](https://docs.exaloop.io/codon/general/differences) in the docs.
- :x: *New syntax and language constructs:* We try to avoid adding new syntax, keywords or other language
features as much as possible. While Codon does add some new syntax in a couple places (e.g. to express
parallelism), we try to make it as familiar and intuitive as possible.
## Install
@ -76,9 +102,21 @@ codon build -release -llvm fib.py
See [the docs](https://docs.exaloop.io/codon/general/intro) for more options and examples.
This prime counting example showcases Codon's [OpenMP](https://www.openmp.org/) support, enabled with the addition of one line.
The `@par` annotation tells the compiler to parallelize the following `for`-loop, in this case using a dynamic schedule, chunk size
of 100, and 16 threads.
You can import and use any Python package from Codon. For example:
```python
from python import matplotlib.pyplot as plt
data = [x**2 for x in range(10)]
plt.plot(data)
plt.show()
```
(Just remember to set the `CODON_PYTHON` environment variable to the CPython shared library,
as explained in the [the docs](https://docs.exaloop.io/codon/interoperability/python).)
This prime counting example showcases Codon's [OpenMP](https://www.openmp.org/) support, enabled
with the addition of one line. The `@par` annotation tells the compiler to parallelize the
following `for`-loop, in this case using a dynamic schedule, chunk size of 100, and 16 threads.
```python
from sys import argv
@ -133,16 +171,6 @@ mandelbrot(pixels, grid=(N*N)//1024, block=1024)
GPU programming can also be done using the `@par` syntax with `@par(gpu=True)`.
## What isn't Codon?
While Codon supports nearly all of Python's syntax, it is not a drop-in replacement, and large codebases might require modifications
to be run through the Codon compiler. For example, some of Python's modules are not yet implemented within Codon, and a few of Python's
dynamic features are disallowed. The Codon compiler produces detailed error messages to help identify and resolve any incompatibilities.
Codon can be used within larger Python codebases via the [`@codon.jit` decorator](https://docs.exaloop.io/codon/interoperability/decorator).
Plain Python functions and libraries can also be called from within Codon via
[Python interoperability](https://docs.exaloop.io/codon/interoperability/python).
## Documentation
Please see [docs.exaloop.io](https://docs.exaloop.io/codon) for in-depth documentation.

View File

@ -26,6 +26,9 @@ const std::string EXPORT_ATTR = "std.internal.attributes.export";
const std::string INLINE_ATTR = "std.internal.attributes.inline";
const std::string NOINLINE_ATTR = "std.internal.attributes.noinline";
const std::string GPU_KERNEL_ATTR = "std.gpu.kernel";
const std::string MAIN_UNCLASH = ".main.unclash";
const std::string MAIN_CTOR = ".main.ctor";
} // namespace
llvm::DIFile *LLVMVisitor::DebugInfo::getFile(const std::string &path) {
@ -426,18 +429,24 @@ void executeCommand(const std::vector<std::string> &args) {
void LLVMVisitor::setupGlobalCtorForSharedLibrary() {
const std::string llvmCtor = "llvm.global_ctors";
auto *main = M->getFunction("main");
if (M->getNamedValue(llvmCtor) || !main)
if (M->getNamedValue(llvmCtor))
return;
main->setName(".main"); // avoid clash with other main
auto *main = M->getFunction(MAIN_UNCLASH);
if (!main) {
main = M->getFunction("main");
if (!main)
return;
main->setName(MAIN_UNCLASH); // avoid clash with other main
}
auto *ctorFuncTy = llvm::FunctionType::get(B->getVoidTy(), {}, /*isVarArg=*/false);
auto *ctorEntryTy = llvm::StructType::get(B->getInt32Ty(), ctorFuncTy->getPointerTo(),
B->getInt8PtrTy());
auto *ctorArrayTy = llvm::ArrayType::get(ctorEntryTy, 1);
auto *ctor = cast<llvm::Function>(
M->getOrInsertFunction(".main.ctor", ctorFuncTy).getCallee());
auto *ctor =
cast<llvm::Function>(M->getOrInsertFunction(MAIN_CTOR, ctorFuncTy).getCallee());
ctor->setLinkage(llvm::GlobalValue::InternalLinkage);
auto *entry = llvm::BasicBlock::Create(*context, "entry", ctor);
B->SetInsertPoint(entry);
@ -1122,7 +1131,7 @@ void LLVMVisitor::writeToPythonExtension(const PyModule &pymod,
B->SetInsertPoint(block);
if (auto *main = M->getFunction("main")) {
main->setName(".main");
main->setName(MAIN_UNCLASH);
B->CreateCall({main->getFunctionType(), main}, {zero32, null});
}
@ -1461,8 +1470,10 @@ void LLVMVisitor::visit(const Module *x) {
auto *strlenFunc = llvm::cast<llvm::Function>(
M->getOrInsertFunction("strlen", B->getInt64Ty(), B->getInt8PtrTy()).getCallee());
// check if main exists already as an exported function
const std::string mainName = M->getFunction("main") ? MAIN_UNCLASH : "main";
auto *canonicalMainFunc = llvm::cast<llvm::Function>(
M->getOrInsertFunction("main", B->getInt32Ty(), B->getInt32Ty(),
M->getOrInsertFunction(mainName, B->getInt32Ty(), B->getInt32Ty(),
B->getInt8PtrTy()->getPointerTo())
.getCallee());

View File

@ -73,8 +73,7 @@ template <typename KV> struct GCMapAllocator : public std::allocator<KV> {
GCMapAllocator() = default;
GCMapAllocator(GCMapAllocator<KV> const &) = default;
template <typename KV1>
GCMapAllocator(const GCMapAllocator<KV1>&) noexcept {}
template <typename KV1> GCMapAllocator(const GCMapAllocator<KV1> &) noexcept {}
KV *allocate(std::size_t n) { return (KV *)seq_alloc(n * sizeof(KV)); }

View File

@ -8,6 +8,7 @@
* [Frequently asked questions](intro/faq.md)
* [Differences with Python](intro/differences.md)
* [Release notes](intro/releases.md)
* [Roadmap](intro/roadmap.md)
## Language

View File

@ -309,8 +309,8 @@ When subclassing nodes other than types (e.g. instructions, flows, etc.), be sur
# Utilities
The `codon/ir/util/` directory has a number of utility and generally helpful functions, for things like
cloning IR, inlining/outlining, matching and more. `codon/ir/util/irtools.h` in particular has many helpful
The `codon/cir/util/` directory has a number of utility and generally helpful functions, for things like
cloning IR, inlining/outlining, matching and more. `codon/cir/util/irtools.h` in particular has many helpful
functions for performing various common tasks. If you're working with CIR, be sure to take a look at these
functions to make your life easier!

View File

@ -15,16 +15,19 @@ can create a shared library containing `foo` (assuming source file
*foo.codon*):
``` bash
codon build -o libfoo.so foo.codon
codon build --relocation-model=pic --lib -o libfoo.so foo.codon
```
Now we can call `foo` from a C program:
Now we can call `foo` from a C program (if you're using C++, mark the
Codon function as `extern "C"`):
``` c
#include <stdint.h>
#include <stdio.h>
int64_t foo(int64_t);
// In C++, it would be:
// extern "C" int64_t foo(int64_t);
int main() {
printf("%llu\n", foo(10));
@ -34,12 +37,19 @@ int main() {
Compile:
``` bash
gcc -o foo -L. -lfoo foo.c
gcc -o foo -L. -lfoo foo.c # or g++ if using C++
```
Now running `./foo` should invoke `foo()` as defined in Codon, with an
argument of `10`.
Note that if the generated shared library is in a non-standard path, you
can either:
- Add the `rpath` to the `gcc` command: `-Wl,-rpath=/path/to/lib/dir`
- Add the library path to `LD_LIBRARY_PATH` (or `DYLD_LIBRARY_PATH` if
using macOS): `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/lib/dir`.
# Converting types
The following table shows the conversions between Codon and C/C++ types:
@ -51,5 +61,5 @@ The following table shows the conversions between Codon and C/C++ types:
| `bool` | `bool` |
| `byte` | `char` or `int8_t` |
| `str` | `{int64_t, char*}` (length and data) |
| `tuple` | Struct of fields |
| `class` | Pointer to corresponding tuple |
| `@tuple` | Struct of fields |

View File

@ -1,9 +1,12 @@
While Codon's syntax and semantics are virtually identical
While Codon's syntax and semantics are nearly identical
to Python's, there are some notable differences that are
worth mentioning. Most of these design decisions were made
with the trade-off between performance and Python compatibility
in mind.
Please see our [roadmap](roadmap.md) for more information about
how we plan to close some of these gaps in the future.
# Data types
- **Integers:** Codon's `int` is a 64-bit signed integer,
@ -17,6 +20,10 @@ in mind.
- **Dictionaries:** Codon's dictionary type does not preserve
insertion order, unlike Python's as of 3.6.
- **Tuples**: Since tuples compile down to structs, tuple lengths
must be known at compile time, meaning you can't convert an
arbitrarily-sized list to a tuple, for instance.
# Type checking
Since Codon performs static type checking ahead of time, a

View File

@ -16,7 +16,22 @@ codebases might require modifications to be run through the Codon compiler. For
of Python's modules are not yet implemented within Codon, and a few of Python's dynamic features
are disallowed. The Codon compiler produces detailed error messages to help identify and resolve
any incompatibilities. Codon supports seamless [Python interoperability](../interop/python.md) to
handle cases where specific Python libraries or dynamism are required.
handle cases where specific Python libraries or dynamism are required, and also supports writing
[Python extension modules](../interop/pyext.md) that can be imported and used from larger Python
codebases.
## Why Codon?
Python is arguably the world's most popular programming language, and is gradually becoming the
*lingua franca* particularly amongst non-technical or non-CS practitioners in numerous fields.
It provides a readable, clean syntax, is easy to learn, and has an unmatched ecosystem of libraries.
However, Python's achilles heel has always been performance: a typical codebase in pure Python is
orders of magnitude slower than its C/C++/Rust counterpart.
Codon bridges the gap between Python's simplicity and ease-of-use, and the performance of low-level
languages like C++ or Rust, by using [novel compiler and type checking techniques](https://dl.acm.org/doi/abs/10.1145/3578360.3580275)
to statically compile code ahead-of-time, avoiding all of vanilla Python's runtime overhead and
performance drawbacks.
## How does Codon compare to...
@ -48,6 +63,12 @@ handle cases where specific Python libraries or dynamism are required.
Codon type checks the entire program ahead of time. Codon also tries to circumvent the learning
curve of a new language by adopting Python's syntax and semantics.
- **Mojo?** Mojo strives to add low-level programming support/features to the Python language,
while also supporting the rest of Python by relying on CPython. By contrast, Codon aims to
make Python itself more performant by using new type checking and compilation techniques,
without trying to be a superset or drop-in replacement. Codon tries to minimize new syntax
and language features with respect to Python.
You can see results from [Codon's benchmark suite](https://github.com/exaloop/codon/tree/develop/bench)
suite at [exaloop.io/benchmarks](https://exaloop.io/benchmarks).
More benchmarks can be found in the [2019 paper](https://dl.acm.org/doi/10.1145/3360551)
@ -61,6 +82,9 @@ which can be used within Python codebases. This will compile only the annotated
and automatically handle data conversions to and from Codon. It also allows for
the use of any Codon-specific modules or extensions, such as multithreading.
Codon can also [compile to Python extension modules](../interop/pyext.md) that can be
imported and used from Python.
## What about interoperability with other languages and frameworks?
Interoperability is and will continue to be a priority for Codon.

View File

@ -0,0 +1,110 @@
Codon's goal is to be as close to CPython as possible while still
being fully statically compilable. While Codon already supports
much of Python, there is still much to be done to fully realize
its potential. Here is a high-level roadmap of the things we want
to add, implement or explore.
# Core features
- Type system improvements:
- First-class types and compile-time metaclasses
- Full class member deduction
- Implicit union types to support mixed-type collections
- Variadic type arguments (e.g. `Foo[Bar, ...]`)
- Parallelism
- `async`/`await` support
- `multiprocessing` support
- Automatic locking in parallel code (e.g. if mutating a
data structure shared between threads)
- Race detection
- Compatibility with Python 3.10+:
- Argument separators (`/` and `*`)
- Constructor object matching in the `match` statement
- Support accessing various object properties (`__dict__`, `__slots__`
etc.) as much as possible in a static context
- Optional automatic switching between Codon and CPython (i.e.
compile only compatible functions and leave the rest to Python)
- Better error messages
- Warning support
- Explain performance considerations
- Explain that a CPython feature is not supported
- Modules and incremental compilation
- Cache compilation modules
- Fast generics compilation in debug mode for quick turnarounds
- Memory management
- Auto-tune GC
- Optional alternative memory management modes like reference
counting
- GPU support
- Target Apple, AMD and Intel GPUs
- GPU-specific compiler optimizations (e.g. for using various
Python constructs on the GPU)
- Interoperability with other languages
- Direct C++ interoperability via Clang
- R interoperability
# Libraries
Currently, missing Python functionality can be easily accessed via a
`from python import foo` statement, which is sufficient in most cases
as many libraries are just thin wrappers around a C library and/or not
performance-sensitive.
However, in the near future, we would like to support the following
modules natively:
- Python's standard library
- Complete builtins support
- 1-to-1 compatibility with existing Python functions and modules
- File modules: `os`, `sys`, `struct`, `pathlib` and so on
- Pretty much everything else on an as-needed basis
- Native NumPy, Pandas, etc.: Having Codon-native versions of the most
popular 3rd-party libraries would allow them to work with Codon's
other features like multithreading and GPU. We're currently prioritizing
NumPy and Pandas but aim to later target other popular libraries as well.
- Unicode support
- Python's testing infrastructure
# Infrastructure & Tools
- Windows support
- A sane package manager similar to Rust's
[Cargo](https://github.com/rust-lang/cargo)
- Auto-detection of installed Python libraries
- Improved `codon.jit` library support
- Better error messages
- Better installation flow
- Fully static binary support like Go
- Remove `libcodonrt` (runtime library) dependency if needed
- Remove `libcpp` dependency
- Improved Jupyter support
- Auto-completion and code inspection
- Jupyter magic command support
- Plugins for Visual Studio Code, Vim, Emacs and so on
# Documentation
- Fully document major differences with CPython
- Document Codon IR API, with guides and tutorials
- Document all supported modules
# Nice to have
- Implement Codon in Codon

View File

@ -26,7 +26,7 @@ foo2 = my2(4, 3.2)
```
{% hint style="warning" %}
When importing external non-Codon functions, you must explicitly specify
When importing C functions, you must explicitly specify
argument and return types.
{% endhint %}

View File

@ -1,4 +1,4 @@
# Copyright (C) 2022 Exaloop Inc. <https://exaloop.io>
# Copyright (C) 2022-2023 Exaloop Inc. <https://exaloop.io>
__all__ = ["jit", "convert", "JITError"]

View File

@ -1,4 +1,4 @@
# Copyright (C) 2022 Exaloop Inc. <https://exaloop.io>
# Copyright (C) 2022-2023 Exaloop Inc. <https://exaloop.io>
from argparse import ArgumentError
import ctypes

View File

@ -1,4 +1,4 @@
# Copyright (C) 2022 Exaloop Inc. <https://exaloop.io>
# Copyright (C) 2022-2023 Exaloop Inc. <https://exaloop.io>
from libcpp.string cimport string
from libcpp.vector cimport vector

View File

@ -1,4 +1,4 @@
# Copyright (C) 2022 Exaloop Inc. <https://exaloop.io>
# Copyright (C) 2022-2023 Exaloop Inc. <https://exaloop.io>
# distutils: language=c++
# cython: language_level=3

View File

@ -1,4 +1,4 @@
# Copyright (C) 2022 Exaloop Inc. <https://exaloop.io>
# Copyright (C) 2022-2023 Exaloop Inc. <https://exaloop.io>
import os
import sys

View File

@ -11,7 +11,7 @@ RUN mkdir -p /github/codon
COPY cmake /github/codon/cmake
COPY codon /github/codon/codon
COPY docs /github/codon/docs
COPY extra /github/codon/extra
COPY jit /github/codon/jit
COPY stdlib /github/codon/stdlib
COPY test /github/codon/test
COPY CMakeLists.txt /github/codon

View File

@ -201,7 +201,7 @@ class __internal__:
if idx < 0:
idx += len
if idx < 0 or idx >= len:
raise IndexError(f"tuple index {idx} out of range 0..{len}")
raise IndexError("tuple index out of range")
return idx
def tuple_getitem(t: T, idx: int, T: type, E: type) -> E:
@ -495,7 +495,7 @@ class __magic__:
# @dataclass parameter: container=True
def getitem(slf, index: int):
if staticlen(slf) == 0:
raise IndexError(f"tuple index {index} out of range 0..0")
__internal__.tuple_fix_index(index, 0) # raise exception
else:
return __internal__.tuple_getitem(slf, index, type(slf), tuple_type(slf, 0))

View File

@ -3,3 +3,7 @@ a = ['a', 'b', 'c']
@export
def foo(n: int):
print(''.join(a) * n)
@export
def main():
a.append('d')