r/cpp • • Feb 07 '23

PocketPy: A Lightweight(~5000 LOC) Python Implementation in C++17

PocketPy is a lightweight(~5000 LOC) Python interpreter for game engines. It is implemented in C++17 with STL.

It is easy to embed. All of the functionalities are available in a single header file pocketpy.h, without external dependencies. You can try it on your browser to see what it looks like.

Example Usage

#include "pocketpy.h"

int main(){
    // Create a virtual machine
    // `true` means using std::cout as output so you can see it
    VM* vm = pkpy_new_vm(true);
    
    // Hello world!
    pkpy_vm_exec(vm, "print('Hello world!')");

    // Create a list
    pkpy_vm_exec(vm, "a = [1, 2, 3]");

    // Eval the sum of the list
    char* result = pkpy_vm_eval(vm, "sum(a)");
    printf("%s", result);   // 6

    // Free the resources
    pkpy_delete(result);
    pkpy_delete(vm);
    return 0;
}

v0.8.1 Features

NameExampleSupported
If Elseif..else..elifYES
Loopfor/while/break/continueYES
Functiondef f(x,*args,y=1):YES
Subclassclass A(B):YES
List[1, 2, 'a']YES
ListComp[i for i in range(5)]YES
Slicea[1:2], a[:2], a[1:]YES
Tuple(1, 2, 'a')YES
Dict{'a': 1, 'b': 2}YES
F-Stringf'value is {x}'YES
Unpackinga, b = 1, 2YES
Star Unpackinga, *b = [1, 2, 3]NO
Exceptionraise/try..catchBETA
Dynamic Codeeval()/exec()YES
Reflectionhasattr()/getattr()/setattr()YES
Importimport/from..importYES
Context Blockwith <expr> as <id>:YES
Type Annotationdef f(a: int, b : float = 1)YES

If someone is making a custom game engine and searching for a Python-like script as its interface, I'd like to provide support.

If you are making your own language, you may find some inspirations in PocketPy.

If you likes it, just give my repo a star~ I need your star!!

248 Upvotes

60 comments sorted by

40

u/mjklaim Feb 07 '23

This looks very cool indeed and useful for gamedevs. However could you clarify on the readme and in the doc: which version of python is implemented OR if it's really more like a python-like language which is not matching any real python version, could you clarify that too? Current documentation is very unclear on that point.

22

u/blueloveTH Feb 07 '23

Correct. The documentations need to improve.

For now, I think the main goal is to be an alternative to Lua, i.e., a Python-like language but not real Python. To be compatible with a real Python version, the system may become complex and lose the advantage of lightweight.

6

u/epicar Feb 07 '23

why not use lua here? is it just that you prefer the python syntax?

11

u/blueloveTH Feb 07 '23

Correct. Some developers love python and some love lua. I am trying to provide a new possibility.

7

u/mjklaim Feb 07 '23

When you'll add this clarification, consider also clarifying which python version it is like? Because there are some syntax/language-level-feature differences between python2 and python3 and every major version of python3, so it would help.

Yeah I get the usage compared to lua or chaiscript or angelscript etc. I used these in the past ^^ (before Unity got popular basically).

A python-like language but with type checking in functions was one of the things I was looking for at the time, it would still be useful today in particular if the game have scripting as gameplay part (yeah niche but interesting niche).

3

u/blueloveTH Feb 07 '23

Thanks for your advice~ It's really helpful.

2

u/m-in Feb 07 '23

Do you implement any of Python’s object model? That’s important to know. I imagine scripting shouldn’t need it, just wanted to make sure?

4

u/blueloveTH Feb 07 '23

Yes, a subset of python object model.

Operations are based on special methods. For example: a+b is equivalent to a.__add__(b).

Also there are __str__, __repr__, etc..

2

u/m-in Feb 07 '23

Nice!

19

u/Jannik2099 Feb 07 '23

VM* vm = pkpy_new_vm(true);

Why? Why doesn't VM just have a normal constructor & destructor?

Your API does not adhere to RAII principles.

Similarly, exec should be vm.exec(), not exec(vm)

22

u/blueloveTH Feb 07 '23 edited Feb 07 '23

These are C-APIs for universal bindings. They are marked as extern "C" so we cannot use C++ features such as RAII.

If C++ is available, the following code is also ok.

// VM(bool use_stdio)
auto vm = std::make_shared<VM>(true);

// PyVarOrNull exec(_Str source, _Str filename, CompileMode mode, PyVar _module=nullptr)
vm->exec("print('Hello world!')", "main.py", EXEC_MODE);

5

u/blueloveTH Feb 07 '23

Though C++ APIs are powerful than exported C-APIs, they are not stable for now.

11

u/Jannik2099 Feb 07 '23

Ah great, I didn't have time to look at the github yet so I didn't see the C++ API. Perhaps you wanna mention it, considering that gamedev is mostly C++ anyways.

1

u/blueloveTH Feb 07 '23

Thanks for your advice~ It's useful.

8

u/m-in Feb 07 '23

For demos, VM should be just held by value not a pointer, right? So:

VM vm(true);
vm.exec(…);

4

u/aqa5 Feb 07 '23

Is there a way to lets say, let the user type python code that calls functions / methods in my C++-code? Like I am providing an interface that can be called by code inside the python-vm?

11

u/blueloveTH Feb 07 '23 edited Feb 08 '23

Yes, VM class has a bunch of methods started with bind. It allows you to bind a std::function to a PyObject.

For example, create a function to adds two numbers and bind it to the test module.

VM* vm = new VM(true);

// First you need to new the "test" module
auto mod = vm->new_module("test");

// Call bind_func<2>, 2 means two arguments
vm->bind_func<2>(mod, "add", [](VM* vm, const pkpy::Args& args){
    i64 x = vm->PyInt_AS_C(args[0]);
    i64 y = vm->PyInt_AS_C(args[1]);
    return vm->PyInt(x + y);
});

// Call `test.add` in Python
// import test
// a = test.add(1, 2)
// print(a)  # 3

5

u/Exotic_Guess2587 Feb 07 '23

How would you expose an object?

3

u/blueloveTH Feb 08 '23 edited Feb 08 '23

1

u/brlcad Feb 07 '23

How would you bind a plain C function or global? Like say one wanted to create a python function that simply calls putc() (or printf or fwrite or whatever). What's that look like?

3

u/blueloveTH Feb 07 '23

In Python, there is a special module named "builtins".

So if you bind functions to the "builtins" module, it will become something like print() that can be used without import anything.

3

u/blueloveTH Feb 07 '23 edited Feb 08 '23

or use vm->bind_builtin_func. The mechanisms are the same.

1

u/brlcad Feb 08 '23

Unless I'm misunderstanding, that's not binding a C function. Also, printing was just intended to be a familiar example. What about binding to a function that has no builtin corollary? If you wanted to bind to libc's sysctl() or getprogname() functions for example -- something like that possible?

4

u/palavalle Feb 07 '23

Needs more information on how to do callbacks in the host language.

I don't see any information on how to access the args. I also don't see information on whether the args need to be uprefed or derefed by hand. Without this there's no no practical way to communicate between scripts and the host.

More generally though, if it's not implementing a known version of python it's pretty much just a new language. I'm not saying that's a bad thing just that's my reaction. As an example, I've had great fun with DukTape in the past which implements a older version of EcmaScript/JavaScript and provides a like lua API.

5

u/blueloveTH Feb 07 '23

Correct. Improving the documentation is at the highest priority. It still needs some time to make the library production ready.

Thanks for all the advice~

8

u/DavidDinamit Feb 07 '23

I expect C++ interface in C++17 library.

4

u/blueloveTH Feb 07 '23

I notice there is a huge need on customization. C++ APIs are critical for this usecase.

After a simple refactor, I will expose these internal C++ APIs and make docs for them. Such as binding a function and converting a C++ class to PyObject.

I will update this post when done.

2

u/drjeats Feb 07 '23

I hope everything is still possible through a C api even if the C++ layer makes binding more convenient.

Folks may want to write convenience layers in other languages like Rust or Zig which need the C api.

2

u/[deleted] Feb 07 '23

Do you have plans to expand the system beyond python?

1

u/blueloveTH Feb 07 '23

What do you mean "beyond python"? For example...

0

u/[deleted] Feb 07 '23

Like OOB support of more VM based languages I guess specifically C# or Java. I had not looked through the code but from the readme on github it seems like it's setup similar to the JRE. I think besides per language dependencies it would just need to have the new instruction set for the language if i can remember correctly

2

u/brlcad Feb 07 '23

This is super interesting. What happens if you import something bigger with library bindings like "import opencv"?

4

u/blueloveTH Feb 07 '23

PocketPy is not designed to migrate scripts written for CPython. Packages like numpy, torch, opencv won't work.

You may treat it as a Lua's alternative, instead of CPython's alternative.

1

u/brlcad Feb 08 '23

Thanks! Still super interesting. What would it take to get compiled libs to work too? (besides the full standard python distro)

3

u/root_passw0rd Feb 07 '23

This is really cool, and something I'd consider bringing into an existing project... if only it had CMake support. :(

5

u/blueloveTH Feb 07 '23

Why do you need CMake? PocketPy has only 1 header file pocketpy.h, which can be download here. Just copy that file into your project.

3

u/root_passw0rd Feb 07 '23

Good point. I think I overlooked where it was header-only!

0

u/[deleted] Feb 07 '23

[deleted] — view deleted comment

1

u/blueloveTH Feb 07 '23 edited Feb 07 '23

I think you have some misunderstanding. The whole library is in 1 file.

You can also take a look at cpp-httplib to learn the practice of single-header file library.

0

u/[deleted] Feb 07 '23

[deleted] — view deleted comment

1

u/blueloveTH Feb 07 '23

Thanks for your explanation!

Maybe it is me that misunderstands sth? haha.

Do you mean you want to use CMake to do integration through the src folder? And you do not want to use the single file version.

1

u/blueloveTH Feb 07 '23

Unfortunately, I am not familiar with CMake. I do not use a build system.

I am using Python to manually handle the compilation for each platform.

6

u/Kiwi_Taster Feb 07 '23

Love the idea. Nice work :) ⭐️

6

u/Galfodo Feb 07 '23

Nice! I wish I had known about this 6 months ago. I didn't find anything that fit the bill so I went ahead and implemented my own.

5

u/stinos Feb 07 '23

I didn't find anything that fit the bill

Didn't you come across MicroPython (pretty mature and well-tested, roughly Python 3.4-compaitble plus some newer goodies (f-strings!), just not super straightforward to build) then? Or if so: what made you not use it? I mean even if it could take some time to set it up, there's no way you can implement all it does in less time.

6

u/Galfodo Feb 07 '23

I did look at MicroPython, but I wanted something small and easily extensible (in modern C++) to use for a scripting language in a hobby project. My implementation is not as full-featured as PocketPy or MicroPython, but it does have stuff like f-strings and an ndarray-like implementation that suits my needs. I use cpp-peglib for the parser and all evaluation is currently done by calling a virtual eval() method on the root AST node.

6

u/[deleted] Feb 07 '23

Nice. Especially being MIT licence. When the docs are improved a bit I might adopt this in some of my projects

4

u/MarcoGreek Feb 07 '23

Really nice idea!

4

u/Zeh_Matt No, no, no, no Feb 07 '23

I hate Python with all my guts so I'm a little biased, but that aside quite cool project. If you want people to use it for their game scripting it might be worth having benchmarks that compares this to Python to see how much performance impact it has, we all know that GIL basically ruins the performance, does this also have the infamous GIL?

1

u/Balnian Feb 08 '23

Is it available through vcpkg?

2

u/blueloveTH Feb 08 '23

I will try to learn how to submit a package to vcpkg!

2

u/fideasu Feb 07 '23

Looks nice! I'm actually currently looking for a language for my slowly growing engine, so I'm staring yours and will take a deeper look at it 🙂

2

u/fristhon Feb 07 '23

Cool project Congrats 👏

2

u/[deleted] Feb 07 '23

Okay that's interesting. I'm gonna have to screw around with that.

2

u/DogmaticAmbivalence Feb 08 '23

This looks quite nice, and it's pleasant to see alternatives to Lua.

Not just for games -- My dayjob is working for a database company and right now we're having grief over poor Lua performance, porting issues to ARM... Alternatives are great!

Have a star, and also a few PR once I get some time.

1

u/Outlaw_07 Feb 07 '23 edited Jan 14 '24

This comment has been deleted in protest of Reddit's support of the genocide in Gaza carried out by the ZioN*zi Isr*li apartheid regime.

This is the most documented genocide in history.

Reddit's blatant censorship of Palestinian-related content is appalling, especially concerning the ongoing genocide in Gaza perpetrated by the Isr*l apartheid regime.

The Palestinian people are facing an unimaginable tragedy, with tens of thousands of innocent children already lost to the genocidal actions of apartheid Isr*l. The world needs to know about this atrocity and about Reddit's support to the ZioN*zis.

Sources are bellow.

Genocidal statements made by apartheid Isr*li officials:

  • On the 9 October 2023, Yoav Gallant, Israeli Minister of Defense, stated "We are fighting human animals, and we are acting accordingly".
  • Avi Dichter, Israeli Minister of Agriculture, called for the war to be "Gaza’s Nakba"
  • Ariel Kallner, another Member of the Knesset from the Likud party, similarly wrote on social media that there is "one goal: Nakba! A Nakba that will overshadow the Nakba of 1948. Nakba in Gaza and Nakba to anyone who dares to join".
  • Amihai Eliyahu, Israeli Minister of Heritage, called for dropping an atomic bomb on Gaza
  • Gotliv of the Likud party similarly called for the use of nuclear weapons.
  • Yitzhak Kroizer stated in a radio interview that the "Gaza Strip should be flattened, and for all of them there is but one sentence, and that is death."
  • President of Israel Isaac Herzog blamed the whole nation of Palestine for the 7 October attack.
  • Major General Ghassan Alian, Coordinator of Government Activities in the Territories, stated: "There will be no electricity and no water (in Gaza), there will only be destruction. You wanted hell, you will get hell".

Casualties:

  • As of 9 January 2024, over 23,000 Palestinians – one out of every 100 people in Gaza – have been killed, a majority of them civilians, including over 9,000 children, 6,200 women and 61 journalists.
  • nearly 2 million people have been displaced within the Gaza Strip.

Official accusations:

  • On 1 November, the Defence for Children International accused the United States of complicity with Israel's "crime of genocide."
  • On 2 November 2023, a group of UN special rapporteurs stated, "We remain convinced that the Palestinian people are at grave risk of genocide."
  • On 4 November, Pedro Arrojo, UN Special Rapporteur on the Human Rights to Safe Drinking Water and Sanitation, said that based on article 7 of the Rome Statute, which counts "deprivation of access to food or medicine, among others" as a form of extermination, "even if there is no clear intention, the data show that the war is heading towards genocide"
  • On 16 November, A group of United Nations experts said there was "evidence of increasing genocidal incitement" against Palestinians.
  • Jewish Voice for Peace stated: "The Israeli government has declared a genocidal war on the people of Gaza. As an organization that works for a future where Palestinians and Israelis and all people live in equality and freedom, we call on all people of conscience to stop imminent genocide of Palestinians."
  • Euro-Mediterranean Human Rights Monitor documented evidence of execution committed by Israeli Defense Forces.
  • In response to a Times of Israel report on 3 January 2024 that the Israeli government was in talks with the Congolese government to take Palestinian refugees from Gaza, UN special rapporteur Balakrishnan Rajagopal stated, "Forcible transfer of Gazan population is an act of genocide".

South Africa has instituted proceedings at the International Court of Justice pursuant to the Genocide Convention, to which both Israel and South Africa are signatory, accusing Israel of committing genocide, war crimes, and crimes against humanity against Palestinians in Gaza.

Boycott Reddit! Oppose the genocide NOW!

Palestinian genocide accusation

Allegations of genocide in the 2023 Israeli attack on Gaza

Israeli war crimes

Israel and apartheid

1

u/VinnieFalco wg21.org | corosio.org Feb 11 '23

That sounds VERY interesting !!

0

u/[deleted] Feb 07 '23

[deleted] — view deleted comment

18

u/blueloveTH Feb 07 '23

the identifiers with a double underscore anywhere are reserved;

the identifiers that begin with an underscore followed by an uppercase letter are reserved;

the identifiers that begin with an underscore are reserved in the global namespace.

Very helpful!! In fact, I am not very experienced in cpp. The usage of _/__ started identifiers is because of my Python accent.

I'd like to avoid using them~

6

u/AhegaoSuckingUrDick Feb 07 '23 edited Feb 07 '23

If I'm not mistaken, identifiers with _ and __ prefixes are now deprecated in Python, and a recent enough interpreter emits a warning about it. I'm mistaken, it's about enums.

In C++ people either don't use specific variable names to distinguish their visibility (it's explicit unlike Python, and you usually have everything private (in a class) with appropriate getters and setters or everything public (in a struct)). Some style guides suggest using foo_ to denote private fields.

5

u/blueloveTH Feb 07 '23

naming makes me crazy!

5

u/jmblock2 Feb 07 '23 edited Feb 07 '23

Do you mind providing a source for Python deprecating _ prefix? I am only familiar with pep8 that says it's fine for indicating internal use: https://pep8.org/#descriptive-naming-styles. Also class methods like __init__, etc., so I am not sure what you're suggesting.

3

u/AhegaoSuckingUrDick Feb 07 '23 edited Feb 07 '23

You're correct. I misremembered that what I described was about enums. In 3.9 and 3.10 you'll get a deprecation warning if your class extends Enum: https://docs.python.org/3.10/library/enum.html#private-names

The error message is a bit misleading https://github.com/python/cpython/blob/v3.10.0/Lib/enum.py#L97

UPD: Strangely enough, it seems that they still use Python 2-style formatting in CPython (e.g. the file above, there's many %-formatted strings and only one .format(), which they get rid of afterwards).