go4hx

Written in Golang.
./export.go and ./analysis
Go is a bootstrapped language, and the Go compiler is in the stdlib and accessed via ./export.go
Uses reflection and recursion to turn the Go typed AST into JSON data
Sends the json data over local TCP network to go2hx. go4hx uses the TCP client, and go2hx is the TCP server.
go4hx the name is a play on words of Go being used for Haxe.

go2hx

Written in Haxe

./src

Typer.hx handles all of the AST transformation.
Patch.hx patches the Go stdlib with Haxe code.
Gen.hx generates all of the Haxe files from arrays of TypeDefinition, creates the interop layer.
Main.hx entry point to the compiler handles the CLI arguments and calling ./src/Typer.hx and ./src/Gen.hx.
Printer.hx modified haxe.macro.Printer
Args.hx a fork of a fork of hxargs.
Cache.hx a cache to skip files already generated with the same contents as previously.
Network.hx a network abstraction of the TCP server that accepts connections from go4hx
Types.hx type information copied over from stdgo/_internal/internal/reflect/Reflect and used internally within the compiler and most of the time transformed into haxe.macro.Expr.ComplexType
Util.hx utility functions that are often used by scripts and src

./scripts

StdGo.hx compiles Go's stdlib into Haxe and adds it to package stdgo

./stdgo

Most files here are compiled, the list below are the exceptions

Go.hx and Go.macro.hx the all in one class that handles creating interface{}/any(stdgo.AnyInterface), normal interfaces, it holds recover exception, etc. Best way to learn more is to read the api.
GoNumber.hx, GoInt32.hx etc, there are number abstracts that mimic how Go does math, for example Go integer division results in an integer.
Pointer an abstract class that mimics Pointers in Haxe.
Slice.hx an abstract class that represents the slice type []type.
Slice.hx an abstract class that represents the Go array type [count]type.
AnyInterface an abstract class that represents any type in Go any or interface{}.

Haxe focused

Intentionally written primarily in Haxe code.
Focused on improving the Haxe ecosystem first.
Uses the latest language features from Haxe.
Uses Haxe macros to simulate the Go type system.
Creates an interop layer to interact with the compiled code as if it was handwritten Haxe code.
Plan to remove Go as a dependency.
Haxe focused but not excluding the opportunities in the Go ecosystem, more using the Haxe ecosystem to build up correctness and a community before branching out more.

AST to AST

Transforms Go's AST to Haxe's AST.
No needless intermediate representations.
Keeps the abstraction level practically the same for both language.

Not reinventing the wheel

Uses Go's own compiler to generate the typed AST.
Uses Haxe's macro keyword to create Haxe exprs.
Uses Haxe's own printer to print out the Haxe AST.
go2hx does not have its own parser, lexer, type inference system, printer, or expr data structures.

Built on the backs of giants

Spiritual successor to tardisgo created by Elliott Stoneham who is a core contributor and founder of the project.
Pulling lessons from the projects that came before, and iterating upon the knowledge both technical, design and communication wise.
Uses both tardisgo and gopherjs as a reference implementation for many parts of the compiler.

debugging

The project has no support for a debugger. Debugging can be done on the Haxe code and Go code and compared but often that approach is slow and complex.

The current method is to add prints in the Go code and compile it into Haxe and compare the Go running version with the Haxe version's output.

If you don't want to create a testbed to do this, there is one included within the compiler source.

Rnd

It is found in ./rnd where ./rnd/main.go is the file to edit to make the changes and can be run with the command(requires hashlink by default):

haxe --run Make rnd

Contributions welcome for improvements or adding more sophisticated debuggers!

Go AST to Haxe AST walkthrough

The entry point for the compiler comes from running the command via haxelib.

haxelib run go2hx ./finderrors.go

This command will run Run.hx, this file will clone the necessary repos for the Go part of the compiler, run go build . on the users behalf and then choose a target to build the Haxe part of the compiler, the target is chosen based on what is available or explicit arguments.

go build .

The go build command will build according to the root file go.mod and the main file export.go, go.mod can be thought of a build configuration file for Go and will list the required dependencies. There is a special replace in the go.mod this allows the compiler to use a custom fork of tools found here (not very important but may come up when it comes to default sizes for int and uint)

`export.go`

main entry for the Go part of the compiler (also called go4hx)
Facilitates calling the Go compiler and pulling out all of the typed AST.
Called from the Haxe side of the compiler
Launched by Haxe side with specific arguments and waits to connect to the TCP server (Haxe side).
Once connected sends over the data in JSON form .
After sending all of the data, will stay connected in case the another package will need to be compiled.

export.go also calls other go files such as in package/folder analysis these files perform passes over the Go AST for example, to add pointer variables for each pointer initiated, control flow flattening when goto jumps are detected etc.

The Haxe part of the compiler receives the type information and AST, the AST in haxe is held in ./src/Ast.hx the types are held in ./src/Types.hx not to complicated!

`src/Main.hx`

entry point is function main
Coordinates go4hx and processes all of the command line arguments.
Calls src/Typer.hx for the typing process and after the typing takes place.
Passes the Haxe AST to src/Gen.hx.
Generator and handles calling src/Printer.hx, creating Haxe files and interop layer files.

`src/Typer.hx`

Entry point is function main
Does multiple passes over the Go AST and handles all of the AST transformations.
Most complex and biggest file in the codebase
Mostly takes in Go AST and returns Haxe AST.
Naming convention follows Go AST.
Uses a lot of the macro keyword

The Go AST is File->Decl. Decl->FuncDecl or GenDecl, FuncDecl is a modular function which holds args, params (for generics), return, body (stmts->exprs), and if it is a variadic (final argument is a rest). GenDecl->Array->Spec can be an import, type, or variable, an Array means it is all grouped together. For example an Array or type specs would be:

type (
    X struct{}
    Y int
)

If it 2 separate Specs it would be:

type X struct{}
type Y int

Stmts or Statements are a special category of Exprs, in Haxe all Go stmts would be an expr because in Haxe everything is an expr, for example: for loop, if stmt etc. In Haxe terms these are Exprs that always return void type.

A Stmt also has a 2 special Statements, ExprStmt and DeclStmt. This allows Stmt to go up or down the abstraction layer in AST. DeclStmt allows having a Decl where a Stmt would be and likewise for an ExprStmt.

This is special in the case of DeclStmt because it allows creating DeclTypes inside of a function body (a function body is an Array of statement)

For example:

func main() {
    type X struct{
        x int
        y int
    }
}

func foo() {
    {
        type X struct{}
    }
    {
        type X int
    }
}

This allows scoped types. The types must be able to be inferred by the compiler at compile time, so no runtime types from an if else is allowed for instance.

Next part (advanced notes)

Now onto the interesting part.

We will now go over a real world Go program, source code here

The program was written by me to be able to found the most repeating errors in the compiler from the test logs, in order to prioritize what to fix.

It uses some file operations and text and the go2hx compiler is able to compile it correctly.

The code is not very clean and could do with lots of optimization, but given it's job is one off analysis, it's not too important and hits the goal of going over something real and gaining insights for how the compiler handles it.

So to start off the compiler is ran (details found above) and the start, for our purposes will be inside ./export.go the AST is obtained from:

initial, err := packages.Load(cfg, &types.StdSizes{WordSize: 4, MaxAlign: 8}, args...)

initial now holds all of the packages and parsePkgList will now be called.

parsePkList goes over every package in a loop and calls mergePackage to merge the package into a single File. The package is looped again and parseLocalPackage is called. parseLocalPackage runs all of the analyzers from the analysis folder/package. Then parsePkg is called, which in turn calls parseFile. parseFile loops over the decls, and the specs, and parseData is then called. This function gets called iteratively.

If new information needs to be sent to the Haxe part of the compiler modification can be made by finding the type in the very large type switch stmts and adding a key map to new data exposed, for example:

case *ast.CompositeLit:
    data["exprType"] = parseData(node.Type)
    data["type"] = parseType(checker.TypeOf(node), map[string]bool{})
    data["test"] = node.NewExposedThing // not valid field access for this type

and then have it be exposed in Haxe by adding it in ./src/Ast.hx:

typedef CompositeLit = {
	// > Node,
	type:Expr,
	exprType:Expr,
	lbrace:Pos,
	elts:Array<Expr>,
	rbrace:Pos,
	incomplete:Bool,
	test:String, // this is the new field
};

Whenever a type needs to be parsed, parseType is called.

The interesting part

The Go typed AST will be all be passed into ./src/Typer.hx this is where most important things happen.

package main

Sets what the package is, and is set into info.global.path, info is passed to most functions in the codebase and info.global is always the same across an entire package, where as info fields are passed by value for each function, and some of the fields are reset.

Info holds lots of important context information when transpiling the AST.

import (
	"os"
	"path/filepath"
	"sort"
	"strings"
)

Each import is run with typeImport, and sets info.renameIdents in most cases which is a map that looks for a given identifier and renames it.

var removeStrings = []string{
	",",
	".",
	":",
	"0", "1", "2", "3", "4", "5", "6", "7", "8", "9",
}

typeValue is run and returns an array of TypeDefinition. []string is called with typeExprType which takes a Go AST expr, and turns it into a ComplexType []string -> stdgo.Slice<stdgo.GoString>.

The first check is if it's a destructure, in this case no

value.names each name is ran through nameIdent

All values exist, so defaultValue is not used, and instead every value expr is called with typeExpr

expr = typeExpr(value.values[i], info);

The GoType variable nameType is turned into a haxe.macro.ComplexType via the function toComplexType. GoType is how Go type information is held for Haxe. It is an enum and found in ./src/Types.hx

In the end this creates a Haxe TDField of FVar

Inside of typeExpr the expr is a CompositeLit and therefore typeCompositeLit is called, and turns the type into a ComplexType and runs over an array of Expr. The exprs are all BasicLit and runs typeBasicLit it is a string literal and is turned into a Haxe string expr.

func main() {}

Is transformed inside function typeFunction the decl.body has typeBlockStmt called on it, and goes over every stmt with typeStmt

The first stmt is a DeclStmt this leads to having typeValue be called, because it's a constant the analysis package already handles it so a macro {} is returned to denote nothing.

The second stmt is an AssignStmt which is transformed with function typeAssignStmt this is a destructure version and runs the lhs (left hand side) and rhs (right hand side) exprs with typeExpr.

Walkthrough: encoding/json compiled Haxe code won't compile

notice: This guide requires haxe, hashlink, nodejs, and hxnodejs to be installed along with the compiler.

The compiler is complex and can be intense to try and figure out.

Instead of the conventional way of explaining everything for 30mins and hoping 20% sticks, I would like to walk you through a real issue, and see what the steps are to find the issue and write a test for it.

What is encoding/json?

It's a stdlib pkg (short for package) in golang, all stdlibs for the language can be found here.

The name is self explanatory, however keep in mind the layout, unlike in Haxe where json will be thrown into the haxe package, all of Go's stdlib is categorized, this can help ease navigation and many issues effecting one encoding package will likely effect the others because of shared imports.

The problem

encoding/json does not compile.

To have the compiler build this package:

haxe --run Make std encoding/json

To run the encoding/json tests:

haxe --run Make test encoding/json --interp

haxe --run Make runs the Make.hx file.
test is the argument to tell what to have Make do.
encoding/json is the package to test.
--interp is how to run the test, this could also be --hl test.hl but note that you are responsible for running hl test.hl after.

The output is:

./stdgo/_internal/mime/multipart/Multipart_fileheader_static_extension.hx:22: characters 9-120

 22 |         return stdgo._internal.os.Os_open.open((@:checkr _fh ?? throw "null pointer dereference")._tmpfile?.__copy__());
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | error: stdgo.Ref<stdgo._internal.os.File> should be stdgo._internal.mime.multipart.File
    | have: { _0: stdgo.Ref<...> }
    | want: { _0: stdgo._internal.mime.multipart.File }

First thing I see is the top line:

./stdgo/_internal/mime/multipart/Multipart_fileheader_static_extension.hx:22: characters 9-120

Based on this information we know that type fileheader is using a static extension.

All go2hx generated static extensions will also use the @:using metadata documented here

This is important as when debugging the code it may be tricky to know where some methods are coming from, chances are it is because of this.

This paradigm is done to keep the code fast, by allowing Integer types to have methods like in Go, without needing to allocate an entire class for it.

have: { _0: stdgo.Ref<...> }
want: { _0: stdgo._internal.mime.multipart.File }

Given the above error information it's a little hard to draw any conclusion so let's go to the function in the code: ./stdgo/_internal/mime/multipart/Multipart_fileheader_static_extension.hx:22

We can see that the return type is an anonymous structure:

{ var _0 : stdgo._internal.mime.multipart.Multipart_file.File; var _1 : stdgo.Error; }

This is very useful to know because the return line creates no such anonymous structure, and instead is a single return value.

However if we check the method stdgo._internal.os.Os_open.open we found it, it is also has a multi return type, the types are similar but not identical. One is a multipart File and the other is an os File.

The multipart.File:

@:interface typedef File = stdgo.StructType & {
    function read(_0:stdgo.Slice<stdgo.GoUInt8>):{ var _0 : stdgo.GoInt; var _1 : stdgo.Error; };
    function readAt(_0:stdgo.Slice<stdgo.GoUInt8>, _1:stdgo.GoInt64):{ var _0 : stdgo.GoInt; var _1 : stdgo.Error; };
    function seek(_0:stdgo.GoInt64, _1:stdgo.GoInt):{ var _0 : stdgo.GoInt64; var _1 : stdgo.Error; };
    function close():stdgo.Error;
};

You can see it is decorated with @:interface metadata, this denotes that the original Go type was an interface.

The os.File:

package stdgo._internal.os;
@:structInit @:using(stdgo._internal.os.Os_file_static_extension.File_static_extension) class File {
    @:embedded
    public var _file : stdgo.Ref<stdgo._internal.os.Os_t_file.T_file> = (null : stdgo.Ref<stdgo._internal.os.Os_t_file.T_file>);
    @:local
    var _input : haxe.io.Input = null;
    @:local
    var _output : haxe.io.Output = null;
    public function new(?_file:stdgo.Ref<stdgo._internal.os.Os_t_file.T_file>, ?_input:haxe.io.Input, ?_output:haxe.io.Output) {
        if (_file != null) this._file = _file;
        if (_input != null) this._input = _input;
        if (_output != null) this._output = _output;
    }
    public function __underlying__() return stdgo.Go.toInterface(this);
    public var _close(get, never) : () -> stdgo.Error;
    @:embedded
    @:embeddededffieldsffun
    public function get__close():() -> stdgo.Error return @:check32 this._file._close;
    public function __copy__() {
        return new File(_file, _input, _output);
    }
}

This is a class without @:interface instead it has a @:using, the Go type is also a struct instead, and it's methods are only exposed with the static extension system.

This is very important because of how interfaces are represented in Haxe to have the same behavior as Go.

Explanation of interfaces to understand why the above is a problem

Haxe interfaces are explicit, Go interfaces are implicit, this is why in Haxe the interfaces for Go must be represented via typedef to an anonymous structure.

Structural unification in Haxe is the best representation of Go interfaces but it creates extra problems.

As mentioned above static extension are used to give a named type integer methods without needing to wrap the Integer in a class. Static extension methods however are not real methods attached to the type and therefore structural unification does not work.

The current solution is when a value is cast into an interface (empty interface is a special case to get into later) a structure must be created in order to fulfill the unification with the interface. This must be done explicitly, by calling wrapperExpr in Typer.hx:

private function wrapperExpr(t:GoType, y:Expr, info:Info):Expr {
	var self = y;
	var selfPointer = false;
	if (isPointer(t)) {
		selfPointer = true;
		t = getElem(t);
		self = macro $y.value;
	} else if (isRef(t)) {
		t = getElem(t);
	}
	switch t {
		case named(name, methods, type, alias, params):
			if (!alias && methods.length == 0 && !isStruct(type))
				return y;
			if (type == invalidType)
				return y;
			if (isInterface(type)) {
				return selfPointer ? self : y;
			}
			return macro stdgo.Go.asInterface($y);
		default:
	}
	return y;
}

This function if the criteria is met calls stdgo.Go.asInterface which is a macro function that looks at the type of y and tries to create the type of y with the suffix _asInterface. This suffix denotes all classes that wrap go2hx compiled types that need to be able to be unified with interfaces.

Therefore the first return type multipart.File does not unify with os.File because wrapperExpr is not explicitly being called by the macro function being added to the code. The function if included would create a structure to unify with the return type, solving the issue.

TLDR

To turn non interface types to an interface type most likely stdgo.Go.asInterface will need to be there to create the structure necessary for unification.

Making a test!

Note for devs that are not big test writers:

You may think, that's great you understand the problem just fix it and go to the next issue, but half the battle is finding the issue and isolating it, so put a little bit more effort to create a test and prevent it from occurring again, and also to know that the problem is actually fixed.

This part requires knowing how to write Go code, it's a simple language but it has its quirks and being able to write it to isolate problems is very useful.

Quick testing of Go code is done in ./rnd in there you can find a main file ./rnd/main.go

Rewrite the file to have it run the isolated problem. The problem is that when a return expr is a call expr with a multi return type of one type being struct returning for a type with interface for the first type, then a proper casting is missing.

Here is minimal code to test this problem:

// all Main files need to use a main package, regardless of location
package main

// main entry point func
func main() {
	a()
}
// multi return type, taking an Interface and a Bool 
func a() (WriterInterface, bool) {
    // calls to b which is also a multi return type
	return b()
}

// multi return type, taking a Struct and a Bool
func b() (Writer, bool) {
	return Writer{}, false
}

type Writer struct{}
// A method that in Haxe will be represented using a static extension
// This is required to prevent the interface from being empty
// Empty interfaces have much different behavior and do not rely on structure unification as there is no method to unify with
func (Writer) Write() {}
// Interface that will unify with Writer because they both have the same methods
type WriterInterface interface {
	Write()
}

To run this code:

haxe --run Make rnd

Now after confirming it indeed errors as expected, you can add it to the unit tests by creating a named file such as multireturn0.go and copy-paste the contents from ./rnd/main.go and that is it.

Conclusion

Almost all compiler issues will come from places where Haxe does not have overlap with Go, a good understanding of Go and Haxe is required to be able to diagnose issues effectively.

The best technical ways to learn both languages:

What should be covered next?

Part 2 fixing the encoding/json compiler issue in the compiler source
Detailed walkthrough of how Go code turns into Haxe code (the functions called and file locations along with detailed structure of the compiler)
Explaining the interop system that links go2hx styled Haxe code to normal Haxe code.
The differences between Go and Haxe and how go2hx tries to solve each case.

Compatibility

Everything is believed to be compatible except what is listed below.

Language

unsafe
generics
go routines
Only certain Go versions are supported.

Stdlibs

syscall
reflection
unsafe
cgo
Tested stdlibs listed as failing.
Untested stdlib packages are likely failing.

Go versions

Only version 1.21.3 is tested as of March 3rd 2025.

At the bottom of the Go spec there is a list of language changes that occur for every version.

In addition the patch notes of a version list what stdlib changes there are, although there is a backwards compatibility promise in place, this only covers the outward facing api, so the internals can be changed up and this can cause problems for go2hx which may rely on their consistency.

Go generics

The language feature has currently no support in the compiler.

The current idea to support this language feature is to use: overloads and create overloaded functions for all possible functions.

Cgo

Unsupported and can be thought of as almost a side adventure.

The side adventure in question has many fundamental questions that should likely be answered in the beginning for those interested:

What targets would be supported?
How would the bridge operate?
How would the memory management look?

Potential useful links

Go routines

Goroutines are coroutines with thread pools for targets that support threads. At present Haxe has no coroutines in the language but is planned for Haxe 5.

The current implementation uses a simple 1 thread per 1 goroutine.

Reflection

The package is fully patched out, and replaced with a system that uses the construction of type information with enums created for go2hx emulation of the Go type system.

Many reflection methods are unimplemented or incorrect, heavily used reflection methods will likely work but are not directly tested as other stdlibs are.

The patches for reflection can be found stdgo/_internal/internal/reflect/Reflect and src/Patch.hx.

Syscall

Fully unsupported. Many of the syscalls are possible to support on targets such as C++, however no work has been done to support them.

Unsafe

Very unsupported but planned using emulated memory addresses to memory regions.

FAQ

Does the compiler support Go as a Haxe target?

No and it's not within the scope of the project, however happy to knowledge transfer and support any Go target for Haxe.

Can high performance be achieved with the compiled code?

Yes! There is no inherent systemic issue with the compiled code having comparable speeds with normal Haxe code. The compiler does AST to AST translation so it is at the same level of abstraction for the Haxe compiler to be able to optimize the code in the same way as if it was handwritten.

The layer functions or interop files for the go2hx styled Haxe code to normal Haxe code does have a performance penalty, however for basics types it can be 0 because of Haxe's 0 cost abstractions.

In the case of structures or other cases where there is an allocation, it is possible to use the internal go2hx data types to not pay the penalty (at the cost of ease of use).

This area is truthfully not very well explored, as the current priority of the project is correctness.

If it is a pain point for your use case, we would love to hear more about it and improve it.

Why not use externs instead of compiling Go code into Haxe?

Because externs prevent multi target usage of the compiler and they need to be maintained and written. This project is meant to provide 100s of thousands of high quality tested Go libraries with no maintenance and no need to write an abstraction level.

Cgo support?

Not supported but happily accepting contributions for it!

More information

How does this compare to Gopherjs or Go wasm?

go2hx's design is built with Haxe devs in mind, therefore the goals align with Haxe dev advantages of the compiler, with that said go2hx does have some advantages already, smaller code generation, access to Haxe's compiler tooling such as dce and optimizations, and Haxe as a language being very portable, high level and statically typed.

What internals does stdlib use?

The compiler uses Haxe stdlib implementations when interacting with the file system and primitive math operations. In other cases the internals are transpiled into Haxe code and uses those.

Challenges

Didn't think your life was easy enough?

Wanted a new type of challenge to embark on... welcome!

Bootstrapping

{
    "Imports": [
        "bytes",
        "compress/zlib",
        "embed",
        "encoding/binary",
        "encoding/json",
        "fmt",
        "github.com/go2hx/go4hx/analysis",
        "go/ast",
        "go/constant",
        "go/importer",
        "go/printer",
        "go/token",
        "go/types",
        "golang.org/x/tools/go/packages",
        "golang.org/x/tools/go/types/typeutil",
        "log",
        "math/rand",
        "net",
        "os",
        "path",
        "path/filepath",
        "reflect",
        "runtime",
        "runtime/debug",
        "strconv",
        "strings"
    ]
}

Get this list of imports working along with the code and slowly allow the compiler to compile it's self.

go2hx docs