Compiler versioning system

The Z2 compiler, called z2c has almost reached its next internal stable version, PT6. It is in the fixing stage of the last few known bugs and undergoing additional testing and should be done before Christmas. Since the detailed change-log is lost (see the first post and the blog) and the change-log is meaningless anyway to people who are not already using the compiler, I thought this would be a great moment to detail the versioning scheme, release structure and schedule of the compiler.

The compiler is developed in two stages. The first stage will have the full set of core features and a full standard library and will officially be labeled “1.0”. Stage two will enhance the language with some optional meta-programming related features and add a few others inspired by some scripting languages. Stage two can be safely ignored for now. While we do have the exact plan on what features we want to add into stage two, development on them won’t begin until stage one is done.

For stage one, we are labeling each internal compiler version with PT##, where ## is a number. PT1 though PT5 are done and PT6 is just about fished. PT comes from “pre-tier”, signaling the alpha unreleased quality of the software.

Depending on the progress we actually achieve, either PT8 or PT9 will be the first version available for public preview, so right now it is not possible to download a public working compiler from anywhere. Since these milestones are already labeled with “pre”, there is no use to create special preview releases. You’ll be able to download the exact compiler package we use for testing. Starting with PT10, the software should reach beta level quality and by PT15 we’d like to reach 1.0.

Each compiler package comes with at least the following components:

  • z2c: the command line compiler and full build tool. The compiler is also a build tool capable of building arbitrarily complex projects out of the box with a single execution of the tool.
  • zide: a GUI cross platform IDE designed to offer a minimal golden standard of features to new adopters of the language who wish to edit code using an IDE.
  • zut: a command line unit-test execution tool, serving as a sanity check for the compiler and standard library.
  • a full copy of the standard library as source code
  • all documentation as present at the moment of release

Most of these tools are self explanatory, but I’d like to focus a bit on ZIDE, the Z2 IDE. During the past few years, several programming languages have cropped up, some broad scoped, some niche, some popular, some not. But in general, when a new language pops up, especially if it is not a scripting language, the tool support at release is very poor. It can take a lot of time before even a few syntax highlighters crop up for some popular IDEs and it can take years before any decent tools get created. It is very common for somebody picking up such a language to have to edit code in some random editor that does not understand the syntax of the language, have to compile in the command line because said editor can’t launch the appropriate compilation commands and be forced to use “printf” debugging, since a real debugger with breakpoints and watches (as a bare minimum) is years away.

It is our goal to eliminate this problem by having the official package come with an IDE, called ZIDE. Since PT5 the package includes this GUI editor and in each PT it is getting improved as part of the official development schedule. The command line compiler is married to ZIDE as development efforts go, so even if the compiler and library are 100% ready for the next release, if ZIDE has not been updated, we will delay the release until some new features have been added to it. ZIDE is supported on all platforms where the compiler is officially supported and requires a X server or Windows for now. It can be made to work under Cocoa, but no such implementation effort is currently under way.

ZIDE is really not meant to be the best IDE in the world. Hopefully, other better tools will be created by third parties. ZIDE is there to offer a few much needed features and conveniences from day one! A few major features like syntax highlighting, code browsing, project creation and navigation, auto-complete, compiling and debugging are considered by us as being a worthwhile development effort. An early adopter will not be forced to use whatever tools one can muster because ZIDE will always be there if needed. To make sure that ZIDE is pleasant to use and has all the necessary features, the Z2 standard library is fully developed using ZIDE and ZIDE only. Additionally, non-automated testing is also done mostly using ZIDE.

Now that I presented what is in the compiler package, I shall finish by describing how it compiles.

Z2 uses the common IR paradigm (intermediate representation). The compiler has a front-end, which when compiling source code will create an IR tree. And with this the job of the front-end is done. A back-end is needed to output some meaningful machine code based on IR.

The compiler does not have a back-end capable of outputting machine code and this is intentional. It wouldn’t take use more than a couple of months to create one such back-end and it would function properly, but it would produce terribly un-optimized machine code and would also be completely tied to one single machine architecture. Even with a team 20 times larger and working for years, it would still be highly improbable to produce a better optimized machine code generator than some of the solutions that are available today. GCC has been in development for 28 years and LLVM for 12, just to name a couple. So we want the back-ends to use existing solutions for machine code generation. Solutions supporting every major combination of architecture and operating system, while still being capable of outputting highly optimized code.

In order to achieve this, we are developing two back-ends: one that is looking back and one that is looking forward.

The one that is looking back is the C/C++ back-end. This back-end converts IR to C/C++. This is not a binary switch between C and C++: there is a whole spectrum of options related to what kind of C/C++ constructs to use and which parts of the code base to output. As an example, one of the configurations outputs C++ code with maximal formatting, meant to be as readable as hand written code and includes all the code available in a package. This option is meant to make Z2 code available as a library for C++ projects. This way, the Z2 standard library can be used from C++ without having to maintain two libraries for two programming languages. Using ghost-classes and this back-end, it is also possible for Z2 code to call C++ code, for maximum interoperability. Another example is a set of options that produces very short and ugly code, without any meaningful formatting and using short mangled names. This option is used when you only care about the resulting binary and want to potentially speed up compilation. And pretty much everything in between these two extremes is supported.

Since this back-end outputs C/C++ code, but Z2 tries to offer convenience features like ZIDE, it won’t let you to your own devices to compile the resulting C/C++ code. The command line compiler will detect installed GCC versions on Linux and use them to compile the resulting back-end output. Under Windows, a package optionally bundled with a MINGW will be available for binary generation out of the box. Alternatively, Visual Studio versions 7.1, 8, 9, 10, 11 and 12 are auto-detected and can be selected for compilation. These build methods can be edited or added to after detection in order to support custom paths towards other compilers.

The second back-end, the one looking forwards is the LLVM back-end. In consequence it supports the features, code generation models and general interoperability capabilites of LLVM. Currently, this back-end is in early stages of prototyping and we will first finish the C/C++ back-end 100% and then focus fully on the LLVM back-end.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s