Z2 PT 9.4.0 Available for Download

Z2 PT 9.4.0 has been released in 32 and 64 bit builds for Windows and a 32-bit Linux version. More specifically, a Mint Linux 18 build. More Linux versions will be tested an supported in the future, but this being the very first Linux release, one version is more than enough as a testing workload.

Version 9.4 was delayed two weeks to fix incompatibility with Visual Studio 2010 and 2012, but this is a fairly slow task. So in the meantime we added a new class, MemoryStream, and here is where the problems started. An apparently simple line revealed serious problems with the current state of the compiler.

In order to fix this issue, we fixed 14 serious bugs, added 3 new features and did a revision of the language’s object model. These were all great steps in making the compiler significantly more stable!

But it was also time consuming and was barely finished last Wednesday. So there wasn’t any time to fix older Visual Studios. The original delay was because such incompatibility with older versions shouldn’t be a trigger point for a release, but updating the object model is reason enough, so 9.4.0 contains the above mentioned fixes, but not the VS2010/2012 fix. These fixes will come in future versions.

In addition, ZIDE got two new features: drag and drop file move and ability to kill the launched processes.

As always, a minor release will be followed by updates to it, with 9.4.1 around the corner. It already has one more important fix and 3 minor ones.

For this release, beyond the normal stuff, we are experimenting with optional opt-in newline statement terminators for the language and with precise auto-complete pop-up for ZIDE.

Advertisements

Status update for 9.4

Today was the scheduled day for PT 9.4 release. Beyond the usual enhancements and fixes, this was supposed to be the first version to ship with a Linux version and, as a bonus, also have a 64 bit client for Windows.

Things are working flawlessly with the C++ backend if you have a fairly modern compiler, be it VS2015/VS2017 on Windows or GCC on Windows and Linux. But while updating the object models that are used in this version, we arrived at some solutions that are not supported on older versions of C++.

The easy solution would be to disable those versions for this release only, but this is a minor release. Such changes should probably be done only in major releases, so the alternative is to delay 9.4.

We expect PT 9.4 to be out in two weeks.

The plan was to release today, explain the new features and also the long delay: 9.3 was released around 4 months ago. But since there is no release today, I’ll explain the delay today and keep the actual release on subject.

Since we have a well documented history on not delivering the Linux version on time and this release being late, I’d like to mention the fact that this time the two events are unrelated. Since 9.3 we were busy with another project, so for two months there was no work done on Z2.

And then came December. Around this part of the world, people will take long holidays if they can and I’ve been on holiday 3 weeks. So progress on the compiler was minimal in December.

But January was full throttle Z2 work. The previous efforts on Linux porting have payed of, and this time we managed to get the Linux version to work fully in around a week of extra work. It actually worked fairly well after a day in “hello world” scenarios, but there are hundreds of tests to be passed. This includes some custom porting and features to guarantee feature parity across operating systems.

The rest of the work was on cleaning up and enhancing the compiler, its object model, trait system and introducing the bind feature. These features will be explained in the release announcement.

Some other pieces of progress were updates to the docs and the doc generating system: both MD and HTML docs looks and behave now a lot better.

Some new parts of the compiler were open-sourced and so has ZIDE in its entirety (but with the non-functional debugging stripped out; no use submitting dead code to GitHub).

Hopefully this issue with the older compilers can be fixed on time and I’ll update again when the version is up!

 

Z2 PT 9.3.1 Available for Download

Z2 Compiler version PT 9.3.1 is now available for download under Windows!. Here are some links:
PT 9.3.1
Prepachakged GCC TDM for Windows 32/64 bit
Previous versions
Main GitHub page for standard library

This version brings minor improvements and fixes, but also fixes a massive backend compilation performance/resulting binary size issue. The fix is not that pretty but it works. Unfortunately, it is only supported for modern backends, so for older ones an even uglier fix must be found. This issue will continue to improve as versions get released.

But the main thing that this version has is better documentation. Last version we intruded the basic documentation architecture, but that was the first time we attempted anything like this so it was far from perfect. A new 2.0 documentation format has been introduced, much more readable and powerful. Additionally, more classes have been documented. This is an ongoing process and we’ll make an announcement when all of the API has been documented.

When some of the most common container algorithms were added to the standard library, the focus was on the design of the API and correctness, not on performance. So a lot of the common operations like insertion and deletion were created to set the tone of the library, but they are not particularly fast. This version we experimented with a standardized set of optimizations that will be rolled into the object model to help make the API as fast as humanly possible. The experiment show that performance has been multiplied by a double digit number in the worst case scenarios, meaning at least ten 10 faster. This applies for any subclass that can be inserted into containers. We also experimented with a particularly aggressive form of optimization that can only be applied to more special classes, where the performance gain can be more than doubled.

We left out the aggressive optimization out of 9.3.1 and only included the normal one in some very small parts as a test. If no issues are found, 9.3.2 will have both optimizations included everywhere.

So what is next for 9.3.2? Linux version of course! Except for optimization described above and smaller fixes we can include, 9.3.2 is on feature freeze and all the work will go into finally finalizing Linux support!

Here is the change-log for 9.3.1:

Z2 PT 9.3.1
=================================================================================
z2c:
– literal constants keep track of base 16 in C++ backend output
– better C++ backend else if handling
– const, mem and return nodes cleanup
– refactored literal CArrays

stdlib:
– new AsciiParser class

zide:
– new doc format
– option to regenerate all docs
– empty doc entries in the DB are marked

bugfix:
– massive backend compilation time fix for large CArray literals
– nameless constructors fixed for core types
– calling nested CArray method bug fixed

PS:

In the heat of the release announcement, I’ve completely forgot to mention the new automated build scripts. Creating a build is not that easy or fast: every time we need to go though roughly 14 steps and it can take up to 2 hours. Every build we get faster and faster at it, but you still need to do a fresh compile of 3 binaries, package sources, licenses and other files, run the basic test suite, the full test suite and do some visual QA, since we have a GUI too (our IDE) and GUIs are hard to test without somebody using their eyes.

So we created an automated script that removes some of those steps and cuts down time by 20-30 minutes. It is not that smart yet, but eventually it will do the whole process from checking out the source code to running tests, leaving just the visual QA as a manual step.

PT 9.3.1 is the first release created by the script, so if it is worse than usual or missing something, blame the scripts! All versions from now on will use the script and it will be improved each version.

Z2 PT 9.3 Available for Download

Z2 PT 9.3 is now available for download:
GitHub Win32 Prebuilt binary

There has been some restructuring and cleanup in the GitHub repository, including project renames and the use of separate isolated branches for releases, but you can find the standard library sources here:
GitHub Z2 Stdlib

There has been quite some time since the last release was announced. And there were minor releases that were not announced at all. We need to do something about lengthy pauses between releases, especially if in that period bugs are resolved. You shouldn’t have to download an old release with bugs that might have been fixed weeks or even months ago.

In order to fix this, we are restructuring releases and what they mean. Each release will now be a platform with a plan. PT 9.3 is foundation build that will receive multiple updates. Here is what to expect from 9.3 during its lifetime:

Biweekly minor releases

Every two weeks we’ll release a new minor version. If there are some major bugs solved and the solutions are stable, there might even be a weekly bugfix release, but as a general rule we’ll stick to bi-weekly. This means that two weeks from now, 9.3.1 will be released.

Language nears completion

9.3 will implement all major language features, except for lambdas and advanced meta-programming features. Delegates and reflection will be added before 9.4, to name the two largest missing features.

Documentation

If you look inside the GitHub repository, you might notice some new “*.api.md* files. These files are documentation. They are readable but not particularly pretty Markdown files which you can read directly from GitHub to get some idea on what the standard library API does.

At the same time, these file are read by ZIDE and using the actual source code, the information inside these *.md files is enhanced and presented as small documentation flash-cards when browsing a documented source code. This is only the first incarnation of the documentation system. The *.md files will become more readable, ZIDE will allow you to browse documentation in a separate web-browser like tab and the enhanced union of the *.md descriptions and the formatting extracted from the source code will be exportable as standalone HTML and PDF in order to enable the browsing of documentation outside of ZIDE.

This is only the first step in documentation, so only class APIs are documented. And not all of them yet. Each minor release will add more and more documentation, until 100% of the API is documented. And there will be also non API documentation created.

Integrated debugging

This was supposed to be the major ZIDE improvement in 9.3. If you observe the file sizes, zide.exe has gone up a bit from the previous versions in size. This is the new debugger code. Unfortunately, we were not able to get his working on time. It kind of works, but it is very crash prone. So this feature is disabled for now.

Future minor versions will re-enable it, as soon as it stops crashing. Currently, only an experimental PDB based debugger was implemented. A second GDB based one is planned.

OSSing the compiler

When Z2 was originally released, only the standard library sources were made available. The compiler and tools were distributed only as pre-built binaries. And only on Windows. We always planned to release all the sources, but we wanted the compiler to be self-hosting and buildable without any hacks. This is still the plan.

But that will still take some time, probably more than half e year. Until then, we started releasing C++ sources for the compiler. Not just a simple release, where we take that actual compiler sources and dump them to GitHub. Instead, piece by piece, the code base is cleanup up, improved and tested further. So at this time, only some parts of the compiler were released on GitHub. As the minor 9.3 versions roll out, more and more parts of the source code will be released, up to the point where a full package will be available, for everybody to compile their own full featured Z2 compiler.

Releasing the compiler sources piece by piece is a welcome opportunity to refactor it all. The code-base is starting to show its age. With the benefit of hindsight, most of algorithms and data structures that the compiler uses currently would have been replaced with better ones. The compiler is due to a major rewrite with techniques that work better. We can’t put the project on hold and rewrite it optimally, but we can do small incremental changes, refactoring one small system at a a time. As the pieces are refactored, they will make their way to GitHub.

Linux

This was promised long ago but it is still not available. 9.3 is compatible under Linux and even has some support for it, like detecting GCC, but at the end of the day, the compiler is not working under Linux. With the OSSing effort it would be silly to not go ahead and fix Linux once and for all. So 9.3 will start providing full Linux support for the compiler.

Take not that this means only the compiler. Once the compiler is ported, you can compile arbitrary Z2 code with it on Linux. This does not mean that the Z2 standard library will be ported to in the same effort. That will be a separate effort. Most of the standard library is OS agnostic, but some bits need to be ported. Things like identifying the CPU, IO, clock and environment queries all need to be ported before a full Linux SDK will be functional.

Z2C PT9.2 “Kitchen Sink” Released

For the last couple of weeks I’ve been looking at a white on black console output, checking automated unit-testing results and frantically fixing bugs every time a test failed. But the output of the text suite has been stable and promising in the last few days:

1386 out of 1386 tests have passed in a total of 330.129 seconds.

We are golden! Roll the tape with the release music…

So finally, 4 months too late, with a total of 7 months passed since the last release, a new version is finished! PT 9.2 is out! Normally, we like to keep releases smaller, having a fixed set of features and work on that until the version is stable enough. But this time we did not manage that and in the 4 months over the planned time and even in the 3 months before, a lot of non-planned features managed to receive attention. Some are in the latest release, some are not quite done (still!) and are deactivated for this release and some were so experimental that they were removed for good. So this release is more of a kitchen-sink release, hence the code-name.

Let us take a look at the main new features:

Reimplemented package system

The old design for the package system has been updated. Things are still pretty similar, but the new design is more robust, scalable and has more defined roles and rules. In addition to this conceptual and design cleanup/enhancement, the implementation has been redone pretty much from scratch.

Cache system for building

The new design for the package system needs to locate resources and this is so much more important because the entire Z2 building process is controlled by the dependency manager. So the dependency manager needs to be able to locate resources that are later used to locate other types of resources for optimal builds. All these lookups are now helped by a cache.

This cache is only one of the needed features for really fast compilation times. The goal is for Z2 to compile faster than C++, even when using the C++ backend. The second pillar of such compilation times is binary packages. Binary packages are not implemented yet, but even the cache on its own helps a lot. I will detail the results in another post.

Redesigned vectors

The existing vector design has proven to be not ideal and there has been an attempt to replace it for ages now. But vector design is really complicated and might be one of the most complicated subjects, so replacing them was no easy task.

The new design is much better and more natural to use. It is still not perfect so expect minor updates, but no more compatibility breaking major changes.

Function parameters are const

Z2 has inherited the concept of const-correctness from C++. But it does need some updates. If we can’t improve on it, maybe it is better to remove this concept from the language.

One of the improvements is the solution to the problem of mandatory const keywords in function parameters. As in C++, the de-facto standard for passing parameters to functions is by const and often forgetting to add the const is a (minor and often harmless) error. Yet the syntax for this de-facto standard is more verbose than the alternative which is rarely used.

Starting with PT9.2, all function parameters come in two flavors: const by default or reference parameters. Const parameters do not need to be decorated by a keyword and are immutable. Reference parameters are introduced by the ref keyword and they can mutate the passed-in object.

This change is experimental and at the same time quite controversial. But this is the optimal time to test its viability in practice, when the user base of the language is low.

Fancy logs

There are now new fancy HTML5 Bootstrap logs. Really fancy! With color coding, collapsible panels and navigation options.

Going over these logs can be a joy when compared to simple non-navigable logs. They are also very slow to generate and not that useful if I’m honest, so they have been deactivated for this build. It is better to compile fast rather than having Bootsrap enabled logs.

Bug fixing

While a longer development cycle can introduce more bugs, there is also more time to fix existing bugs. A ton of bugs has been fixed and Z2 has never been this stable. Probably another 4 months of bug-fixing without adding any new features and I would call the compiler production ready.

Java backend

I’m mentioning this last because the new Java backend does not work. It tries to, but it fails. But if it will work, this might add a new backend. I’m not sure if Java is a good fit for the Z2 object model and that’s putting it politely.


So what are the next steps? Well, there are two sets of plans for what will make it into PT 9.3. I can’t tell exactly which one will make it though. And after 9.3, PT 10 will come out. This should be a really major and mature version.

But until then, even with the long development cycle, PT 9.2 still has some problems:

  • Code completion. This was developed and tested in the isolating context of a code editor. It works decently in such a context. But in the context of a full IDE, it can be sometimes annoying, eating shortcut keys, popping up when not needed and so on.
  • Some const-correctness problems. The new changes to const parameters has introduced some const-correctness problems. These problems are hard to find so it shouldn’t effect potential users, but they are there.
  • Bugs. So many bugs solved, but so many new features. While beyond the 2 problems mentioned above there are no known bugs in this version (but there are of course many incomplete features we know about), I’m sure this larger release has an above average number of bugs.

In about two weeks, PT 9.2.1 will come out, addressing these and other problems!

Plans for 2017

So the Linux update sure is taking longer than planned.

Some part of the issue was with scheduling and work load, the last couple of months being particularly busy. But the real problem is that we were treading uncharted waters with the Linux release and it caused us to effectively spin our wheels in place some of the time.

So when an approach is not working, you can keep at it and see where it leads or you can try something new. We’ll try the later and hopefully this will help things. So no longer will we dedicate most of the effort to building a single working Linux version to the detriment of the compiler development and instead we will set the Linux port as a background task, to be finished and released as soon as it is done, but in the meantime continue a good release schedule for the compiler and standard library.

Second, we realized that we will never be able to cover a meaningful subset of Linux distributions with official releases so most of the Linux development effort was centered around making the compiler open-source, so that anybody could compile for the rest of the platforms if a version was not available. Currently, the compiler is mostly written in C++, but some parts are written in Z2. The bridge between them is a horrible collection of hacks. So we attempted to port everything to Z2 and release only that, with no hacks. This meant not only porting the compiler, but a whole lot of support C++ library. This was far too much of a substantial task to finish in any decent time period.

But actually this was a bit of a self defeating effort. Z2 is designed with two backends in mind, a C++ one and a LLVM one, and the C++ can offer excellent comparability, allowing both Z2 to be called form C++ code and they other way around (with some limits of course). So we will focus the effort on making the natural compatibility closer to production ready, rather than porting. After this, new parts of code will be written only in Z2 and every time a bit of C++ code gets some non-trivial update, we will rewrite that bit of code in Z2, one function at a time. And thanks to the designed comparability, this should work without any of the hacks that are currently needed. Slowly, over time, the number of Z2 functions will go up and the C++ one will go down, but in the meantime builds will continue to be hybrid. This way we don’t need to wait until we have a full port that we can open-source. Every single future version will come with the parts that are written in Z2 included in the package.

But having everything OSSed is only the beginning. This won’t really help potential users because as a fledgling language, there is ultimately an almost 0% chance of getting included into any Linux distribution. So you will have to compile yourself on all platforms that are not officially supported. And it won’t be that simple, since bootstrapping is needed. But Linux users willing to try a new programming language should have little problems with the process.

With an ever increasing projected amount of Z2 code in the compiler, it will soon come to us having to fix develop, fix bugs and debug Z2 code in a real project, the compiler itself. Doing this in C++ is easy because the tools are there and work well. Doing it in Z2 is harder, so ZIDE and the rest of the tools must become top notch. Particularly the current SCU output method, where methods are reorganized into a single file in an order that is best for the compiler but not that easy to use by a human won’t do. We’ll keep this method around, but a new more natural and traditional unit based compilation process will be introduced.

This will be a challenge since the goal is for Z2 to compile a faster than C++.

In January we want to release PT 9.2, a version that will support this new compilation method. It will also have a higher percentage of Z2 code in the compiler, this time OSSed. The library will get some additions. We also noticed that it would be possible to improve in some areas on one of the core tenets of the language: in general, there should be one way/syntax of doing a single thing, and if there are more than one, this should be particularly well explained and documented. The language is still not as lean and mean as we would like it, so some parts are getting removed. This should have pretty much zero impact on the library and the capabilities of the language, it will just become less redundant and have a more focused syntax. Part of these updates will be included in 9.2, including the new much improved and less ambiguous way of using vector literals.

Once this process is fully finished, we will release further PT 9.x version. These will have higher meta-programming capabilities and some other features. PT 10 should bring full Linux support and before 11, all source code, including the compiler should finally be OSSed.

Over the holidays I’ll try to write a new Crash Course. I am also experimenting on a larger written piece, titled “Language specs by minigun”, where I’ll try to present the language using short phrases and examples, fired at a break-neck speed. Summarizing pages form the spec into a couple of lines at a time. And hopefully once that piece is done, it will go though the hands of an editor so we can improve on the generally low level of written English on this blog.

Linux unofficial version inbound

About a month ago we released PT 9.1 on GitHub and since the repository has gotten zero commits. So it is time for a blog post to describe what has been going on ever since.

Immediately after the release, like on the same day, work started on code auto-complete for ZIDE. It is a bit rudimentary for now and only works on top-level expressions, but we focused a lot on getting it to feel just about right. ZIDE has received in the past simple code navigation capabilities which after a few more passes should do a fair job. Now it received auto-complete, which again after a few more passes will be decent. This is the pattern. The next thing we will add to ZIDE will be related to breakpoints, so you can step though your code and debug. Here is a screenshot of auto-complete in action:

auto

Slowly but surely ZIDE should become a decent editing tool. ZIDE is not on GitHub yet, so these changes did not result in any commits. But on the other hand, a first pass of cleaning it up has been done and it is fairly close to be in a good shape for OSS, so the code should be committed in about 2-3 months tops.

Next the subject of Linux was tackled for the first time! Step one was creating a development environment. Now, there are 200.000 incompatible versions of Linux out there, so we can’t just pick one up and use it with a native install. We need a flexible approach, being able to move from one Linux to another with ease, so it was decided to use virtualization.

First I tried VirtualBox, but I’m not sure if I managed to get it to work right. I’m sure it is a great tool, but my sample compiles lasted around 9 minutes. This is completely unacceptable performance. So I abandoned the idea of VirtualBox and switched over to VMWare Player. The same sample compile took 1:08 minutes, so this is what we shall be using.

Then a Linux distribution was selected: Mint Linux 18. We will try to support more popular versions, but for the first release, only one version can be supported due to time constrains. With z2c, zut and ZIDE, there are 3 tools that need to be ported, tested and maintained. Not to mention porting over the standard library.

After all this came the lengthiest part: the actual porting. The project is routinely compiled with MINGW to make sure that it is not accidentally using some Windows specific stuff, but apparently MINGW is designed to emulate Visual C++ behavior and a few things did not compile under GCC. These errors had to be fixed.

Other problems followed. Did you know that Windows uses ‘\’ and Linux uses ‘/’? We sure did. My day job is actually as a Linux programmer. So I made sure that the code base was portable, both related to ‘/’ vs ‘\’ and file name case sensitivity. But still, a couple of issues managed to squeeze though and it took a while to track them down. There are still a few minor bugs in the compilers, but it will soon be done.

ZIDE also had its fair share of small bugs, like it completely refusing to resize in both dimensions. Not a huge problem, but certainly something that is annoying and made you look quite unprofessional. There are also some minor X specific bugs with auto-complete, including the fact that pressing the PrintScreen key causes the autocomplete window to close, so creating the screenshot with it under Linux is for now impossible.

And finally, the actual porting of the library. There are still some parts to be implemented and the fixing of the above mentioned bugs, but in about one week, PT 9.1.5 should be out. It will be highly experimental, but it should detect the presence of both 32-bit and 64-bit G++. PT 9.2 should have a more robust Linux support, including Clang support.

With all this out-of-the-way, it is time to talk about Linux distribution and OSS. Our goal is to make every part of the project OSS and this is why the standard library was OSS and available from day one. But just the act of doing the initial porting to Linux took more than a month, a month in which other than auto-complete, nothing else was done. PT 9.1.5 will have the exact same features as 9.1 with the only difference being Linux support too.

Other than the standard library, all parts will be OSSed, one by one, but this will take a lot of time. Porting to Linux shut down the progress on the language by a month and OSSing everything would shut us down for over half a year. We can’t afford to not update the language and standard library for half a year, so there is only one solution.

For the next few months, Linux versions will be binary only distributions! I understand that this is a contentious subject, but there is no way around it! This is why the Linux support is marked as unofficial. It works and is getting better and better by the day, but since we don’t have the resources to do anything more than binary distribution, these releases are not official: they are available if you want to try them, if not, wait for the official release.

In conclusion, PT 9.1.5 is about to come out with its only feature being Linux support, binary distribution only, and then, PT 9.2 should have proper new features for the language and 9.4 should see the language near completion, only excluding the most advanced features.

Z2 PT 9.0 Available for Download

Here it is, our first pre-alpha release, just on time!

Well, one day latter, since when setting the release date, nobody checked to see what day it was and the 15th of August is a Holiday. No problem, the 16th is just as fine. And this isn’t even the only mistake related to this release: turns out that WordPress.com doesn’t allow you to host ZIP files and make them available for public download. So the release package will have a temporary home until a more permanent location is found:
Download z2-pt-9.0.zip for Windows 32.

As detailed in a previous post, this first release is Windows only. Future releases will be Linux too. Sorry!

So what is the status of the release? The estimation puts it 3-4 weeks behind where it was planned to be, but the delay doesn’t effect it that much. It is pretty stable all things considered. And do note, this is a pre-alpha, so don’t expect the compiler or ZIDE to be production ready. A bug-fixing patch is planned 2 weeks from now.

The delay effected the compiler and there are two known bugs in this release. For starters, aliases are so bugged that they were disabled them. Aliases are a very useful feature, but more widely used with larger projects. Our small scale testing didn’t show the bugs until it was too late. With the current implementation, there is no easy fix. What we need to do is replace this implementation with the class lookup mechanics we use everywhere else in the compiler. The reason this was not done yet because this class lookup code is scheduled for an update for PT 9.1, so working on it now would have been in wain. The second bug is related to function overloading. Last time we refactored function overloading, it was believed to be sufficient and handle all cases. Turns out there are still some issues. This will take some time to fix, so either PT 9.1 or 9.2.

The standard library came out pretty much as planned, but the two above bugs prohibited us from including the Color classes. They exhibit both the overloading bug and use aliases. There wasn’t enough time to include 3 more container classes. Including Vector, the most commonly used of them all. Scheduled for 9.1.

ZIDE was barely effected by the delay. Maybe some minor polish could have been performed, especially related to directory renaming, but it is fine for now. PT 9.1 will introduce a very early code-completion feature.

That’s about it for this post. A new post, or even better, a static page, detailing what the package contains and how to use ZIDE is in the works. That page will then be used as reference for all releases from now on so we don’t have to repeat basic instructions.

Developer Preview on August 15th

With this project it is our goal not just to create a programming language and a standard library for it, but also to do a lot of experimenting, to figure out what works best and why. This is why we followed up upon a very interesting and aggressive optimization technique for containers. You couldn’t apply it in all cases, but when you could, it was pretty much the fastest way possible of getting a new instance into a container. We were pretty sure we were onto something quite valuable here but… in the end it turned out to no work at all well with exceptions. Now, the technique is not dead and buried. It can still be used when we detect that the code can’t throw, but this pushes the feature from a valuable top tier optimization to a niche optimization you may or may not do when implementing the last features of an otherwise mature compiler.

The moral of the story is that at this stage we can’t afford to spend time on such features. We are 2 months behind schedule and there are 3 draft posts that are just waiting there for their final edits in order to get published. Additionally, because of our perfectionism, we didn’t manage to commit anything to GIT that is not related to UT. So we need to refocus and get something out there.

We will release a pre-alpha package labeled a Developer Preview on the date of August 15th. A set date is needed in order to focus the development process on high yield areas, not niche optimization. But the date is so close that this Developer Preview will not be as ambitious as it could. Future previews will be more substantial, but this one will have and intentionally reduced feature set and its goal is to be a working downloadable and testable prototype. I will now detail all the corners that will be cut in order to make this deadline.

 

The focus will be on the language

We could try to release the language as complete as stable as possible at the expense of the standard library. Or we could focus on the standard library, since the language is in a pretty good shape right now. Or maybe somewhere in the middle.

It was decided that for the first preview, the focus will be on language. During the next month, the language and the compiler will receive most of the focus and the library will only get small fixes and maybe a couple of new methods here and there.

 

Only the C++ backend will be supported

As you may know, the compiler is designed with a flexible backends and the plan is for it to eventually ship with 3 backends: LLVM, C++ and C. The LLVM is planned to be developed only after the C and C++ ones, so it was of course not going to be included in the first release, but we decided to also exclude the C one. It is significantly easier to polish a single backend per release and it will probably end up being a higher quality one this way. We are not 100% sure yet, but probably the C one will be pushed back after the LLVM completion.

We’ve been polishing the C++ backend for two weeks now and the work has greatly payed off. The old backend was designed with several profiles in mind which all balance used language features, performance of code, performance of compilation and code aesthetics. There was a profile that generated quick and extremely ugly code. One which generated extremely beautiful code meant for C++ programs to call the Z2 standard library. The new backend has done away with this and now all generated code is decent looking and carefully selected to compile and behave the same on all supported compilers. There is a minimum level of aesthetics. And the code looks a lot more natural, using a more C++ style. And an additional library mode which goes the extra mile.

 

First version will only support a single OS

As with the multiple backends, focusing on just one OS will allow us to give that version extra polish. So the first version can be either Linux or Windows. Unfortunately here we don’t get to choose: most of the standard library is platform agnostic, but some system calls are made when printing to the console or using synchronization and these are currently implemented using Windows API. So the first version will target Windows only. And only 32 bit builds. Developer preview #2 will support 32 and 64 bit builds and the Linux version.

Just to be clear, Z2 is not tied in any way to Windows. It even works today on Linux. ZIDE is build using cross-platform open-source GUI, so there are zero issues with it. The compiler is again portable and indeed does compile 100% of the test-cases under Linux. Unfortunately, it will also fail to link-edit said binaries under Linux because of the WinAPI calls that need to be replaced for Linux. Were the release date two months form now, the first version would support Linux and Windows, but since it is just one month, only Windows will be included. In 2 to 4 weeks after developer preview #1 is out, both Windows/Unix support and 32/64 bit support will be finalized and stable. And after that we shall be targeting Mac too.

 

Support for compilers out of the box will be limited to MSC and GCC

The C++ backend of course requires a C++ compiler installed on your system. We believe that it is vital that Z2 works out of the box, so you will not be left to your own devices in invoking this compiler. Z2 will autodetect the following compilers:

  • TDM GCC
  • Visual Studio 2010/MSC10
  • Visual Studio 2012/MSC11
  • Visual Studio 2013/MSC12
  • Visual Studio 2015/MSC14

For GCC, TDM GCC will be included in the package so that you can compile out of the box. This is the default build method. For Visual Studio versions, the auto-detect should pick up both the commercial and the Express/Community editions. You do need to have the Platform SDK installed. If you can compile WinAPI programs with your installed compiler, you should be fine. It will not auto-detect Visual Studio “15”/MSC15, Microsoft’s own Visual Studio “developer preview”, because it is not out yet. It will be supported when it is out, granted it still supports traditional Windows executables. The are no plans to support Windows 10 styled universal apps (UWP).

The auto-detected methods are stored in the “buildMethods.xml” configuration file, the one file that is used by all the tools inside the Z2 package that need a compiler. So if your compiler is not auto-detected or you want to use the same version of GCC/MINGW/TDM that you have installed and normally use, you can go in and edit that XML.

 

Backends will only be used in SCU mode

Well, this one is actually not true. A bit of a white lie. First, let me explain what SCU is. It stands for Single Compilation Unit. Manual SCU is pretty difficult to use, but automatic SCU can have a lot of benefits. Initially, Z2 had only support for automatic SCU. Meanwhile, we transitioned to multiple CUs that are still automatic. This has more advantages. So what we mean by SCU mode is that it is automatic and you have no control over your compilation units. SCU must be retroactively made to stand for “Smart Compilation Units”.

Now, in a more mature Z2 compiler you can optimally have full control of your compilation units and deactivate/fine tune the automatic system. This feature will not be supported in dev preview #1 and you will have to do with SCU.

 

There will be bugs!

In an ideal world, the package you get on 15th will have zero bugs. But that is highly unlikely. We’ll fix as many bugs we can, but some will get though. Also, some language features might not make it in their final form. There will probably be very few breaking changes in the future, but this is not a guarantee. Developer preview #1 is pre-alpha after all!

GitHub repository is up

The official GitHub repository is up and can be found at: https://github.com/MasterZean/z2c.

For the first phase, this repository will hold the standard library source code and some other related code, like unit testing code, documentation and benchmarks. The license is “Apache License Version 2.0”. To be honest, I have personally studied licenses in my off-time for two weeks and have reached the conclusion that not only do you need to be a lawyer to truly discern 100% of the real life implications of open-source licenses, but you also need to consult with other lawyers too. So what I’m saying is that while we do like the general principles behind open-source and we want to open-source the code, we are not married or feel strongly towards any of the individual license offerings, including Apache License Version 2.0, which may be transitory. Additionally, choosing the absolute best license at this point is beyond our means.

A few first commits were made to the repository, but for now only UT code has been added. But not the interesting UT code, but the boring kind. If you wanted to show the language to somebody by code, the UT code would be a good place to start, since it is a bunch of relatively short snippets of code, each showing off and testing some language feature sometimes in isolation, sometimes testing their flow together. In consequence the UT code can be interesting. But not the one we committed. With the amount of refactoring in PT7, things broke often and in non-obvious ways, especially when it comes to function overloading. So we added about 50 new tests, all for single parameter overloading, creating a very complete coverage of numerical types so that we can have some measure of security that overloading never breaks again. Probably about 10 tests would have been sufficient, but maybe 50 is safer.

We’ll continue to add tests to the “master” branch of the repository, but probably not 50 in one go. But it is best to add enough tests at once to reasonably cover one small feature or API element at a time.

A branch has been created with the code from “CrashCourse – 004 – Building an Int“. The next 2 posts in this series will evolve a few classes closer to their final form while explaining design decisions and once this is done, the branch will be merged into “master”. After this, the real standard library classes will start to be added to the repository, one by one, as they are documented. The documentation infrastructure still needs some work. We have documentation in the source files using comments and XML, but we would also like to evolve this tried and true formula to also work with the exact same XML tags, but externally to the source code, so that you have the choice of documenting code on the spot, with the trade-off of making the code harder to navigate, or having the documentation fully/partially in an outside file.

Now that a few pieces of code are in the repository and soon more will come, we are forced to release a super-alpha version of ZIDE. As mentioned before, the goal of ZIDE is to offer a minimum golden standard of features out of the box for editing and building software using Z2, so you don’t have to resolve to ad-hoc solutions and command line. So a version of ZIDE must exist as long as there is Z2 code, and now there is.

At the begging of April, a super-early alpha version of ZIDE will be made available, meant for developers. It won’t have many features and it will be buggy, but hopefully this early release will help us to make it better based on feedback. Unfortunately, we don’t have the time or resources for multi-platform releases right now, so this first version will only be available on Windows. Starting with the second or third release, we will have a release for Linux too.