Z2 PT 9.3 Available for Download

Z2 PT 9.3 is now available for download:
GitHub Win32 Prebuilt binary

There has been some restructuring and cleanup in the GitHub repository, including project renames and the use of separate isolated branches for releases, but you can find the standard library sources here:
GitHub Z2 Stdlib

There has been quite some time since the last release was announced. And there were minor releases that were not announced at all. We need to do something about lengthy pauses between releases, especially if in that period bugs are resolved. You shouldn’t have to download an old release with bugs that might have been fixed weeks or even months ago.

In order to fix this, we are restructuring releases and what they mean. Each release will now be a platform with a plan. PT 9.3 is foundation build that will receive multiple updates. Here is what to expect from 9.3 during its lifetime:

Biweekly minor releases

Every two weeks we’ll release a new minor version. If there are some major bugs solved and the solutions are stable, there might even be a weekly bugfix release, but as a general rule we’ll stick to bi-weekly. This means that two weeks from now, 9.3.1 will be released.

Language nears completion

9.3 will implement all major language features, except for lambdas and advanced meta-programming features. Delegates and reflection will be added before 9.4, to name the two largest missing features.

Dcoumentation

If you look inside the GitHub repository, you might notice some new “*.api.md* files. These files are documentation. They are readable but not particularly pretty Markdown files which you can read directly from GitHub to get some idea on what the standard library API does.

At the same time, these file are read by ZIDE and using the actual source code, the information inside these *.md files is enhanced and presented as small documentation flash-cards when browsing a documented source code. This is only the first incarnation of the documentation system. The *.md files will become more readable, ZIDE will allow you to browse documentation in a separate web-browser like tab and the enhanced union of the *.md descriptions and the formatting extracted from the source code will be exportable as standalone HTML and PDF in order to enable the browsing of documentation outside of ZIDE.

This is only the first step in documentation, so only class APIs are documented. And not all of them yet. Each minor release will add more and more documentation, until 100% of the API is documented. And there will be also non API documentation created.

Integrated debugging

This was supposed to be the major ZIDE improvement in 9.3. If you observe the file sizes, zide.exe has gone up a bit from the previous versions in size. This is the new debugger code. Unfortunately, we were not able to get his working on time. It kind of works, but it is very crash prone. So this feature is disabled for now.

Future minor versions will re-enable it, as soon as it stops crashing. Currently, only an experimental PDB based debugger was implemented. A second GDB based one is planned.

OSSing the compiler

When Z2 was originally released, only the standard library sources were made available. The compiler and tools were distributed only as pre-built binaries. And only on Windows. We always planned to release all the sources, but we wanted the compiler to be self-hosting and buildable without any hacks. This is still the plan.

But that will still take some time, probably more than half e year. Until then, we started releasing C++ sources for the compiler. Not just a simple release, where we take that actual compiler sources and dump them to GitHub. Instead, piece by piece, the code base is cleanup up, improved and tested further. So at this time, only some parts of the compiler were released on GitHub. As the minor 9.3 versions roll out, more and more parts of the source code will be released, up to the point where a full package will be available, for everybody to compile their own full featured Z2 compiler.

Releasing the compiler sources piece by piece is a welcome opportunity to refactor it all. The code-base is starting to show its age. With the benefit of hindsight, most of algorithms and data structures that the compiler uses currently would have been replaced with better ones. The compiler is due to a major rewrite with techniques that work better. We can’t put the project on hold and rewrite it optimally, but we can do small incremental changes, refactoring one small system at a a time. As the pieces are refactored, they will make their way to GitHub.

Linux

This was promised long ago but it is still not available. 9.3 is compatible under Linux and even has some support for it, like detecting GCC, but at the end of the day, the compiler is not working under Linux. With the OSSing effort it would be silly to not go ahead and fix Linux once and for all. So 9.3 will start providing full Linux support for the compiler.

Take not that this means only the compiler. Once the compiler is ported, you can compile arbitrary Z2 code with it on Linux. This does not mean that the Z2 standard library will be ported to in the same effort. That will be a separate effort. Most of the standard library is OS agnostic, but some bits need to be ported. Things like identifying the CPU, IO, clock and environment queries all need to be ported before a full Linux SDK will be functional.

Z2C PT9.2 “Kitchen Sink” Released

For the last couple of weeks I’ve been looking at a white on black console output, checking automated unit-testing results and frantically fixing bugs every time a test failed. But the output of the text suite has been stable and promising in the last few days:

1386 out of 1386 tests have passed in a total of 330.129 seconds.

We are golden! Roll the tape with the release music…

So finally, 4 months too late, with a total of 7 months passed since the last release, a new version is finished! PT 9.2 is out! Normally, we like to keep releases smaller, having a fixed set of features and work on that until the version is stable enough. But this time we did not manage that and in the 4 months over the planned time and even in the 3 months before, a lot of non-planned features managed to receive attention. Some are in the latest release, some are not quite done (still!) and are deactivated for this release and some were so experimental that they were removed for good. So this release is more of a kitchen-sink release, hence the code-name.

Let us take a look at the main new features:

Reimplemented package system

The old design for the package system has been updated. Things are still pretty similar, but the new design is more robust, scalable and has more defined roles and rules. In addition to this conceptual and design cleanup/enhancement, the implementation has been redone pretty much from scratch.

Cache system for building

The new design for the package system needs to locate resources and this is so much more important because the entire Z2 building process is controlled by the dependency manager. So the dependency manager needs to be able to locate resources that are later used to locate other types of resources for optimal builds. All these lookups are now helped by a cache.

This cache is only one of the needed features for really fast compilation times. The goal is for Z2 to compile faster than C++, even when using the C++ backend. The second pillar of such compilation times is binary packages. Binary packages are not implemented yet, but even the cache on its own helps a lot. I will detail the results in another post.

Redesigned vectors

The existing vector design has proven to be not ideal and there has been an attempt to replace it for ages now. But vector design is really complicated and might be one of the most complicated subjects, so replacing them was no easy task.

The new design is much better and more natural to use. It is still not perfect so expect minor updates, but no more compatibility breaking major changes.

Function parameters are const

Z2 has inherited the concept of const-correctness from C++. But it does need some updates. If we can’t improve on it, maybe it is better to remove this concept from the language.

One of the improvements is the solution to the problem of mandatory const keywords in function parameters. As in C++, the de-facto standard for passing parameters to functions is by const and often forgetting to add the const is a (minor and often harmless) error. Yet the syntax for this de-facto standard is more verbose than the alternative which is rarely used.

Starting with PT9.2, all function parameters come in two flavors: const by default or reference parameters. Const parameters do not need to be decorated by a keyword and are immutable. Reference parameters are introduced by the ref keyword and they can mutate the passed-in object.

This change is experimental and at the same time quite controversial. But this is the optimal time to test its viability in practice, when the user base of the language is low.

Fancy logs

There are now new fancy HTML5 Bootstrap logs. Really fancy! With color coding, collapsible panels and navigation options.

Going over these logs can be a joy when compared to simple non-navigable logs. They are also very slow to generate and not that useful if I’m honest, so they have been deactivated for this build. It is better to compile fast rather than having Bootsrap enabled logs.

Bug fixing

While a longer development cycle can introduce more bugs, there is also more time to fix existing bugs. A ton of bugs has been fixed and Z2 has never been this stable. Probably another 4 months of bug-fixing without adding any new features and I would call the compiler production ready.

Java backend

I’m mentioning this last because the new Java backend does not work. It tries to, but it fails. But if it will work, this might add a new backend. I’m not sure if Java is a good fit for the Z2 object model and that’s putting it politely.


So what are the next steps? Well, there are two sets of plans for what will make it into PT 9.3. I can’t tell exactly which one will make it though. And after 9.3, PT 10 will come out. This should be a really major and mature version.

But until then, even with the long development cycle, PT 9.2 still has some problems:

  • Code completion. This was developed and tested in the isolating context of a code editor. It works decently in such a context. But in the context of a full IDE, it can be sometimes annoying, eating shortcut keys, popping up when not needed and so on.
  • Some const-correctness problems. The new changes to const parameters has introduced some const-correctness problems. These problems are hard to find so it shouldn’t effect potential users, but they are there.
  • Bugs. So many bugs solved, but so many new features. While beyond the 2 problems mentioned above there are no known bugs in this version (but there are of course many incomplete features we know about), I’m sure this larger release has an above average number of bugs.

In about two weeks, PT 9.2.1 will come out, addressing these and other problems!

Plans for 2017

So the Linux update sure is taking longer than planned.

Some part of the issue was with scheduling and work load, the last couple of months being particularly busy. But the real problem is that we were treading uncharted waters with the Linux release and it caused us to effectively spin our wheels in place some of the time.

So when an approach is not working, you can keep at it and see where it leads or you can try something new. We’ll try the later and hopefully this will help things. So no longer will we dedicate most of the effort to building a single working Linux version to the detriment of the compiler development and instead we will set the Linux port as a background task, to be finished and released as soon as it is done, but in the meantime continue a good release schedule for the compiler and standard library.

Second, we realized that we will never be able to cover a meaningful subset of Linux distributions with official releases so most of the Linux development effort was centered around making the compiler open-source, so that anybody could compile for the rest of the platforms if a version was not available. Currently, the compiler is mostly written in C++, but some parts are written in Z2. The bridge between them is a horrible collection of hacks. So we attempted to port everything to Z2 and release only that, with no hacks. This meant not only porting the compiler, but a whole lot of support C++ library. This was far too much of a substantial task to finish in any decent time period.

But actually this was a bit of a self defeating effort. Z2 is designed with two backends in mind, a C++ one and a LLVM one, and the C++ can offer excellent comparability, allowing both Z2 to be called form C++ code and they other way around (with some limits of course). So we will focus the effort on making the natural compatibility closer to production ready, rather than porting. After this, new parts of code will be written only in Z2 and every time a bit of C++ code gets some non-trivial update, we will rewrite that bit of code in Z2, one function at a time. And thanks to the designed comparability, this should work without any of the hacks that are currently needed. Slowly, over time, the number of Z2 functions will go up and the C++ one will go down, but in the meantime builds will continue to be hybrid. This way we don’t need to wait until we have a full port that we can open-source. Every single future version will come with the parts that are written in Z2 included in the package.

But having everything OSSed is only the beginning. This won’t really help potential users because as a fledgling language, there is ultimately an almost 0% chance of getting included into any Linux distribution. So you will have to compile yourself on all platforms that are not officially supported. And it won’t be that simple, since bootstrapping is needed. But Linux users willing to try a new programming language should have little problems with the process.

With an ever increasing projected amount of Z2 code in the compiler, it will soon come to us having to fix develop, fix bugs and debug Z2 code in a real project, the compiler itself. Doing this in C++ is easy because the tools are there and work well. Doing it in Z2 is harder, so ZIDE and the rest of the tools must become top notch. Particularly the current SCU output method, where methods are reorganized into a single file in an order that is best for the compiler but not that easy to use by a human won’t do. We’ll keep this method around, but a new more natural and traditional unit based compilation process will be introduced.

This will be a challenge since the goal is for Z2 to compile a faster than C++.

In January we want to release PT 9.2, a version that will support this new compilation method. It will also have a higher percentage of Z2 code in the compiler, this time OSSed. The library will get some additions. We also noticed that it would be possible to improve in some areas on one of the core tenets of the language: in general, there should be one way/syntax of doing a single thing, and if there are more than one, this should be particularly well explained and documented. The language is still not as lean and mean as we would like it, so some parts are getting removed. This should have pretty much zero impact on the library and the capabilities of the language, it will just become less redundant and have a more focused syntax. Part of these updates will be included in 9.2, including the new much improved and less ambiguous way of using vector literals.

Once this process is fully finished, we will release further PT 9.x version. These will have higher meta-programming capabilities and some other features. PT 10 should bring full Linux support and before 11, all source code, including the compiler should finally be OSSed.

Over the holidays I’ll try to write a new Crash Course. I am also experimenting on a larger written piece, titled “Language specs by minigun”, where I’ll try to present the language using short phrases and examples, fired at a break-neck speed. Summarizing pages form the spec into a couple of lines at a time. And hopefully once that piece is done, it will go though the hands of an editor so we can improve on the generally low level of written English on this blog.

Linux unofficial version inbound

About a month ago we released PT 9.1 on GitHub and since the repository has gotten zero commits. So it is time for a blog post to describe what has been going on ever since.

Immediately after the release, like on the same day, work started on code auto-complete for ZIDE. It is a bit rudimentary for now and only works on top-level expressions, but we focused a lot on getting it to feel just about right. ZIDE has received in the past simple code navigation capabilities which after a few more passes should do a fair job. Now it received auto-complete, which again after a few more passes will be decent. This is the pattern. The next thing we will add to ZIDE will be related to breakpoints, so you can step though your code and debug. Here is a screenshot of auto-complete in action:

auto

Slowly but surely ZIDE should become a decent editing tool. ZIDE is not on GitHub yet, so these changes did not result in any commits. But on the other hand, a first pass of cleaning it up has been done and it is fairly close to be in a good shape for OSS, so the code should be committed in about 2-3 months tops.

Next the subject of Linux was tackled for the first time! Step one was creating a development environment. Now, there are 200.000 incompatible versions of Linux out there, so we can’t just pick one up and use it with a native install. We need a flexible approach, being able to move from one Linux to another with ease, so it was decided to use virtualization.

First I tried VirtualBox, but I’m not sure if I managed to get it to work right. I’m sure it is a great tool, but my sample compiles lasted around 9 minutes. This is completely unacceptable performance. So I abandoned the idea of VirtualBox and switched over to VMWare Player. The same sample compile took 1:08 minutes, so this is what we shall be using.

Then a Linux distribution was selected: Mint Linux 18. We will try to support more popular versions, but for the first release, only one version can be supported due to time constrains. With z2c, zut and ZIDE, there are 3 tools that need to be ported, tested and maintained. Not to mention porting over the standard library.

After all this came the lengthiest part: the actual porting. The project is routinely compiled with MINGW to make sure that it is not accidentally using some Windows specific stuff, but apparently MINGW is designed to emulate Visual C++ behavior and a few things did not compile under GCC. These errors had to be fixed.

Other problems followed. Did you know that Windows uses ‘\’ and Linux uses ‘/’? We sure did. My day job is actually as a Linux programmer. So I made sure that the code base was portable, both related to ‘/’ vs ‘\’ and file name case sensitivity. But still, a couple of issues managed to squeeze though and it took a while to track them down. There are still a few minor bugs in the compilers, but it will soon be done.

ZIDE also had its fair share of small bugs, like it completely refusing to resize in both dimensions. Not a huge problem, but certainly something that is annoying and made you look quite unprofessional. There are also some minor X specific bugs with auto-complete, including the fact that pressing the PrintScreen key causes the autocomplete window to close, so creating the screenshot with it under Linux is for now impossible.

And finally, the actual porting of the library. There are still some parts to be implemented and the fixing of the above mentioned bugs, but in about one week, PT 9.1.5 should be out. It will be highly experimental, but it should detect the presence of both 32-bit and 64-bit G++. PT 9.2 should have a more robust Linux support, including Clang support.

With all this out-of-the-way, it is time to talk about Linux distribution and OSS. Our goal is to make every part of the project OSS and this is why the standard library was OSS and available from day one. But just the act of doing the initial porting to Linux took more than a month, a month in which other than auto-complete, nothing else was done. PT 9.1.5 will have the exact same features as 9.1 with the only difference being Linux support too.

Other than the standard library, all parts will be OSSed, one by one, but this will take a lot of time. Porting to Linux shut down the progress on the language by a month and OSSing everything would shut us down for over half a year. We can’t afford to not update the language and standard library for half a year, so there is only one solution.

For the next few months, Linux versions will be binary only distributions! I understand that this is a contentious subject, but there is no way around it! This is why the Linux support is marked as unofficial. It works and is getting better and better by the day, but since we don’t have the resources to do anything more than binary distribution, these releases are not official: they are available if you want to try them, if not, wait for the official release.

In conclusion, PT 9.1.5 is about to come out with its only feature being Linux support, binary distribution only, and then, PT 9.2 should have proper new features for the language and 9.4 should see the language near completion, only excluding the most advanced features.

Z2 PT 9.0 Available for Download

Here it is, our first pre-alpha release, just on time!

Well, one day latter, since when setting the release date, nobody checked to see what day it was and the 15th of August is a Holiday. No problem, the 16th is just as fine. And this isn’t even the only mistake related to this release: turns out that WordPress.com doesn’t allow you to host ZIP files and make them available for public download. So the release package will have a temporary home until a more permanent location is found:
Download z2-pt-9.0.zip for Windows 32.

As detailed in a previous post, this first release is Windows only. Future releases will be Linux too. Sorry!

So what is the status of the release? The estimation puts it 3-4 weeks behind where it was planned to be, but the delay doesn’t effect it that much. It is pretty stable all things considered. And do note, this is a pre-alpha, so don’t expect the compiler or ZIDE to be production ready. A bug-fixing patch is planned 2 weeks from now.

The delay effected the compiler and there are two known bugs in this release. For starters, aliases are so bugged that they were disabled them. Aliases are a very useful feature, but more widely used with larger projects. Our small scale testing didn’t show the bugs until it was too late. With the current implementation, there is no easy fix. What we need to do is replace this implementation with the class lookup mechanics we use everywhere else in the compiler. The reason this was not done yet because this class lookup code is scheduled for an update for PT 9.1, so working on it now would have been in wain. The second bug is related to function overloading. Last time we refactored function overloading, it was believed to be sufficient and handle all cases. Turns out there are still some issues. This will take some time to fix, so either PT 9.1 or 9.2.

The standard library came out pretty much as planned, but the two above bugs prohibited us from including the Color classes. They exhibit both the overloading bug and use aliases. There wasn’t enough time to include 3 more container classes. Including Vector, the most commonly used of them all. Scheduled for 9.1.

ZIDE was barely effected by the delay. Maybe some minor polish could have been performed, especially related to directory renaming, but it is fine for now. PT 9.1 will introduce a very early code-completion feature.

That’s about it for this post. A new post, or even better, a static page, detailing what the package contains and how to use ZIDE is in the works. That page will then be used as reference for all releases from now on so we don’t have to repeat basic instructions.

Developer Preview on August 15th

With this project it is our goal not just to create a programming language and a standard library for it, but also to do a lot of experimenting, to figure out what works best and why. This is why we followed up upon a very interesting and aggressive optimization technique for containers. You couldn’t apply it in all cases, but when you could, it was pretty much the fastest way possible of getting a new instance into a container. We were pretty sure we were onto something quite valuable here but… in the end it turned out to no work at all well with exceptions. Now, the technique is not dead and buried. It can still be used when we detect that the code can’t throw, but this pushes the feature from a valuable top tier optimization to a niche optimization you may or may not do when implementing the last features of an otherwise mature compiler.

The moral of the story is that at this stage we can’t afford to spend time on such features. We are 2 months behind schedule and there are 3 draft posts that are just waiting there for their final edits in order to get published. Additionally, because of our perfectionism, we didn’t manage to commit anything to GIT that is not related to UT. So we need to refocus and get something out there.

We will release a pre-alpha package labeled a Developer Preview on the date of August 15th. A set date is needed in order to focus the development process on high yield areas, not niche optimization. But the date is so close that this Developer Preview will not be as ambitious as it could. Future previews will be more substantial, but this one will have and intentionally reduced feature set and its goal is to be a working downloadable and testable prototype. I will now detail all the corners that will be cut in order to make this deadline.

 

The focus will be on the language

We could try to release the language as complete as stable as possible at the expense of the standard library. Or we could focus on the standard library, since the language is in a pretty good shape right now. Or maybe somewhere in the middle.

It was decided that for the first preview, the focus will be on language. During the next month, the language and the compiler will receive most of the focus and the library will only get small fixes and maybe a couple of new methods here and there.

 

Only the C++ backend will be supported

As you may know, the compiler is designed with a flexible backends and the plan is for it to eventually ship with 3 backends: LLVM, C++ and C. The LLVM is planned to be developed only after the C and C++ ones, so it was of course not going to be included in the first release, but we decided to also exclude the C one. It is significantly easier to polish a single backend per release and it will probably end up being a higher quality one this way. We are not 100% sure yet, but probably the C one will be pushed back after the LLVM completion.

We’ve been polishing the C++ backend for two weeks now and the work has greatly payed off. The old backend was designed with several profiles in mind which all balance used language features, performance of code, performance of compilation and code aesthetics. There was a profile that generated quick and extremely ugly code. One which generated extremely beautiful code meant for C++ programs to call the Z2 standard library. The new backend has done away with this and now all generated code is decent looking and carefully selected to compile and behave the same on all supported compilers. There is a minimum level of aesthetics. And the code looks a lot more natural, using a more C++ style. And an additional library mode which goes the extra mile.

 

First version will only support a single OS

As with the multiple backends, focusing on just one OS will allow us to give that version extra polish. So the first version can be either Linux or Windows. Unfortunately here we don’t get to choose: most of the standard library is platform agnostic, but some system calls are made when printing to the console or using synchronization and these are currently implemented using Windows API. So the first version will target Windows only. And only 32 bit builds. Developer preview #2 will support 32 and 64 bit builds and the Linux version.

Just to be clear, Z2 is not tied in any way to Windows. It even works today on Linux. ZIDE is build using cross-platform open-source GUI, so there are zero issues with it. The compiler is again portable and indeed does compile 100% of the test-cases under Linux. Unfortunately, it will also fail to link-edit said binaries under Linux because of the WinAPI calls that need to be replaced for Linux. Were the release date two months form now, the first version would support Linux and Windows, but since it is just one month, only Windows will be included. In 2 to 4 weeks after developer preview #1 is out, both Windows/Unix support and 32/64 bit support will be finalized and stable. And after that we shall be targeting Mac too.

 

Support for compilers out of the box will be limited to MSC and GCC

The C++ backend of course requires a C++ compiler installed on your system. We believe that it is vital that Z2 works out of the box, so you will not be left to your own devices in invoking this compiler. Z2 will autodetect the following compilers:

  • TDM GCC
  • Visual Studio 2010/MSC10
  • Visual Studio 2012/MSC11
  • Visual Studio 2013/MSC12
  • Visual Studio 2015/MSC14

For GCC, TDM GCC will be included in the package so that you can compile out of the box. This is the default build method. For Visual Studio versions, the auto-detect should pick up both the commercial and the Express/Community editions. You do need to have the Platform SDK installed. If you can compile WinAPI programs with your installed compiler, you should be fine. It will not auto-detect Visual Studio “15”/MSC15, Microsoft’s own Visual Studio “developer preview”, because it is not out yet. It will be supported when it is out, granted it still supports traditional Windows executables. The are no plans to support Windows 10 styled universal apps (UWP).

The auto-detected methods are stored in the “buildMethods.xml” configuration file, the one file that is used by all the tools inside the Z2 package that need a compiler. So if your compiler is not auto-detected or you want to use the same version of GCC/MINGW/TDM that you have installed and normally use, you can go in and edit that XML.

 

Backends will only be used in SCU mode

Well, this one is actually not true. A bit of a white lie. First, let me explain what SCU is. It stands for Single Compilation Unit. Manual SCU is pretty difficult to use, but automatic SCU can have a lot of benefits. Initially, Z2 had only support for automatic SCU. Meanwhile, we transitioned to multiple CUs that are still automatic. This has more advantages. So what we mean by SCU mode is that it is automatic and you have no control over your compilation units. SCU must be retroactively made to stand for “Smart Compilation Units”.

Now, in a more mature Z2 compiler you can optimally have full control of your compilation units and deactivate/fine tune the automatic system. This feature will not be supported in dev preview #1 and you will have to do with SCU.

 

There will be bugs!

In an ideal world, the package you get on 15th will have zero bugs. But that is highly unlikely. We’ll fix as many bugs we can, but some will get though. Also, some language features might not make it in their final form. There will probably be very few breaking changes in the future, but this is not a guarantee. Developer preview #1 is pre-alpha after all!

GitHub repository is up

The official GitHub repository is up and can be found at: https://github.com/MasterZean/z2c.

For the first phase, this repository will hold the standard library source code and some other related code, like unit testing code, documentation and benchmarks. The license is “Apache License Version 2.0”. To be honest, I have personally studied licenses in my off-time for two weeks and have reached the conclusion that not only do you need to be a lawyer to truly discern 100% of the real life implications of open-source licenses, but you also need to consult with other lawyers too. So what I’m saying is that while we do like the general principles behind open-source and we want to open-source the code, we are not married or feel strongly towards any of the individual license offerings, including Apache License Version 2.0, which may be transitory. Additionally, choosing the absolute best license at this point is beyond our means.

A few first commits were made to the repository, but for now only UT code has been added. But not the interesting UT code, but the boring kind. If you wanted to show the language to somebody by code, the UT code would be a good place to start, since it is a bunch of relatively short snippets of code, each showing off and testing some language feature sometimes in isolation, sometimes testing their flow together. In consequence the UT code can be interesting. But not the one we committed. With the amount of refactoring in PT7, things broke often and in non-obvious ways, especially when it comes to function overloading. So we added about 50 new tests, all for single parameter overloading, creating a very complete coverage of numerical types so that we can have some measure of security that overloading never breaks again. Probably about 10 tests would have been sufficient, but maybe 50 is safer.

We’ll continue to add tests to the “master” branch of the repository, but probably not 50 in one go. But it is best to add enough tests at once to reasonably cover one small feature or API element at a time.

A branch has been created with the code from “CrashCourse – 004 – Building an Int“. The next 2 posts in this series will evolve a few classes closer to their final form while explaining design decisions and once this is done, the branch will be merged into “master”. After this, the real standard library classes will start to be added to the repository, one by one, as they are documented. The documentation infrastructure still needs some work. We have documentation in the source files using comments and XML, but we would also like to evolve this tried and true formula to also work with the exact same XML tags, but externally to the source code, so that you have the choice of documenting code on the spot, with the trade-off of making the code harder to navigate, or having the documentation fully/partially in an outside file.

Now that a few pieces of code are in the repository and soon more will come, we are forced to release a super-alpha version of ZIDE. As mentioned before, the goal of ZIDE is to offer a minimum golden standard of features out of the box for editing and building software using Z2, so you don’t have to resolve to ad-hoc solutions and command line. So a version of ZIDE must exist as long as there is Z2 code, and now there is.

At the begging of April, a super-early alpha version of ZIDE will be made available, meant for developers. It won’t have many features and it will be buggy, but hopefully this early release will help us to make it better based on feedback. Unfortunately, we don’t have the time or resources for multi-platform releases right now, so this first version will only be available on Windows. Starting with the second or third release, we will have a release for Linux too.

Z2 Compiler PT7 in feature freeze

There has been a lack of new information on the blog in 2016. Sorry, I didn’t have time to write posts since I was busy with PT7. For PT7, we wanted to simplify a lot of the complexity that can be found deep within the heart of the implementation of some of the standard classes. This tuned out to be a far more lengthy task then expected. But now is the right time to iron out the last few remaining kinks, even if this means breaking compatibility.

Module system fine-tuning

First, we changed the module system. The using keyword was used to make a source file refer to another source file. After the using clause, a sequence formatted like foo.bar.(...).baz was interpreted as a reference to an existing module/source file on disk and imported as such. Like a lot of things in Z2, this is an evolution of a system from C++. In this case this was a more advanced form of f#include, but this time coupled with a powerful module system. And it worked very well. But we did find a small problem that in practice may have been a moot point, but it was still worth to fix. Top level source files were referring to lower lever modules and so on until the bottom level ones were reached. Using this hierarchy, a net semi-unavoidable increase in the number of imported symbols was noticed. The module system gave you to power to choose what to make available and what to keep private, but still a small increase was leaking from layer to layer.

So we changed the sequence after the using to refer to a fully qualified entity and decoupled it from its location on disk. I shall explain this in detail one day on the blog, but it is a variant on the C# system. But in short, a source file can refer to one or more fully qualified classes and other entities and it is the job of the compiler to supply the on disk location to them. You can still organize source codes in any way you see fit and there is no compounding public symbol pollution. And since we made sure for the standard library than fully qualified class names were in the right place on disk, this change had zero compatibility break. Compilation has gotten slightly slower because of this change, but we’ll fix this in the next versions.

Greatly simplified parameter type system

Z2 is all about combining good and best-practice inspired designs with powerful compilers to reduce the complexity and fussiness of problems. This is why we use the term of “dependency/declaration order baby-sitting”, declared it a “bad thing” and went ahead to eliminate it. Another thing we wanted to eliminate was ambiguity, especially when it came to calling functions. Like most things in life and programming, ambiguity is not binary, but a spectrum. Things that have low to medium ambiguity are often resolved in programming languages by conventions and rules. In Z2 we took this to its limit and created a language than can resolve any level of ambiguity, if it is possible to resolve of course. In consequence, the rules were extremely complicated when dealing with the types of formal parameters. We set out to create a language more powerful than C++, but with better and more sane rules, and for the most part we think we were very close to achieving this. But for formal parameters and eliminating all ambiguity by rules, we failed: we created a rule-set that is easy to learn, but almost impossible to master. The exact opposite of our goal.

We tried a solution in 2015 and now in January 2016 we tried another one. Things were better but still too complicated. So we reevaluated the problem and the value proposition of having all ambiguities resolved and came to the conclusion that… it is not worth it! We rewrote the system and now common, expected and useful ambiguities are resolved by a set of easy to learn and master rules and for the rest, we are not attempting to resolve them. We give an ambiguity related compilation error! This brings Z2 in line with other languages when it comes to the effort of learning and overall we feel that this is far better place to be in related to complexity!

Low level array clean up

Z2 has a wealth of high-level containers you should use, like Vector. It also has a couple of very low-level static vectors, RBuffer and SBuffer. Unless you are writing bindings for an existing C library, using some existing OS API or declaring static look-up tables, there is no good reason to use these low-level buffers. Still, they are in use in the heart of the standard library when calling OS features, and there was a small problem with them.

RBuffer (short for RawBuffer) is the new name starting with PT7. Before, it was called Raw. It is a raw static fixed size low-level array, like standard arrays in C. Unlike C, RBuffer has a an immutable Length and Capacity, but they are not stored in RAM. When needed, they are supplied based on compile time information. So if you have a RBuffer of Byte with a Capacity of 4, it will occupy exactly 4 bytes of RAM, not 4 plus the size of the buffer. SBuffer (short for SizeBuffer) was called previously Fixed and is low-level static fixed capacity array, that has a mutable Length that is stored in RAM and an immutable Capacity that is not stored in RAM. So a SBuffer of Byte with a Capacity of 4, will occupy RAM like this: a PtrSize sized chunk to store the Length and exactly 4 bytes. The Capacity is supplied based on compile time information, without it taking up memory. So the difference between SBuffer and RBuffer is that SBuffer has an extra field to store Length.

So far so good. The small problem of library design came from the way we were using these two types. We noticed that in most cases, a RBuffer was passed as in input, but when using RBuffer as an output, we always had an extra reference parameter to store “length”. So we refactored the standard library and now in 99% of cases a const RBuffer is used as in input and a ref SBuffer is used as an output parameter. Additionally, the low-level parts of the library no longer use pointers when those pointers are meant to be a C arrays, but use RBuffer instead. This creates a cleaner standard library.

Function overloading code refactored

All these welcome simplifications worked together and allowed us to refactor and greatly simplify the function overloading code. Simple rules give simple code and now the old super complicated implementation is gone and replaced with a far shorter one that is faster than ever!

Conclusion and future versions

This chunk of simplifications turned out very well. The language is in a far better place right now. Easier to learn, easier to master and cleaner API overall.

On the downside, implementing all this took more than expected. Starting with PT8, we want to do shorter release cycles. This means that the target final PT version moves up from about PT15 to about PT20. To compensate for this, we won’t wait until we have a very stable and feature complete compiler before we make it available for testing and instead will release a super early pre-alpha version of the compiler and ZIDE for adventurous people.

PT7 is in feature freeze and we need a couple of weeks more to fix some bugs, but starting with PT8 the standard library code will begin upload to GitHub. An account was created and I’ll write a couple of explanatory post related to the standard library and then one by one, classes will be tested, documented and uploaded.

The site is dead, long life the site!

Hi and welcome to our new blog!

This is the official blog of the Z2 project! The project is a programming language and a research project. As a programming language, Z2 is a modern pure OOP general-use performance-centric design-driven systems programming language, with a rich standard library and a programming style designed around clear object relationships and lifetime rules in order to achieve automatic deterministic management of most resources without the need for garbage collection. It is similar to C++ in capabilities and performance profile, but tries to improve upon the less desirable features of C++ and better support modern programming styles and some best practices.

But enough about the project for now! We have the entire future of the blog to talk about said subject. Today I will explain why the move to a blog and what happened to the old site.

We had a full site, domain and hosting, where I started uploading some scarce content. The progress on the site was very slow because we are not web developers, nor do we enjoy such tasks. C++ programmers do not always enjoy HTML and CSS work.

When creating the site, there was an offer for multiple sites with a deep discount, so we went for it. We had a third party buy the plan so that the rest of the sites could be used for different purposes not related to the project. This third party did not inform me that the hosting plan was approaching the renewal period, nor about the multiple warning e-mails that arrived in their inbox and the site got suspended, with the content deleted.

Luckily, there wasn’t that much content lost. The detailed version-by-version changelog was lost, as were the source code statistics. We also had a fairly long description of what the language is about.

Since nothing that important was lost, it was decided to move forward using a wordpress.com blog. The old site idea was ambitious, with it being a SSO solution containing project related content, documentation, blog, news, forum and a code browser, to name the main features it was supposed to have. At the current moment a simple blog should serve us better so we can concentrate on just publishing information, rather than web development. If we ever outlive the capabilities of this blog, the site will return, but this time we shall hire a dedicated web developer and the site should be finished in a couple of months at most.

The plan is to provide frequent updates using this blog. The only problem is the source code. Z2 will be open-sourced, starting with the standard library. The old site idea had the code browser component meant to offer proper syntax highlighting for the language.

With the site being dead, we will upload the code to some form of public repository, using probably either GitHub or Gitorious. This will work well as a version control and distribution platform, but shall lack proper syntax highlighting for our language, so using those services for online code browsing will be a less than ideal solution.

For the immediate future, I shall create a post detailing the project structure and versioning scheme. And I shall create a long running CrashCourse series designed to walk you though the main language features. And I shall try to pick a better theme for the blog and customize it a bit.

See you next time!