The Delivery Barrier

This is a repost of an article published on the SAP Community Network, extended by a proposal I made later in the discussion.


Introduction

In the last years, ABAP development has become a lot easier. The individual developer’s perspective has improved considerably through advanced tooling (ADT!), better online documentation and community support. When I started ABAP development around 2001, you still needed a dedicated machine with quite a price tag only to run the development system of our landscape at a decent speed. Today, that’s easily accomplished using a virtual machine, even on cheap off-the-shelf hardware (which is obviously not recommended for a production system, but hey – what do you think the average Hudson CI server in a small development shop runs on?). With pre-packaged demo systems, a complete ABAP development environment is in reach for most people – I just installed a system a within a few hours (including downloading 15 GB of installation files, setting up the VirtualBox server and the underlying OS, while doing other stuff alongside). If you don’t want to run the system on your own hardware, you can get a CAL account and run the systems in the cloud as well.

From an organizational point of view, things have become easier (that is: cheaper) as well. For the most part, you no longer need to invest in large-scale hardware (see above), and there are plans available (see below) that will provide you with up to 25 developer licenses, a registered namespace and access to pretty much everything you need for very reasonable prices.

A lot has been done to lower the entry barrier, to get individual developers and possibly small startup companies to embrace the ABAP ecosystem more easily, and I am very grateful for all the hard work that must have gone on behind the scenes to make this a reality. If you know ABAP and have some great ideas for additional products that complement some existing solution, just build your product, test it thoroughly, write whatever documentation you deem necessary, sell it to your customers and then deliver it.

“Just deliver it” – unfortunately, that is the one point that is still easier said than done. This has not been a problem so far, when ABAP development was only available to medium-to-large-scale enterprises anyway. For a small company comprising of only a few enthusiastic employees, ABAP development has now become rather easy and cheap, while the delivery process has yet to undergo that transformation. There are a couple of options available that I would like to discuss in this article. I’m very well aware that this has become a rather lengthy article, but I hope it’s worth the read – giving the discussions of the last weeks, it most certainly needed writing.

The Obvious: “No. Nonononono.”

Okay, that’s not a serious alternative. You could, if you were so inclined, manually copy your solution over to each and every customer system. Don’t. Even. Think. About. It.

The Automated Obvious: “No.”

There’s a great tool available to get development objects out of the system and back into the system: SAPlink. Don’t get me wrong, it really is a great tool, if used by a knowledgeable mind for the right purposes. Software delivery is definitely not one of those.

SAPlink has a few advantages, which do look compelling, from a distance:

  • It’s free, which is always good, right?
  • It’s comparatively easy to learn – no complex concepts, just export to XML and reimport. Anyone can learn that fast.
  • It’s extensible – if it doesn’t support what you need, just add the missing pieces yourself.

However, when using it as a delivery tool, there are some serious issues.

  • It is most definitely not supported by SAP. Some of the import/export implementations available might not even use supported APIs to extract and insert development objects – probably because there are none. While this might not be huge deal for in-house use of SAPlink, the situation changes when you’re an external solution provider. You’ll be providing components for mission-critical enterprise systems – better make sure you’re covered when it comes to maintenance issues.
  • There are virtually no integrity checks during either the export or the import process. A lot of things can go wrong when packaging software, and SAPlink is simply not designed to handle any of these issues.
  • There is no support for object deletion. This happens frequently in product maintenance – you no longer need a class, so you delete it. SAPlink might be able to deliver the class, but it can’t deliver the deletion.
  • There’s no dependency resolution during import. Imagine a class that uses a structure that contains a data element that in turn refers to an interface. You might need to import some of these objects in the right order because it largely depends on the object type import/export plugin whether you can import inactive objects that refer to other objects that don’t yet exist. Sometimes you can’t, and then you have to keep tabs on the dependencies manually. Not cool.
  • Speaking of the plugins, the support for certain objects heavily relies on the plugins working correctly. Since the plugins come from a handful of loosely connected volunteers, they will naturally vary in quality, so YMMV.
  • One of the most important points might be that on the target side, SAPlink actually does little less than automate the object creation. You can create a class using SAPlink just like you could manually – and vice versa, you can’t (legally, and most of the time technically as well) create a class that you couldn’t create manually as well. That means that you have to develop and deliver your solution in a namespace that is fully writeable by your prospective customer. Either you use Y or Z and risk conflicts with existing customer objects (besides demonstrating that you probably shouldn’t be delivering a product just yet), or you give the customer full access (production key) to a registered namespace, essentially rendering its protective character useless. Welcome to maintenance hell – oh and don’t forget that some of the object type plugins don’t (yet) support objects in registered namespaces.

Compared to the Copy&Paste approach, SAPlink certainly is a huge step – but for professional software delivery, a huge step in an entirely wrong direction.

The Slightly Less Obvious: “Not A Good Idea.”

Anyone who knows a bit about the ABAP environment will know what’s coming next: Transports. From my personal experience, this is probably the most popular way to deliver add-ons (it certainly is within the Healthcare sector) – we even get add-ons from the consulting departments that are as close to SAP as can be via transports. And why not – this method does have a number of advantages:

  • It is supported by SAP. Perhaps not exactly for the use case of delivering software to customers, but it is somehow covered by the standard maintenance.
  • Not only is it officially supported, but it is also widely tested, and it is guaranteed to be implemented in every system landscape you might want to deliver software to – simply because everyone needs it.
  • There is support for all relevant object types – obviously, since you need that capability within a customer system landscape anyway.
  • It does a really great job of handling stuff like import-level ordering, inactive import and mass activation and import post-processing. There is a huge amount of complexity involved that is cleverly hidden underneath a tool that every ABAP developer uses without even thinking about it. I’d highly recommend the course ADM325 to every ABAP developer to get a deeper knowledge about the inner workings of the system.

This looks like the perfect solution, but: the CTS/TMS that you know from your development or customer landscape was designed for transports within the system landscape. It was not designed for software delivery between different landscapes, although it is frequently used for this. Because of this off-label use, there are some nasty issues that are just waiting to bite you:

  • Transport identifiers are generated automatically using the system ID (SID) of the exporting system (<sid>K9<nnnnn>). Not considering some special values, there is no telling what SIDs you might encounter in a customer system landscape. If the SID of your delivery system exists in a target landscape, transport identifiers will collide sooner or later, with very unpleasant effects. That means trouble, and there is no easy way to solve this.
  • Only an extremely limited consistency checking is performed before exporting a transport. Basically, if it’s a valid and mostly consistent object, you can export it. That includes Z-Objects, Test stubs, Modifications and a few other things that might easily slip into delivery transport unnoticed. You can implement checks for this using CTS extension points, but you have to be aware of the danger and prepare for it.
  • There is no support for modification adjustments, no SPAU/SPDD upgrade assistance. Your customer can modify your delivered objects (provided that you supply the modification key of the namespace, which you should), but then what? With the next delivery of that object, the customer has to backup his modifications (manually), have your transport overwrite it and re-implement the modifications (again, manually).
  • There is no integrated dependency management. The CTS/TMS is supposed to be used in a system landscape that has a homogeneous arrangement of software versions, so whatever you can safely export from the development box, you’ll probably can import safely into the other boxes, right. If the transport originates from a system where HR happens to be installed and maybe you used some HR data elements or function modules just because they seemed convenient, you can export the transport easily. If the customer doesn’t have HR installed, you won’t land on the moon today – and you have no way of ensuring this beforehand, you’ll just notice during the import. The same pattern applies if you want to supply multiple add-ons that rely on each other – you can’t ensure that your customer will import these in the right order and only using matching versions.
  • Speaking of versions – there is no version management to speak of, you’ll have to store the version number and patch level manually if you want to and build your own display function. Not a big deal, but cumbersome none the less.
  • The import order of individual transports is not governed in any way. This not only affects dependencies (as discussed above), it also allows for mishaps like partial downgrades, import of patches in the wrong order and numerous other issues that will keep your support staff unnecessarily busy. Even worse, unintended downgrading of database tables might lead to data loss.
  • One rather subtle problem lies hidden in the area of object deletion and skipped transports. With CTS/TMS transports, it’s easily possible to export a deletion record for an object that will cause the object to be deleted in the target system as well. Let’s assume you export that deletion record with version 5. The customer decides (consciously or by accident) to skip version 5 and upgrade directly from 4 to 6. In that case, the deletion record is not imported and the object stays in the system. In most cases, that won’t be a problem, but if you think of class hierarchies, interface implementations and other heavily interconnected objects, you might end up with leftovers of the unpleasant sort. This isn’t easy to solve, either: It’s not trivially possible to add a deletion record to the transport of version 6 because the TADIR entry of the object was deleted when exporting version 5, and you can’t add the deleted object to the transport of version 6 without creating the TADIR entry first. It’s possible, but not trivial – BTDT.
  • There’s a procedural trapdoor that might lead to unexpected results as well. Since you’re essentially using the normal change management software logistics system of the customer system landscape, your software upgrades might be imported by the TMS along with regular in-landscape transports (that they technically are!) inadvertently. If that happens at the wrong time – especially when importing into the production system – bad things might occur. Avoid if possible.
  • As a last hidden obstacle: There’s no support for a clean upgrade path between releases. Your software will inevitably use some components of the NetWeaver Basis, ECC, CRM or any other SAP product – I’ll simply call this “the foundation” from now on. For different releases of the foundation your product relies upon, you will frequently have to deliver slightly different product versions. This means that during a major upgrade, objects may have to be deleted while others might have to be added or changed. You have to figure out a way to support this manually – there’s nothing in the CTS/TMS that will help you with that.

As you can see, while this obvious solution will be workable with a number of limitations for a wide variety of scenarios, it is far from ideal, it places a huge burden on the people managing the export and import process, and it will totally collapse in certain situations (SID collisions). One would think that there just has to be a better way.

Pricey Professional Product Preparation

Fortunately, there is a better way – after all, SAP manages to deliver a wide range of software based on the NetWeaver ABAP platform, and it all has to be packaged somehow. The software to do so (or at least a software that is capable of doing so) is available; it’s known as the Add-On Assembly Kit (AAK for short; the software component is AOFTOOLS, probably for Add-On Factory Tools).

The online documentation of the AAK is freely available at http://help.sap.com/aak, so I won’t try to replicate all that’s written there. In a nutshell, the AAK provides two tools that assist with defining, checking and maintaining the contents of a deliverable software package (Software Delivery Composer) and turning that package into an installable file (Software Delivery Assembler). While the whole process uses TMS tools internally, the entire process is much more sophisticated and specially designed to support the “delivery to anonymous customer” scenario.

Note that although the AAK is an installable unit, it’s not a product that you can buy a license for. You sign a separate contract with the SAP Integration and Certification Center to have your solution certified, and as part of the process you get the AAK. The details are specified here and here as well as in note 929661.

The advantages of this approach, to name only a few, are:

  • Supported by SAP, used by SAP. What better reference could you wish for?
  • There’s extensive documentation available – online reference, 130 pages PDF, even SAP Tutor introductory videos when I last had the chance to use it. In addition, you’ll be assigned a personal contact, and at least for the people I’ve had the pleasure to work with, their competence and professionalism leaves nothing to be desired.
  • The import tools, namely transactions SAINT and SPAM, are known to most basis admins. In contrast to the common TMS transport operations, they are designed for imports of large software packages and deal with all kinds of issues.
  • If you ever wondered where the strange software component names come from – with the AAK, you get your shot at creating your own software component. The name of the software component is based on a registered namespace and therefore guaranteed to be unique; the delivery object lists contain the software component identifier and are therefore unique as well. The final EPS delivery files contain the system ID and installation number, which in this combination are unique as well (at least unique enough for all intents and purposes). Collisions with customer system IDs are thus avoided.
  • There are extensive consistency checks during the packaging and export process that can even be extended by customer checks. For instance, as an i.s.h.med developer, you may want to stop some generated function groups from being delivered because they need to be generated on the customer system. Writing a custom check for this is rather straightforward.
  • As the import uses the well-known SAINT/SPAM route, you’ll get full SPDD/SPAU modification support, including modification adjustment transports that can be used to automatically adjust the modifications on QA and production systems.
  • There’s an integrated dependency management system that allows you to specify which software components have to be present or may not be present in which versions. These dependencies are checked early during the import process – if a dependency is not met, the entire import won’t happen.
  • The AAK provides support for various upgrade or cross-grade scenarios, including release upgrades. You can build special Add-On Exchange Upgrade packages that allow you to cleanly remove objects that are no longer required in the new release and import whatever new objects are needed. This is fully integrated into the upgrade process itself.
  • Hardly worth mentioning, but of course there is full support for deleting objects. With the rather recently released version 5.0, there’s even support for deletable add-ons.
  • Compared with transports, a lot of additional checks take place during the installation process, including version checks (e. g. a downgrade protection) and collision checks with other application components.
  • Since an entirely different set of tools is used for import, AAK Add-Ons can’t be mixed up with regular transports.
  • Aside from regular versions (“Releases”), the AAK provides support for patches. The patch import process enforces the correct order of patches, thus ensuring a consistent software state on the customer system.
  • Your software will be listed in the component version list via System > Status, which is something that every software developer should aspire.
  • Finally, the certification process will get you a check by SAP and a certification logo, as well as a listing in the partner directory.

Considering the many advantages, one would have to think that this just has to be the solution to all problems. So why isn’t everyone using it? Obviously, there have to be some drawbacks. Let’s see…
It’s largely unknown to the general public, or at least that’s my observation. I hope I’m helping to change that, bit by bit.

  • The certification isn’t exactly cheap. One of the articles mentioned above has the figures – 15k€ for the first year and 10k€ for every subsequent year.
  • An extensive system landscape is required or at least strongly recommended for the packaging and testing process. The usual rule of thumb is three systems per foundation release supported – depending on your requirements, this number might change in either direction, but it’s a good estimate for starters.
  • The delivery process looks huge at first. It can be cut down once you get to know the system better and once the scenario is well-defined – there are many special cases that might not apply in your scenario, but you’ll have to think about them and decide whether to handle them consciously.

That last point is actually not necessarily a drawback: The AAK forces you to think about a lot of things that could go wrong and devise ways to prevent them. It imposes a certain level of quality – you can’t just deliver anything, at least not without consciously suppressing the warning telling you that you’re about to make a mistake.

The Steep Incline

After going through the delivery processes in detail, what’s the point?

Starting SAP development is nowadays relatively easy and cheap – if I read the SAP PartnerEdge Program for Application Development information correctly, you get a full-fledged package for 25 named developers for 2.5k€ per year. This will enable you to develop software and deliver it using transports, which – as we have seen – is not an optimal solution.

If you wanted to upgrade to the professional solution, you’ll easily end up paying ten or twenty times as much. You have to consider the basic certification fee (15k€ for the first year) for one thing, but you will also have to provide and maintain the system landscape. This might be an ideal opportunity for the cloudizationalized world – just use some pre-configured instances available from the SAP Cloud Appliance Library, link them up in a transport domain and there you are, right? Right, just that each system will easily set you back 1k€ or more in CAL and cloud provider fees – per month). So for certification and three cloud-based systems during the first year, you’ll probably end up with a total bill in the area of 30-50 k€. Add to this the effort required to setup the delivery process if nobody is familiar with the AAK and the basis administration tasks (which is probably the case), and the point becomes clear:

Getting the software to the customer using the tool that is ideally suited for the job is so expensive in time and money that many (most?) small companies and even some of the largest there are resort to a sub-optimal solution. One would hope this might change, so that now that we have the development environment available for a fraction of the cost of a few years ago, the same would apply for the delivery solution.


Addition:

"SAP R/3" could provide a much better user and customer experience than it does today if it was easier to produce deliverable software. With AAK 5, companies would have a way to "just try" the software since add-ons can be made uninstallable now.

I also understand the need for a certified product, but at the moment, that leads to a deadlock situation: Nobody wants to (or is able to) invest >> 20k€ to build some cool add-on and deliver it in a certified way, so no market interest is generated, so there are no prospective customers, so nobody will even try to invest. This leads to the zoo of custom in-house additions of (usually) inferior quality you described, plus a "grey market" of add-ons delivered via transport where you can't even be sure that installing it will not harm your system.

How about a different approach, just as an argument in this discussion (I don't get to decide things like these anyway):

  1. Release AAK for customers and PartnerEdge partners for free use, perhaps after signing an agreement that will protect SAP (like the current agreement does).
  2. Existing and future certified partners get a cryptographic key, like the ones used to protect the namespaces (maybe you could even tie this to the namespace registration system?). That key can be configured and is used for signing the deliverable when packaging and delivering the add-ons.
  3. When installing the add-on, SAINT checks for a signature. If no signature is present, the customer is notified that he's trying to install an uncertified package and that SAP takes no responsibility whatsoever.

This would open AAK up for a more widespread use while still preserving the additional value of the certification. And if a small solution provider grows through sales of "uncertified" add-ons, at some point it would be in his own interes to get that certification just to get that nasty message out of the way...

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer