But before we all start panicking, let me preface this by saying it was on my employer's z/D&T (Development and Test) environment — not a client system. No mainframes were hurt (much) in the making of this blog post.
So what happened?
TLDR:
It turns out the SYS1.NUCLEUS dataset can't have extents allocated. It must be exact in size because it needs to be loaded into contiguous memory at IPL time. This is probably common knowledge for experienced mainframers, but for baby mainframers like me this is completely new information — hence the post about it, I must share this exciting news.
So our system issued the following message upon re-IPL:
AWSEMI307I Warning! Disabled Wait CPU 0 = 00020000 00000000 00000000 00000081
And if we go to the IBM z/OS 3.1 Documents and look at the disabled wait codes it tells us exactly that — "Initial program load (IPL) tried to load a module from the SYS1.NUCLEUS data set. The SYS1.NUCLEUS data set or an IEANUC0x or IEAVEDAT member occupies more than one extent." [1]
I will tell myself that IBM wouldn't have this documented if people didn't do it on the regular, to make myself feel better.
And Now for the Long Version
Let's lay down some facts first.
- Our test system runs z/OS from the target libraries of the SMP/E zone it is installed in.
- Is this a bad thing? Yeah, a little bit.
- Did I make an inactive copy of the target libraries and apply maintenance to those instead? No way, this is a test system and it is prime cowboy hours over here (yeehaw).
And thus, my debacle began.
I applied maintenance with a standard SMP/E apply job. The job came back and reported that SYS1.MIGLIB ran out of space during the apply. No worries. We'll increase the size.
Usually I do this by making a copy of the dataset into a *.NEW with a larger allocation — maybe I even throw a little secondary space on there so I don't have to come back later (hint: first mistake). Just because some system datasets can have secondary space, doesn't mean it's wise or best practice to do that.
I now have my slightly larger SYS1.MIGLIB.NEW allocated. Hooray for me.
Now comes the second big no-no of the day. I want to swap my SYS1.MIGLIB.NEW with SYS1.MIGLIB. For datasets with no ENQs against them, I can just rename the original SYS1.MIGLIB to SYS1.MIGLIB.OLD and rename SYS1.MIGLIB.NEW to SYS1.MIGLIB. And the system is (hopefully) none the wiser.
Except SYS1.MIGLIB has ENQs on our system — specifically from XCFAS and LLA. That's the cross-system coupling facility and the library look-aside.
I can stop LLA temporarily with a /P LLA console command and start it with an /S LLA — I've done it before and the world hasn't ended, so that's our LLA ENQ taken care of. Next, our LPAR being worked on is a monoplex. So the XCFAS is probably not vital to its existence. But I can't just stop it. XCFAS is part of JES and stopping JES is… dramatic.
So I went looking for another way and found a very promising article about the dynamic linklist facility. Specifically, I found this set of commands:
SETPROG LNKLST,UNALLOCATE SETPROG LNKLST,ALLOCATE
Did I read the rest of the article? Of course not, who has time for that (mistake the third). I took my linklist unallocate command and went on my merry way.
I stop LLA, I unallocate the entire linklist, I swap the dataset names, then I reallocate the linklist and start LLA back up. And everything seems great — so I rerun my apply job to get the rest of that maintenance on now there's enough space.
The maintenance fails two more times for out of space in SYS1.LINKLIB and SYS1.NUCLEUS. But that's OK because I know the workaround I used for SYS1.MIGLIB! Let's do that two more times and give these datasets loads of secondary space (I'm going to stop counting the mistakes — suffice to say, please don't do this).
This was around the time I started getting abends all over the place — certain commands weren't working, it was chaos! Because LLA couldn't look up the commands I was running in storage — the secondary storage had moved everything around when the dataset expanded to accommodate the new members installed by the SMP/E apply job.
So according to some very fast panic-googling I did, this problem can be solved with an /F LLA,REFRESH (a modify refresh of LLA to force it to update the locations of all the members). However that turned out to be a lie and jobs and commands were still abending. A second source I found recommended rebuilding the linklist. No problem, I've had a little practice at that… but that didn't fix things either.
It is at this point that I decided to consult one of my more senior co-workers (you know who you are) who very kindly puts up with my 24/7 dumb questions and breakage of things. I would like to know if there is any way out of this besides an IPL? There is not.
Which circles nicely back to the beginning of this post. Wait State 081.
So How Do We Fix This?
I have no way of mounting this SYSRES elsewhere to any of the other LPARs, because they've all been set up as copies of this first one and all the SYSRES volumes have the exact same VOLSER name.
BUT. That also means I can just pinch one of the other LPARs' SYSRES volumes quickly and IPL off that. Huzzah, the LPAR is saved (ish).
And now begins the process of making an inactive target copy of the z/OS SMP/E zone — so that we can apply maintenance to that without cooking a second SYSRES. Once we're ready to build a new one, we can initialize the volume with the dodgy SYS1.NUCLEUS, redeploy a copy of the new target datasets to that, and hopefully nothing else goes sideways.
To be continued…?
References
[1] IBM, Wait State Codes: https://www.ibm.com/docs/en/zos/3.1.0?topic=wsc-081
[2] IBM, Using Dynamic Lnklst Facility: Using Dynamic Lnklst Facility Safely and Properly